The GPU Proving Interface Deep-Dive: Unlocking Per-Partition Groth16 Proof Generation in Filecoin's SUPRASEAL_C2 Pipeline

Introduction

In the high-stakes world of Filecoin proof generation, where peak memory footprints of ~200 GiB threaten economic viability and throughput bottlenecks determine whether a proving operation can turn a profit, few questions are as consequential as this: can the GPU be called with just one circuit at a time? The answer determines whether the Sequential Partition Synthesis optimization—a proposal to dramatically reduce memory by streaming partitions sequentially rather than processing them all in parallel—is even feasible. If the GPU requires all partitions to be batched together, the memory wall remains. If it can handle one partition at a time, the entire architecture of proof generation can be reimagined.

This article synthesizes a deep-dive investigation into the SUPRASEAL_C2 Groth16 proof generation pipeline for Filecoin Proof-of-Replication (PoRep), conducted within an opencode coding session. The investigation, spanning ten messages across a subagent session, systematically traced the entire GPU call chain from Curio's Go-level task orchestration through Rust FFI bindings into C++ host code and finally CUDA kernels. The result was a comprehensive understanding of the GPU proving interface, its four internal phases, the mysterious ~23-second b_g2_msm cost, and—most importantly—a definitive answer about per-partition viability.

The Strategic Context: Why This Investigation Mattered

The root session had already mapped the full SUPRASEAL_C2 pipeline and identified nine structural bottlenecks contributing to its ~200 GiB peak memory footprint [10]. Three optimization proposals had been developed, including Sequential Partition Synthesis, which aimed to reduce peak memory by proving one partition at a time instead of loading all ten partitions' circuits into GPU memory simultaneously [5]. However, this proposal depended on a critical assumption: that the GPU proving interface could be called with num_circuits=1 (a single partition) without incurring prohibitive overhead.

The end-to-end benchmarks from Segment 20 had revealed a painful trade-off [segment 20]. The standard batch pipeline achieved 47.7 seconds per proof with 228 GiB peak memory, while the partitioned (slotted) pipeline reduced memory to 71 GiB but regressed throughput to 72 seconds per proof—a 50% slowdown. The parent investigation needed to understand whether this throughput gap was fundamental or whether it could be closed by optimizing how the GPU was called. Specifically, if the GPU could be invoked per-partition with num_circuits=1, and if the mysterious b_g2_msm cost (known to take ~23 seconds for batched calls) could be mitigated, then the partitioned path might be made competitive while retaining its memory advantage [6].

This strategic context shaped every aspect of the subagent investigation. The user's initial message (message 0 of the subagent session) was not an idle exploration—it was a targeted reconnaissance mission to determine the viability of a key optimization strategy [6].

The Methodical Descent: Tracing the Call Chain

The investigation proceeded through a systematic, top-down descent through layers of abstraction. The assistant began by reading the three primary Rust files identified by the user: supraseal.rs (the bellperson GPU prover), mod.rs (the proving assignment setup), and pipeline.rs (the Curio orchestration layer) [1][5]. These files revealed the high-level API—functions like prove_from_assignments() and synthesize_circuits_batch()—but they are wrappers. The actual GPU work happens deeper, in C++ CUDA kernels accessed through a Rust FFI binding.

Recognizing this gap, the assistant searched for the FFI layer, discovering the supraseal-c2 crate and its source files [2]. A pivotal moment came when the assistant read supraseal-c2/src/lib.rs, the Rust FFI binding that wraps the C++ CUDA code [9]. This file revealed the function signature of generate_groth16_proof—the single entry point through which all GPU proving flows. The assistant also ran a targeted grep for b_g2_msm, ntt, msm_h, and generate_groth16_proof across the entire codebase, finding 51 matches that mapped the distribution of GPU-phase logic across three key files [9].

The next step was to find the actual C++/CUDA implementation. The assistant attempted targeted glob patterns for groth16*.cu, groth16*.cpp, and groth16*.hpp files, but all three returned no results [2]. This failure revealed that the CUDA files followed different naming conventions than expected. The assistant then broadened the search to **/supraseal-c2/**, which successfully discovered the four critical CUDA files: groth16_cuda.cu, groth16_ntt_h.cu, groth16_split_msm.cu, and groth16_srs.cuh [8].

With the file inventory established, the assistant read all four CUDA files in a single round [4]. This was the critical transition from API-level understanding to implementation-level understanding. In groth16_cuda.cu, the assistant found the generate_groth16_proofs_c C FFI function—the actual entry point that the Rust wrapper calls. This file contained the static mutex serializing all GPU calls, the branching logic for b_g2_msm parallelization, and the four internal GPU phases [4][10].

The Four GPU Phases: Decomposing the Monolith

The assistant's analysis decomposed the monolithic generate_groth16_proofs_c function into four distinct phases, each with different computational characteristics [10]:

Phase A: Prep MSM (CPU thread). A CPU thread runs in parallel with the GPU, performing pre-processing of density bitvectors, mask merging across circuits, scalar/base extraction, and—critically—the b_g2_msm computation. This phase runs on the CPU because G2 operations involve Fp2 arithmetic (BLS12-381's extension field), which the GPU kernels are not optimized for.

Phase B: NTT + MSM_H (GPU). This is the most GPU-intensive phase. For each circuit, the GPU runs Number Theoretic Transforms (NTT) on the a, b, c polynomial evaluations, performs coefficient-wise multiplication, subtraction and scaling, inverse coset NTT, and finally MSM_H—a multi-scalar multiplication against the H SRS (Structured Reference String) points. This phase accounts for ~35-45 seconds per partition and scales linearly with the number of circuits.

Phase C: Batch Addition (GPU). After the CPU prep thread signals completion via a barrier, the GPU runs batch additions for L, A, B_G1, and B_G2 points. This handles the "trivial" part of the split MSM optimization, where scalars that are 0 or 1 are handled by simple point additions via bitmaps rather than full Pippenger MSM.

Phase D: Tail MSMs (GPU). For each circuit, the GPU runs Pippenger MSM for the non-trivial scalars in L, A, and B_G1. This is the remaining "significant" portion of the MSM work after the split optimization has removed the trivial scalars.

Phase E: Proof Assembly (CPU). Combines all MSM results with verifying key scalars (alpha_g1, beta_g1, beta_g2, gamma_g2, delta_g1, delta_g2) to produce the final proof points for each circuit.

This decomposition was essential for understanding where time is spent and which phases are CPU-bound versus GPU-bound. The NTT+MSM_H phase is the dominant GPU cost, while the b_g2_msm (part of Phase A) is the dominant CPU cost [10].

The b_g2_msm Mystery: Why ~23 Seconds?

One of the user's most pressing questions concerned the b_g2_msm phase and its peculiar fixed ~23-second cost for num_circuits >= 2. This question was rooted in empirical evidence from profiling—the user had clearly seen timing logs revealing this cost [6]. The assistant's analysis of the C++ code in groth16_cuda.cu (lines 521-539) revealed the answer [10]:

The C++ code branches on num_circuits:

The Definitive Answer: Per-Partition GPU Calls Are Viable

The assistant's analysis confirmed definitively that the GPU interface can be called with num_circuits=1 [10]. The evidence was drawn from existing code paths:

Proof Assembly and the 192-Byte Format

The investigation also confirmed the proof output format. Each Groth16 proof is 192 bytes: a compressed G1 element (48 bytes) for a, a compressed G2 element (96 bytes) for b, and a compressed G1 element (48 bytes) for c [7]. The Proof struct in bellperson/src/groth16/proof.rs is annotated with #[repr(C)], ensuring C-compatible memory layout for the FFI boundary between Rust and C++ [7].

The gpu_prove() function in pipeline.rs calls prove_from_assignments(), which returns Vec<Proof<Bls12>>, then serializes each proof via proof.write(&mut proof_bytes), producing concatenated 192-byte chunks. The ProofAssembler collects proof bytes from each slot/partition as they complete, indexed by partition range, and sorts them by partition range start to produce the final concatenated output [10].

The Monolithic Constraint: Why Phases Cannot Be Split

The user also asked whether the GPU work could be split so that b_g2_msm could be done once and per-partition NTT+MSM done separately [6]. The assistant's finding was definitive: no, not without modifying the SupraSeal C++ code [10]. The generate_groth16_proofs_c function is monolithic, behind a static mutex (static std::mutex mtx; std::lock_guard<std::mutex> lock(mtx);). There is no API for partial execution or result reuse. The entire pipeline—prep MSM, NTT+MSM_H, batch addition, tail MSMs, b_g2_msm, proof assembly—runs inside a single mutex-guarded call.

However, the assistant argued that for per-partition calling, this separation is unnecessary because the b_g2_msm cost drops to ~3-5 seconds anyway. The monolithic constraint is only a problem for batched calls, where the ~23-second b_g2_msm cost cannot be amortized across phases [10].

The Thinking Process: How the Assistant Reached Its Conclusions

The assistant's reasoning process is visible in the sequence of tool calls across the investigation [3]. The pattern is one of systematic elimination: start at the highest level of abstraction (Rust pipeline), trace calls through the bellperson wrapper, reach the FFI binding, and then dive into the CUDA kernels. Each step narrows the focus while building a map of the codebase.

A key metacognitive moment occurs at message 7, when the assistant declares "Now I have a complete picture" and then immediately qualifies this by seeking one final verification—reading the Proof struct to confirm the proof bytes format [3]. This pattern—assess, identify gap, verify, then synthesize—is a blueprint for systematic technical investigation.

The assistant's decision to read all four CUDA files in a single round (message 6) was strategically important [4]. Rather than reading them sequentially, the assistant recognized that understanding the GPU proving pipeline requires a complete mental model of all the moving parts. Reading the files in parallel minimized latency and allowed synthesis in the next round.

Implications for the Optimization Proposals

The findings from this investigation directly informed the optimization proposals developed in the root session. The confirmation that per-partition GPU calls are viable validated the core assumption of the Sequential Partition Synthesis proposal. The discovery that the b_g2_msm cost drops to ~3-5 seconds for single-circuit calls meant that the partitioned path's throughput penalty was not primarily due to the GPU interface but to other factors (serialization overhead, lack of inter-proof overlap) [10].

The analysis also informed the Persistent Prover Daemon proposal by revealing that the SRS loading overhead (~60 seconds per proof) is separate from the GPU proving phases and could be eliminated by keeping the proving process alive across proofs. The Cross-Sector Batching proposal was informed by the understanding that the freed memory headroom from sequential processing could be used to batch multiple sectors' circuits into single GPU invocations, projecting 2-3× throughput per GPU [10].

Conclusion

The deep-dive investigation into the SUPRASEAL_C2 GPU proving interface achieved its primary objective: confirming that per-partition GPU calls are viable and understanding the precise conditions under which they are advantageous. The investigation traced the entire call chain from Curio's Go-level task orchestration through Rust FFI bindings into C++ host code and CUDA kernels, producing a comprehensive map of the four GPU phases and the critical b_g2_msm branching behavior.

The most surprising finding—that the ~23-second b_g2_msm cost is actually faster for single-circuit calls (~3-5 seconds) than for batched calls—was a direct consequence of the user's line of questioning. This finding transformed the understanding of the partitioned path's performance characteristics and opened new optimization possibilities.

More broadly, this investigation exemplifies a principle that applies across systems engineering: performance characteristics that appear mysterious or inherent often turn out to be artifacts of implementation decisions. The ~23-second b_g2_msm cost is not a law of nature—it is a consequence of choosing single-threaded Pippenger per circuit when num_circuits >= 2. Understanding this distinction is the first step toward making better decisions about system architecture and optimization.

For the Curio project, the path forward is now clearer. The Sequential Partition Synthesis proposal can proceed with confidence that per-partition GPU calls are supported and efficient. The Persistent Prover Daemon can be designed to eliminate SRS loading overhead. And Cross-Sector Batching can exploit the freed memory headroom to improve throughput. The GPU proving interface, once a black box, is now a mapped territory—and that map is the foundation for the next generation of Filecoin proof generation optimization.