From Reconnaissance to Regression: The Phase 4 Compute-Level Optimization Campaign for SUPRASEAL_C2

Introduction

In the high-stakes world of cryptographic proof generation, where a single Filecoin Proof-of-Replication (PoRep) can consume ~200 GiB of memory and take over a minute to compute, optimization is not a luxury—it is a necessity. The SUPRASEAL_C2 Groth16 proof generation pipeline, which underpins Filecoin's storage verification system, had already undergone three phases of investigation: memory accounting that traced every GiB to its source, bottleneck analysis that identified nine structural inefficiencies, and end-to-end GPU validation of cross-sector batching that demonstrated a 1.42× throughput improvement. But these were architectural victories. The next frontier was compute-level micro-optimization—the kind of work that squeezes performance out of individual data structures, memory allocation patterns, and GPU kernel configurations.

This article chronicles Phase 4 of that investigation: a campaign to implement five targeted optimizations from the c2-optimization-proposal-4.md document, spanning both CPU-side circuit synthesis and GPU-side proof computation. The campaign began with a meticulous codebase reconnaissance of the bellperson fork, proceeded through the creation of local forks for two critical upstream dependencies, implemented four optimizations (with one partially reverted), and culminated in an end-to-end benchmark that revealed a surprising regression—a regression that itself became a source of insight, driving the addition of detailed phase-level timing instrumentation that will enable precise A/B testing in the rounds to come.

The Opening Move: Codebase Reconnaissance

Phase 4 opened not with code changes but with questions. The optimization proposals in c2-optimization-proposal-4.md targeted specific data structures and code paths, but their feasibility depended on the exact current state of the codebase. Had the ProvingAssignment struct been refactored? Did the synthesize_circuits_batch function create assignments in a way that pre-sizing could exploit? Were the upstream dependencies—bellpepper-core and supraseal-c2—pulled from crates.io or from local forks?

The user launched a subagent session with a precise, three-part request (see [msg 0]): examine the bellperson fork to find the ProvingAssignment struct definition and its new() method, the synthesize_circuits_batch function, and the Cargo.toml dependencies. This was not casual exploration. As [1] analyzes in detail, the user already knew the codebase intimately—they were sending a scout to verify ground truth before committing to implementation.

The assistant's response was a model of structured investigation. First, it confirmed file existence via parallel glob commands ([msg 1]), verifying that extern/bellperson/src/groth16/prover/mod.rs, src/groth16/prover/supraseal.rs, and Cargo.toml all existed at the expected paths. As [2] notes, this "glob-first" strategy is a defensive programming pattern—verify before you act—that builds a mental model of the codebase before committing to more expensive read operations.

Then came the pivotal data-gathering step ([msg 2]): the assistant read all three files in parallel. As [4] examines, this message is the bridge between knowing that files exist and knowing what they contain. The assistant's decision to batch the reads, its proactive agency in proceeding without waiting for permission, and its clear labeling of each file's output all exemplify the patterns of effective AI-assisted code exploration.

The reconnaissance culminated in a comprehensive report ([msg 3]) that laid bare the architecture of the CPU-GPU boundary. The ProvingAssignment<Scalar: PrimeField> struct, as [3] documents in detail, contains eight fields: three DensityTracker instances for query density, three Vec<Scalar> fields for the A, B, and C polynomial evaluations, and two Vec<Scalar> fields for input and auxiliary variable assignments. The new() method initializes all vectors as empty—no pre-allocation, no capacity hints. The synthesize_circuits_batch function creates one ProvingAssignment per circuit via Rayon's parallel map, synthesizes each circuit into its assignment, then extracts the assignment vectors via std::mem::take and wraps them in Arc for shared read-only access. The Cargo.toml revealed that both bellpepper-core (providing the ConstraintSystem trait) and supraseal-c2 (the CUDA backend) were pulled from crates.io, not from local forks.

This last finding was critical. To modify either library, the team would need to create local forks and patch them into the workspace via [patch.crates-io]. The reconnaissance had confirmed the target and revealed the path forward.

Creating the Local Forks

The dependency chain was now clear. The bellpepper-core crate, at version 0.2, provides the ConstraintSystem trait and the LinearCombination indexer—the data structure that tracks which variables participate in which constraints. The supraseal-c2 crate, at version 0.1.0, provides the CUDA implementation of Groth16 proof generation, including the multi-scalar multiplication (MSM) and number-theoretic transform (NTT) kernels.

Both needed modifications. The SmallVec optimization (A1) would target the LinearCombination indexer in bellpepper-core, replacing heap-allocated Vec<(usize, Scalar)> with stack-allocated SmallVec<[(usize, Scalar); 4]> to eliminate millions of tiny allocations per partition. The pre-sizing optimization (A2) would add a new_with_capacity constructor to ProvingAssignment in bellperson, but this required changes to the ConstraintSystem trait in bellpepper-core as well. On the GPU side, the pinning optimization (B1) and per-MSM window tuning (D4) would require modifications to supraseal-c2.

The team created extern/bellpepper-core/ and extern/supraseal-c2/ directories, populated them with the upstream source, and patched them into the workspace via Cargo's [patch.crates-io] mechanism. This is a standard Rust pattern for modifying upstream dependencies: the patched versions override the crates.io versions at build time, allowing local changes without altering the dependency declarations in each downstream crate.

Implementing the Optimizations

A1: SmallVec for the LC Indexer

The first optimization targeted the LinearCombination indexer in bellpepper-core. In the Groth16 proving pipeline, each constraint in the Rank-1 Constraint System (R1CS) is represented as a linear combination of variables with scalar coefficients. During circuit synthesis, these linear combinations are built incrementally—terms are added one at a time, and the indexer must track which variables appear in which combinations.

The original implementation used Vec<(usize, Scalar)> for each linear combination's term list. With millions of constraints per partition, and each constraint involving multiple linear combinations, this resulted in approximately 780 million heap allocations per partition. Each allocation has overhead: the allocator must find a free block, update its internal metadata, and eventually free the memory. For tiny vectors that rarely exceed 3-4 elements (most constraints involve only a handful of variables), this overhead is disproportionately large.

The fix was to replace Vec<(usize, Scalar)> with SmallVec<[(usize, Scalar); 4]>. The SmallVec type from the smallvec crate stores up to N elements inline on the stack, only spilling to heap allocation when the capacity is exceeded. Since the vast majority of linear combinations in the PoRep circuit have 4 or fewer terms, this means the 780 million allocations per partition become zero-cost stack operations—no heap interaction, no allocator overhead, no cache misses from pointer chasing.

A2: Pre-Sizing for ProvingAssignment

The second optimization addressed a different memory pathology. The ProvingAssignment struct's new() method initializes all vectors as empty. During circuit synthesis, as variables are allocated and constraints are enforced, these vectors grow incrementally. Each push() may trigger a reallocation if the vector's capacity is exceeded, and each reallocation involves allocating a new buffer, copying all existing elements, and freeing the old buffer.

For the PoRep circuit, which involves approximately 2^28 (~268 million) constraints and variables per partition, the cumulative cost of these reallocations is staggering. The analyzer summary estimates ~32 GiB of reallocation copies across all partitions—memory that is allocated, filled, copied, and freed, never contributing to the final proof but consuming bandwidth and causing page faults.

The fix was to add a new_with_capacity constructor to ProvingAssignment that pre-allocates all vectors to their final expected sizes. This requires knowing the circuit's size in advance, which is feasible for the PoRep circuit (it is a fixed circuit, not dynamically generated). With pre-sizing, each vector is allocated once to its final capacity, and all push() operations proceed without reallocation.

However, as we will see, this optimization came with a hidden cost.

A4: Parallelizing B_G2 CPU MSMs

The third optimization moved from CPU-side synthesis to GPU-side proof computation. In the Groth16 proving protocol, the B query involves a multi-scalar multiplication (MSM) over the G2 curve. The original implementation performed this MSM sequentially on the CPU, which is suboptimal because G2 operations are significantly more expensive than G1 operations (G2 has a larger field and more complex curve arithmetic).

The fix was to parallelize the B_G2 MSM using the existing groth16_pool thread pool. By changing a sequential loop to groth16_pool.par_map, the computation is distributed across multiple CPU cores, reducing latency. This is a straightforward parallelization that leverages the existing thread pool infrastructure in the bellperson codebase.

B1: Pinning a, b, c Vectors with cudaHostRegister

The fourth optimization targeted the host-to-device (H-to-D) transfer path. When the GPU needs to access the a, b, and c vectors (the polynomial evaluations computed during synthesis), these vectors must be transferred from CPU memory to GPU memory. The standard CUDA transfer path uses pageable host memory, which requires the CUDA driver to copy the data through an internal pinned buffer—an extra copy that consumes bandwidth and latency.

The fix was to pin the Rust-provided arrays using cudaHostRegister before the transfer, and unpin them with cudaHostUnregister after. Pinned memory allows the GPU to access the host memory directly via direct memory access (DMA), bypassing the intermediate copy. For vectors that are ~4 GiB each (as in the PoRep case), this can save significant transfer time.

However, cudaHostRegister is not free. It involves a system call to pin the memory pages, and for large allocations, this can take tens of milliseconds. The optimization's net benefit depends on whether the pinning overhead is outweighed by the transfer savings—a tradeoff that the initial benchmark would put to the test.

D4: Per-MSM Window Tuning

The fifth optimization addressed the MSM window size. Multi-scalar multiplication in CUDA is typically implemented using a windowing method: scalars are partitioned into windows of a certain bit width, and the points are precomputed for each window. The optimal window size depends on the number of scalars (the "popcount") and the available GPU memory for precomputation tables.

The original implementation used a single msm_t instance with a fixed window size for all MSMs. The fix was to split this into three instances, each tuned for the popcount characteristics of the L, A, and B_G1 MSMs respectively. By matching the window size to the actual distribution of scalar values in each MSM, the optimization reduces the number of point operations and improves throughput.

The Benchmark: Regression as Revelation

With all five optimizations implemented, the team ran an end-to-end single-proof benchmark on the RTX 5070 Ti with real 32 GiB PoRep data. The expectation was a clear improvement over the Phase 3 baseline of 89 seconds total (54.7s synthesis + 34s GPU). The result was a regression: 106 seconds total (61.6s synthesis + 44.2s GPU).

The synthesis time had increased from 54.7s to 61.6s. The culprit was the pre-sizing optimization (A2). The new_with_capacity constructor, when called with the full circuit size, allocated approximately 328 GiB of virtual address space upfront (across all partitions and all vectors). While this memory was not immediately faulted in (Linux uses optimistic memory allocation), the subsequent synthesis process touched all of it, triggering a massive page-fault storm as the kernel mapped physical pages to the newly allocated virtual addresses. The page fault handling overhead dwarfed the savings from eliminated reallocations.

The GPU time had increased from 34s to 44.2s. The culprit was the pinning optimization (B1). The cudaHostRegister calls for 30 vectors of ~4 GiB each (10 partitions × 3 vectors a, b, c) added substantial overhead. Each cudaHostRegister call involves pinning the memory pages, which requires the kernel to lock them in physical RAM and update the page tables. For 4 GiB regions, this takes tens of milliseconds per call, and with 30 calls, the cumulative overhead exceeded the transfer time savings.

The team responded swiftly. The A2 pre-sizing was immediately reverted in the synthesis call sites—the new_with_capacity API was kept available for future use, but the actual call sites reverted to the original new() method. The B1 pinning was not reverted, but its overhead was now precisely measured.

More importantly, the regression drove the addition of detailed phase-level timing instrumentation to the CUDA code. Using std::chrono, the team instrumented each phase of the GPU proof computation: prep_msm, NTT+MSM_H, batch_addition, tail MSMs, B_G2, proof assembly, and pin/unpin separately. This instrumentation will allow precise A/B testing in future rounds, isolating each optimization's true impact and identifying which changes need refinement.

Lessons and the Path Forward

The Phase 4 campaign is a masterclass in the iterative nature of performance optimization. Each optimization was theoretically sound: eliminating heap allocations, avoiding reallocation copies, parallelizing CPU work, reducing transfer overhead, and tuning GPU kernels. Yet the interactions between optimizations, and between optimizations and the operating system's memory management, produced a net regression.

The key lessons are threefold:

First, measure before and after every change. The regression was caught because the team ran a benchmark immediately after implementing all changes. Had they implemented optimizations one at a time with intermediate benchmarks, they would have identified the A2 and B1 regressions earlier. The new phase-level timing instrumentation will enable this granular measurement going forward.

Second, understand the full cost of an optimization. Pre-sizing seemed like an unambiguous win—avoid reallocation copies, save ~32 GiB of bandwidth. But the page-fault cost of faulting in 328 GiB of virtual address space was an order of magnitude larger than the reallocation savings. Similarly, pinning seemed like an unambiguous win—avoid an extra DMA copy—but the cudaHostRegister overhead for 30 × 4 GiB regions was substantial.

Third, keep the API and revert the usage. The team's decision to keep the new_with_capacity API while reverting its usage is a smart engineering practice. The API may prove valuable in a different context—for example, if the memory is pre-faulted using mlock() or if the allocation is done with a hugepage-aware allocator. By keeping the API, the team preserves the option to re-enable the optimization under more favorable conditions.

The Phase 4 campaign is not over. The SmallVec optimization (A1) remains in place and likely provides a net benefit (its overhead is minimal—stack allocation instead of heap allocation—and the page-fault cost is negligible). The parallelized B_G2 MSMs (A4) and per-MSM window tuning (D4) also remain, and their impact will be measured in the next benchmark round. The A2 and B1 optimizations may be re-enabled with refinements—perhaps using a memory pool to pre-fault the allocation, or using cudaHostAlloc instead of cudaHostRegister for the pinned vectors.

The overarching achievement of this chunk is not a faster proof generation pipeline—yet. It is a deeper understanding of the system's behavior under optimization, a set of reusable local forks and APIs, and a detailed timing instrumentation framework that will guide the next iteration. The regression, far from being a failure, is the most valuable data point the team has collected: it reveals that the naive application of theoretically sound optimizations can backfire, and that the path to a faster pipeline runs through careful, incremental, measured experimentation.## References

[1] "The Architect's Probe: Deconstructing a Precision Code-Reading Request in a Groth16 Optimization Pipeline" — Analysis of message 0, the user's initial reconnaissance request.

[2] "The First Step: How a Simple Glob Command Reveals the Architecture of an OpenCode Exploration" — Analysis of message 1, the assistant's glob-first verification strategy.

[3] "Mapping the Bellperson Prover: How a Struct Definition Reveals the Architecture of Groth16 Proof Generation" — Analysis of message 3, the comprehensive code report.

[4] "The Parallel Read: A Pivotal Data-Gathering Step in the Bellperson Fork Investigation" — Analysis of message 2, the parallel file-read transition.