The Regression That Revealed Everything: Benchmarking Phase 4 Compute Optimizations for the cuzk SNARK Pipeline

Introduction

In the lifecycle of any performance engineering project, there comes a moment when theory meets reality — when carefully reasoned optimizations, each justified by micro-benchmarks and algorithmic analysis, are subjected to the unforgiving judgment of an end-to-end test on real hardware. Message [msg 862] captures exactly such a moment in the development of the cuzk pipelined Groth16 proving engine for Filecoin's Proof-of-Replication (PoRep). After successfully implementing Phase 3's cross-sector batching (which achieved a 1.42× throughput improvement), the development team turned to Phase 4: compute-level optimizations drawn from a detailed proposal document (c2-optimization-proposal-4.md). Five distinct changes were implemented across three codebases — a local fork of bellpepper-core, a local fork of bellperson, and a local fork of supraseal-c2 — and then tested with a single 32 GiB PoRep proof on an RTX 5070 Ti GPU. The result was not the hoped-for speedup but a sobering regression: 106 seconds versus the 89-second Phase 3 baseline. This message, and the analysis it triggered, became the catalyst for a deeper understanding of how memory allocation patterns, GPU pinning overhead, and system-level interactions can defeat even well-motivated micro-optimizations.

The Message in Full

The message is concise and procedural. After starting the cuzk daemon with a batch_size=1 configuration and confirming it was ready (messages [msg 860] and [msg 861]), the assistant executes a single benchmark invocation:

echo "=== Phase 4 single proof test ===" && echo "Start: $(date +%T)" && /home/theuser/curio/extern/cuzk/target/release/cuzk-bench --addr http://127.0.0.1:9821 single -t porep --c1 /data/32gbench/c1.json --miner-id 1000 --sector-num 1 2>&1 && echo "End: $(date +%T)"

The output reports a completed proof with the following timings:

• Total: 106,017 ms (106.0 seconds)
• Queue: 248 ms
• SRS load: 0 ms (SRS was already resident from daemon startup)
• Synthesis: 61,550 ms (61.6 seconds)
• GPU prove: 44,219 ms (44.2 seconds) Compared to the Phase 3 baseline of 88.9 seconds total (54.7s synthesis + 34.0s GPU), this represents a 19% regression — synthesis slowed by 12.6% and GPU slowed by 30%. The message itself does not interpret these numbers; it simply records them. But the very next messages ([msg 863] through [msg 866]) launch into a detailed root-cause analysis that would reshape the team's approach to optimization.

The Five Optimizations Under Test

To understand why the regression occurred, one must understand what was being tested. The Phase 4 Wave 1 comprised five changes:

A1 — SmallVec for LC Indexer: In the bellpepper-core fork, the Vec<(usize, Scalar)> used by the Linear Combination Indexer was replaced with SmallVec<[(usize, Scalar); 4]>. This was designed to eliminate approximately 780 million heap allocations per partition by storing small vectors inline on the stack rather than on the heap. This change is purely algorithmic — it trades heap traffic for stack usage and should, in theory, reduce allocation overhead without increasing memory footprint.

A2 — Pre-sizing for ProvingAssignment: Also in the bellperson fork, a new_with_capacity constructor was added to ProvingAssignment, allowing the synthesis engine to pre-allocate all internal vectors to their final capacity before circuit evaluation begins. The intent was to avoid the ~32 GiB of reallocation-and-copy overhead that occurs when vectors grow via repeated doubling. Each circuit has approximately 131 million constraints, and with 8 internal vectors per assignment, the total allocation per circuit is ~32.8 GiB. With 10 circuits running in parallel (the standard PoRep configuration), this means 328 GiB of upfront allocation all at once.

A4 — Parallelize B_G2 CPU MSMs: In the supraseal-c2 CUDA code, a sequential loop that ran B_G2 multi-scalar multiplications one circuit at a time was replaced with a groth16_pool.par_map call, parallelizing across all circuits. This should improve CPU utilization during the B_G2 phase, which is compute-bound on the CPU.

B1 — Pin a,b,c vectors with cudaHostRegister: Also in supraseal-c2, the Rust-provided a, b, and c scalar arrays (each ~4 GiB per circuit) were pinned using cudaHostRegister/cudaHostUnregister to enable faster host-to-device transfers via DMA. For 10 circuits × 3 arrays = 30 calls, each pinning ~4 GiB of memory, this means pinning approximately 120 GiB of host memory per proof.

D4 — Per-MSM Window Tuning: The single msm_t object (sized for the average popcount across L, A, and B_G1 MSMs) was split into three separate instances, each tuned for its specific popcount distribution. This should improve GPU utilization by matching window sizes to the actual data distribution.

Why This Benchmark Matters

This message is the first end-to-end validation of Phase 4. It represents a critical juncture in the development cycle: the point at which isolated micro-optimizations are integrated and tested as a system. The decision to run a single-proof benchmark (rather than jumping straight to batch=2) was deliberate and correct — it isolates the compute changes from the batching changes, providing a clean comparison against the Phase 3 baseline.

The benchmark uses real 32 GiB PoRep data (loaded from /data/32gbench/c1.json) and runs on actual hardware (RTX 5070 Ti). This is not a synthetic micro-benchmark or a unit test; it is the full pipeline executing a production-relevant workload. The timings are wall-clock, end-to-end, including all overheads. This is precisely the kind of measurement that reveals system-level interactions that micro-benchmarks cannot capture.

The Root Cause Analysis

The subsequent messages reveal the thinking process that the benchmark triggered. The assistant immediately identifies two anomalies:

Synthesis regression (54.7s → 61.6s, +12.6%): The A2 pre-sizing change is the prime suspect. Allocating 328 GiB of memory upfront — 10 circuits × 8 vectors × ~4 GiB each — causes a page-fault storm as the kernel maps 328 GiB of anonymous pages into the process address space. On a system with finite physical memory, this triggers massive TLB pressure, potential NUMA effects, and contention across the 10 rayon threads that are all faulting pages simultaneously. The previous incremental-doubling strategy, while causing more reallocation copies, spread the memory pressure over the entire synthesis duration, allowing the kernel to batch and coalesce page faults more efficiently.

GPU regression (34.0s → 44.2s, +30%): The B1 pinning change is the prime suspect. Each cudaHostRegister call on a 4 GiB region takes approximately 50–100 ms (the CUDA driver must walk the page tables, lock the pages, and register them with the GPU). With 30 calls per proof, this adds 1.5–3.0 seconds of overhead. But the real cost may be higher: cudaHostUnregister must undo all of this, and the pinned memory may interfere with the GPU's memory management during the actual computation. The discrepancy between the bellperson-reported GPU time (35.6s) and the wrapper-reported GPU time (44.2s) — an 8.6s gap — is almost exactly explained by the pinning/unpinning overhead.

Decisions Made

The benchmark result triggered two immediate decisions:

1. Revert A2: The synthesize_circuits_batch_with_hint calls were replaced with the original synthesize_circuits_batch (no hint) at all call sites. The _with_hint API was kept available for future experimentation, but the pre-sizing was disabled. This decision was made within two messages of the benchmark result ([msg 866]).
2. Add detailed CUDA timing instrumentation: Rather than guessing at the source of the GPU regression, the team decided to instrument the CUDA code with std::chrono phase-level timers, measuring prep_msm, NTT+MSM_H, batch_addition, tail MSMs, B_G2, proof assembly, and pin/unpin separately. This would allow precise A/B testing to isolate each optimization's true impact. These decisions reflect a mature engineering approach: when an integrated optimization set regresses, do not blindly revert everything. Instead, disable the most suspicious changes, add instrumentation to measure what you cannot currently see, and then test each optimization in isolation.

Assumptions and Lessons

The benchmark exposed several incorrect assumptions:

• "Pre-allocating avoids reallocation overhead" — true in isolation, but false in a multi-threaded context where 328 GiB of simultaneous allocation causes system-level memory pressure that dwarfs the reallocation savings.
• "Pinning memory always helps GPU transfers" — true for small buffers, but for 4 GiB buffers the pinning overhead can exceed the transfer time savings, especially when the pinning is done per-proof rather than once at startup.
• "Optimizations are additive" — the five changes were designed independently, but their interactions (e.g., A2's memory pressure affecting cache behavior for A1's SmallVec operations) were not considered.

Conclusion

Message [msg 862] is a textbook example of why end-to-end benchmarking is indispensable in performance engineering. Five individually reasonable optimizations, each grounded in sound reasoning, combined to produce a 19% regression. The message itself is just a timestamp and a number — but that number triggered a cascade of analysis that revealed deep truths about memory allocation on NUMA systems, the hidden costs of CUDA API calls, and the importance of testing integrated optimization sets as a whole. The regression was not a failure; it was a discovery. And the instrumentation added in its wake would enable the precise, surgical optimizations that would eventually define Phase 4's success.