From Validation to Optimization: How Phase 3 of the cuzk SNARK Engine Was Proven and Phase 4's First Campaign Revealed the Gap Between Theory and Practice

Introduction

The development of high-performance cryptographic proving systems is a discipline of dualities: architecture and micro-optimization, theory and measurement, confidence and humility. This article chronicles a pivotal transition in the cuzk project — a persistent, GPU-resident SNARK proving engine for Filecoin's Proof-of-Replication (PoRep) protocol — spanning the boundary between two distinct engineering phases. In Phase 3, a systematic four-test validation campaign proved that cross-sector batching worked correctly on real hardware, delivering a 1.42× throughput improvement with synthesis costs fully amortized across multiple sectors. In Phase 4, the team pivoted to compute-level micro-optimizations, implementing five changes drawn from a detailed optimization proposal — only to discover that the combined effect was a 19% regression, not an improvement.

The story of this segment is not one of unbroken success. It is a story of disciplined validation followed by honest measurement, of theoretical estimates colliding with real hardware behavior, and of the engineering maturity required to revert changes when the data says no. The segment closes not with a triumphant benchmark but with the addition of detailed CUDA timing instrumentation — a commitment to evidence-driven optimization that will inform every subsequent change.

Phase 3: Completing the Validation Campaign

The Phase 3 cross-sector batching feature had been implemented in the previous segment: a BatchCollector component that intercepts proof requests before they reach the synthesis pipeline, groups them by circuit type, and holds them until either a configurable batch size is reached (max_batch_size) or a timeout expires (max_batch_wait_ms). When a batch is flushed, the collector invokes synthesize_porep_c2_multi{num_sectors=N}, which synthesizes all N sectors' circuits together in a single rayon-parallelized pass. The GPU then proves the combined circuits sequentially, and a split_batched_proofs mechanism separates the combined output into per-sector Groth16 proofs.

The key insight driving Phase 3 was that CPU synthesis is already saturating all available cores when processing a single sector's 10 circuits. Adding another sector's 10 circuits keeps those same cores busy for the same wall-clock duration — the synthesis time is essentially constant regardless of how many sectors are in the batch. The GPU, which processes circuits sequentially, does scale linearly. The net effect is that two proofs complete in roughly the time it would take to synthesize one proof plus prove two, yielding an amortized throughput gain.

But this was a hypothesis. It needed to be tested on real hardware with production-scale data.

Establishing the Baseline

Before any batching could be validated, the assistant needed a reliable baseline. This baseline had already been established in earlier messages: a single PoRep C2 proof processed through the Phase 2 pipeline (async overlap between CPU synthesis and GPU proving) completed in approximately 89 seconds — 55.6 seconds for CPU-bound circuit synthesis across 10 partitions, and 34.4 seconds for GPU-bound multi-scalar multiplication (MSM) and number-theoretic transform (NTT) operations. The peak memory footprint was 202.9 GiB RSS, with the 44 GiB SRS (Structured Reference String) parameters resident in memory throughout.

The baseline was not merely a number — it was a validated prediction. The architecture documents had estimated these figures, and the assistant confirmed them through direct measurement using a memory monitor script that sampled RSS from /proc every second. The baseline CSV contained 628 samples spanning the full test duration, giving the assistant confidence in both the peak and average memory values. This baseline would become the reference point against which all batching improvements would be measured.

The Four-Test Validation Campaign

The assistant designed four orthogonal test scenarios, each targeting a distinct behavioral requirement of the Phase 3 architecture. This systematic decomposition is a hallmark of disciplined engineering: rather than running a single "does batching work?" test, the assistant isolated each behavioral branch and validated it independently.

Test 1: The Timeout Flush ([msg 735]). What happens when a single proof is submitted to a daemon configured with max_batch_size=2? If the BatchCollector waits forever for a partner that never arrives, the system would hang on every singleton submission. The assistant submitted a single PoRep proof and observed a queue time of 30,258ms — within 0.86% of the configured 30,000ms timeout. The daemon logs confirmed synth_timeout_flush{batch_size=1}. The timeout mechanism worked correctly, the SRS preloading eliminated disk I/O overhead (srs=0 ms), and the single-proof path through the batch collector added no synthesis or GPU overhead beyond the baseline.

Test 2: The Batch of Two ([msg 738]). This was the core validation of the entire Phase 3 architecture. The assistant submitted two concurrent PoRep proofs and observed that both completed in 125.3 seconds total, with synthesis time of 55.3 seconds for 20 circuits — virtually identical to the 55.6 seconds for 10 circuits in the baseline. The GPU time of 69.4 seconds was approximately 2× the single-proof 34.4 seconds, confirming linear GPU scaling. The amortized throughput was 62.7 seconds per proof — a 1.42× improvement over the 89-second baseline. The daemon logs confirmed every stage: synth_batch_full{batch_size=2}, synthesize_porep_c2_multi{num_sectors=2}, total_circuits=20, and split_batched_proofs producing correct per-sector boundaries.

Test 3: The Overflow Test ([msg 741]). With max_batch_size=2, three concurrent proofs should result in the first two forming a batch and the third waiting for a timeout before being processed singly. The results confirmed this: proofs 1 and 2 completed in 133.9 seconds (the pair that filled the batch), while proof 3 completed in 186.7 seconds with a queue time of 87,355ms — closely matching the time needed for the first batch's synthesis plus GPU phases. Critically, the third proof's synthesis overlapped with the first batch's GPU phase, demonstrating that the Phase 2 async pipeline continued to function correctly alongside batching.

Test 4: The WinningPoSt Bypass ([msg 744]). Non-batchable proof types must bypass the BatchCollector entirely. WinningPoSt proofs use different circuits and SRS parameters than PoRep, complete in milliseconds rather than minutes, and are time-sensitive. The assistant submitted a single WinningPoSt proof and observed a total time of 807ms (synthesis 52ms, GPU 666ms, queue 88ms). The daemon logs confirmed the bypass: synthesize_winning_post was called directly, with no batch-related messages. The WinningPoSt SRS parameters were loaded lazily on demand, completing transparently within the 807ms total.

Memory Analysis and Documentation

With all four tests passing, the assistant turned to memory analysis ([msg 747]). The results validated the theoretical estimates: single-proof peak RSS was 202.9 GiB, batch=2 steady-state peak was ~360 GiB (closely matching the predicted ~408 GiB when accounting for SRS overhead), and the 3-proof overflow test peaked at 420.32 GiB due to transient pipeline overlap.

The assistant then transformed this experimental data into durable engineering knowledge. It verified system state, surveyed the existing project document structure, and added a comprehensive "E2E Test Results" section to cuzk-project.md with 121 lines of structured content including the four-test matrix, performance comparison table, memory analysis, and architectural confirmation. The changes were committed as 353e4c2a with the message: "docs(cuzk): Phase 3 E2E test results — batch=2 validated at 1.42x throughput". Every Phase 3 testing task was marked as completed in the todo list ([msg 768]), and the assistant asked for direction on Phase 4.## Phase 4: The Optimization Campaign Begins

The user's response was a single, five-word directive ([msg 770]): "Proceeed to phase 4 @c2-optimization-proposal-4.md" (the charming typo preserved from the original). This launched one of the most technically dense optimization campaigns in the project's history. The referenced document — a 1031-line optimization blueprint — had emerged from weeks of call-chain analysis, bottleneck characterization, and performance modeling. It identified 18 distinct optimizations across five categories (CPU synthesis, host-device transfer, GPU NTT, GPU MSM, and micro-optimizations), prioritized into three waves. The user's directive selected Wave 1: the highest-impact, lowest-risk, lowest-effort items.

The Fork-and-Patch Infrastructure

The assistant's first response ([msg 771]) was not to start editing files, but to conduct a thorough codebase reconnaissance. Four parallel subagent tasks were launched to explore the dependency chain: one to examine bellpepper-core's LC Indexer, one to map the CUDA code in supraseal-c2, one to understand bellperson's ProvingAssignment and synthesis pipeline, and one to explore the sppark MSM library.

This reconnaissance revealed a critical fact: both bellpepper-core and supraseal-c2 came from crates.io, not from local forks. This meant the assistant could not simply edit files in place — it had to create local forks and patch them into the workspace via Cargo's [patch.crates-io] mechanism. Creating local forks of upstream dependencies is a non-trivial engineering task. The assistant had to locate the source code in the Cargo registry cache, copy it into extern/bellpepper-core/ and extern/supraseal-c2/, preserve all build infrastructure including Cargo.toml, build scripts, and CUDA source files, add [patch.crates-io] entries to the workspace Cargo.toml, and verify resolution via cargo check. The dependency chain was complex: cuzk-core depends on bellperson (already forked), which depends on bellpepper-core (now patched) and supraseal-c2 (now patched). The supraseal-c2 crate in turn vendors its CUDA dependencies (sppark) directly in its source tree. Each link in this chain had to be verified before any optimization could be implemented.

Implementing the Five Wave 1 Optimizations

With the fork infrastructure in place, the assistant implemented five optimizations across the three codebases.

A1: SmallVec for LC Indexer (bellpepper-core). The PoRep circuit has approximately 130 million constraints per partition. Each constraint is enforced via an enforce() call that creates three LinearCombination objects, each containing two Indexer structs. Each Indexer holds a Vec<(usize, Scalar)> — a heap-allocated vector of (variable index, coefficient) pairs. The key insight was that most LinearCombinations in the PoRep circuit have 1-3 terms — SHA-256 gadgets (XOR, AND, carry, majority, choose) dominate the circuit, and these all produce 1-2 term constraints. The heap allocation for a 1-element Vec was pure overhead.

The fix was to replace Vec<(usize, Scalar)> with SmallVec<[(usize, Scalar); 4]>. The 160-byte inline storage (4 elements × 40 bytes each) fits on the stack, eliminating heap allocation for the vast majority of LCs. The estimated savings: ~780M alloc/dealloc cycles × ~15ns each = ~11.7 seconds per partition. The change required adding smallvec as a dependency, updating the Indexer struct definition, and modifying the insert_or_update method to use smallvec! instead of vec!.

A2: Pre-sizing for ProvingAssignment (bellperson). The ProvingAssignment struct contains four large vectors (a, b, c, aux_assignment), each growing to ~130M elements of 32 bytes each (~4 GiB per vector). These vectors start empty and grow via push(), undergoing ~27 reallocations each due to the doubling pattern of Vec. The total reallocation copies across all four vectors was approximately 32 GiB of memory copies per partition.

The fix was to add a new_with_capacity constructor that pre-allocates all vectors to their final size. The assistant added a SynthesisCapacityHint struct and a synthesize_circuits_batch_with_hint function that accepts the hint and creates ProvingAssignment instances with pre-sized vectors. For 32 GiB PoRep, the hint specified approximately 131M constraints, 131M aux variables, and 64 input variables. This required adding bitvec as a dependency of bellperson (for BitVec::with_capacity) and threading the hint through the PoRep synthesis call sites in cuzk-core.

A4: Parallelize B_G2 CPU MSMs (supraseal-c2). The B_G2 tail MSM computes multi-scalar multiplications in the G2 group for each circuit. The original code ran these sequentially: a for loop iterating over circuits, each calling mult_pippenger with the thread pool. For 10 circuits at ~5s each, this was ~50s on CPU while the GPU might be idle.

The fix was a one-line change in spirit: replace the sequential loop with groth16_pool.par_map(num_circuits, [&](size_t c) { ... }), passing nullptr as the pool parameter to mult_pippenger so each circuit runs single-threaded. The parallelism comes from the par_map itself — each circuit executes on a different thread. With 10 circuits and 32 CPU cores, 10 threads run concurrently, reducing B_G2 time from ~50s to ~5s.

B1: Pin a/b/c Vectors with cudaHostRegister (supraseal-c2). The a, b, c vectors originate from Rust Vec<Fr> allocations, which are pageable host memory. When cudaMemcpyAsync is called with a pageable source, the CUDA runtime must copy the data into an internal pinned staging buffer before initiating DMA, reducing effective bandwidth from ~25 GiB/s to ~10-15 GiB/s.

The fix was to call cudaHostRegister on each circuit's a, b, c arrays at the entry of generate_groth16_proofs_c, and cudaHostUnregister at exit. For 10 circuits × 3 arrays = 30 calls, each pinning ~4 GiB of memory, this meant pinning approximately 120 GiB of host memory per proof. The registration cost was estimated at 50-100ms per call, totaling 1.5-3.0s of overhead.

D4: Per-MSM Window Tuning (supraseal-c2). The three tail MSMs (L, A, B_G1) have different popcount distributions. The original code used a single msm_t object sized for the average popcount, causing suboptimal window sizes for all three. The fix was to create three separate msm_t instances, each tuned for its specific popcount. This improves bucket utilization and integration phase efficiency, with negligible additional GPU memory (~10-50 MB).

The Moment of Truth: Benchmarking

With all five optimizations implemented and compilation verified, the assistant ran an end-to-end single-proof benchmark on the RTX 5070 Ti with real 32 GiB PoRep data ([msg 862]). The result was sobering:

| Phase | Baseline (Phase 3) | Phase 4 (All Wave 1) | Delta | |-------|-------------------|----------------------|-------| | Total | 88.9s | 106.0s | +19% | | Synthesis | 54.7s | 61.6s | +12.6% | | GPU prove | 34.0s | 44.2s | +30% |

Every number moved in the wrong direction. The optimizations that had looked so promising on paper had produced a 19% regression.

Root Cause Analysis and Surgical Reversion

The assistant immediately launched into root-cause analysis. Two prime suspects emerged:

A2 (Pre-sizing) — Synthesis regression. The new_with_capacity constructor allocated 328 GiB of virtual memory upfront (10 circuits × 8 vectors × ~4 GiB each). While the allocation itself was fast (it reserves virtual address space), the subsequent synthesis loop had to fault in physical pages for all 328 GiB. On a system with ~200 GiB already in use (SRS + intermediate data), this caused a page-fault storm. The previous incremental-doubling strategy, while causing more reallocation copies, spread memory pressure over the entire synthesis duration, allowing the kernel to batch and coalesce page faults more efficiently.

B1 (Pinning) — GPU regression. Each cudaHostRegister call on a 4 GiB region required walking the page tables for ~1M pages, locking each page via mlock, and updating the CUDA memory map. With 30 calls per proof, the registration overhead was estimated at 1.5-3.0s, but the actual impact was much larger — possibly due to TLB shootdown, memory pressure, or CUDA driver synchronization delays. The 8.6s gap between the bellperson-reported GPU time (35.6s) and the wrapper-reported GPU time (44.2s) was almost exactly explained by pinning/unpinning overhead.

The assistant's response was surgical reversion ([msg 866]): the A2 hint usage was removed from the synthesis call sites, reverting to the original synthesize_circuits_batch function. Critically, the new_with_capacity API and the synthesize_circuits_batch_with_hint function were kept in the bellperson fork — the API might be useful in the future with proper memory management (e.g., pre-faulting with MADV_WILLNEED or using hugepages). The revert was targeted: only the call sites changed, not the underlying infrastructure.

The Pivot to Instrumented Science

The user's response to the regression data ([msg 867]) was decisive: "Run microbenchmarks and maybe log in detail." This directive transformed the optimization campaign from blind implementation into evidence-driven science. Rather than guessing which optimizations helped and which hurt, the assistant would instrument the code to measure each phase independently.

The assistant added detailed phase-level timing instrumentation to the CUDA code using std::chrono ([msg 870] through [msg 883]). The instrumentation targeted every major phase of the GPU pipeline:

Lessons for Engineering Practice

The Phase 4 optimization campaign, spanning from reconnaissance through regression through instrumentation, offers several enduring lessons that extend far beyond the specific context of SNARK proving.

1. Measure before and after. The assistant had a clear baseline (88.9s from Phase 3) and could immediately detect the 106s regression. Without the baseline, the regression might have gone unnoticed, and the team would have shipped a slower product. This is the most fundamental rule of performance engineering, yet it is violated more often than any other.

2. Theoretical estimates are not reality. The A2 optimization was estimated to save 5-10% of synthesis time by eliminating reallocation copies. In practice, the upfront allocation of 328 GiB caused page faults that dominated the savings. The B1 optimization was estimated to cost 50-100ms per call but caused 8.6s of overhead. The only way to know if an optimization works is to measure it on real hardware with real data. The gap between estimate and reality is not a failure of analysis — it is the normal state of complex systems.

3. Implement incrementally. Implementing all five optimizations simultaneously made it harder to isolate the cause of the regression. A better approach would be to implement and benchmark each optimization individually, or at least to group them by subsystem. The instrumentation added after the regression will enable this incremental approach going forward.

4. Keep the API, revert the usage. When reverting A2, the assistant kept the new_with_capacity API in the bellperson fork. This is good practice: the API might be useful in the future with different memory management (pre-faulting, hugepages), and keeping it avoids having to re-implement it later. The cost of maintaining an unused API is negligible compared to the cost of rediscovering the need for it.

5. Instrument for the future. The timing instrumentation added after the regression will pay dividends in every subsequent optimization attempt. Each change can be evaluated with precision, and regressions can be detected immediately. The CUZK_TIMING markers transform the proving pipeline from a black box into a transparent system where every phase's contribution is visible.

Conclusion

The segment spanning Phase 3's completion and Phase 4's first campaign tells a story that is both specific and universal. Specific in its technical details — SmallVec inline storage, CUDA page-locked memory, Pippenger window tuning, Groth16 proof structure — but universal in its arc: the transition from architectural innovation to micro-optimization, the gap between theoretical estimates and measured reality, the discipline of reverting when data contradicts expectation, and the pivot from blind optimization to instrumented science.

Phase 3 was a triumph of systematic validation. The four-test campaign proved that cross-sector batching worked correctly on real hardware, delivering a 1.42× throughput improvement with synthesis costs fully amortized. The memory analysis validated predictions within 3%. The documentation workflow ensured that every finding was permanently recorded in the project's institutional memory.

Phase 4 was a different kind of triumph — not of performance improvement, but of engineering maturity. The five Wave 1 optimizations were individually plausible, each grounded in sound reasoning and careful analysis. But the system — the real system, running on real hardware with real data — revealed interactions and overheads that no analytical model could have predicted. The 328 GiB page-fault storm from pre-sizing. The 8.6s of pinning overhead from cudaHostRegister. The 19% regression that turned a promising optimization campaign into a debugging exercise.

The engineering response to that regression is what defines this segment. Not the regression itself — regressions happen in every optimization effort — but the methodical, evidence-driven response: measure, hypothesize, revert surgically, instrument for precision, and plan systematic A/B testing. This is performance engineering at its finest: not avoiding failure, but learning from it with the right tools and the right mindset.

The instrumentation added in the wake of the regression transforms Phase 4 from a one-shot optimization attempt into an ongoing, data-driven optimization campaign. Each of the five changes — A1, A2, A4, B1, D4 — can now be tested in isolation, measured with precision, and either validated or discarded based on evidence. The regression was not a failure; it was a discovery. And the discoveries made in this segment will inform every optimization decision that follows.

For anyone studying the cuzk project, this segment serves as a template for how to transition from architectural validation to micro-optimization: validate the architecture systematically, implement changes with discipline, measure honestly, revert when the data says no, and instrument for the future. The 1.42× throughput improvement from Phase 3 and the hard-won instrumentation from Phase 4 are both valuable outcomes — but the methodology that produced them is the lasting contribution.## References

[1] "The Validation of Cross-Sector Batching: How Phase 3 of the cuzk SNARK Proving Engine Was Systematically Tested and Proven" — Chunk article covering the Phase 3 E2E GPU testing campaign (messages 715–769).

[2] "The Phase 4 Optimization Campaign: From Reconnaissance to Regression to Instrumentation in the cuzk SNARK Proving Engine" — Chunk article covering the Phase 4 optimization implementation and regression analysis (messages 770–884).

[3] c2-optimization-proposal-4.md — The 1031-line optimization blueprint identifying 18 distinct optimizations across five categories, prioritized into three waves. Referenced at [msg 770].

[4] cuzk-project.md — Project documentation updated with Phase 3 E2E test results at commit 353e4c2a ([msg 767]).

[5] Bellperson fork (extern/bellperson/) — Local fork of bellperson 0.26.0 with public ProvingAssignment, synthesize_circuits_batch, and prove_from_assignments APIs, created during Phase 2.

[6] Bellpepper-core fork (extern/bellpepper-core/) — Local fork created during Phase 4 for implementing the A1 (SmallVec) optimization.

[7] Supraseal-c2 fork (extern/supraseal-c2/) — Local fork created during Phase 4 for implementing A4 (parallel B_G2), B1 (cudaHostRegister), and D4 (per-MSM window tuning) optimizations.