When Optimizations Backfire: Diagnosing a 19% Regression in Groth16 Proof Generation

In the high-stakes world of Filecoin proof generation, where every millisecond counts and memory is measured in hundreds of gigabytes, even well-motivated optimizations can go wrong. Message [msg 864] captures one of the most instructive moments in the cuzk pipeline development: the first benchmark of Phase 4 compute-level optimizations reveals that every change made things worse. Synthesis slowed by 12.6%, GPU time ballooned by 30%, and the assistant must now diagnose why its carefully crafted improvements have backfired — and decide what to do about it.

The Context: Phase 4 Compute Quick Wins

The message arrives at a pivotal juncture. The cuzk project has already achieved remarkable results: Phase 3 cross-sector batching demonstrated a 1.42× throughput improvement on real 32 GiB PoRep data, with synthesis costs fully amortized across sectors and GPU time scaling linearly. The architecture is sound, the pipeline is validated, and the team is ready for the next frontier: compute-level micro-optimizations.

Phase 4 draws from c2-optimization-proposal-4.md, a document that identifies nine bottlenecks and proposes targeted fixes. The assistant has just implemented five of these in a single wave:

The Diagnosis: Decomposing the Regression

The assistant's analysis in [msg 864] is a masterclass in systematic debugging. The first step is to decompose the aggregate regression into its components using the available timing instrumentation:

Synthesis: 61.6s vs 54.7s baseline (+6.9s, +12.6%). The synthesis phase, which runs entirely on CPU and involves constructing the rank-1 constraint system (R1CS) assignments for all 10 circuits, has gotten slower. The assistant immediately identifies the likely culprit: A2's pre-sizing. The calculation is brutal: 131 million constraints × 32 bytes per scalar = approximately 4 GiB per vector. With 8 vectors per circuit and 10 circuits running in parallel via rayon, that's 328 GiB of upfront allocation happening simultaneously. The previous incremental (doubling) allocation strategy spread this memory pressure over the 55-second synthesis window; the new strategy demands it all at once, triggering a page-fault storm, TLB pressure, and likely NUMA effects.

GPU prove (bellperson timer): 35.6s vs ~34s baseline (comparable). This is the GPU time as measured by bellperson's own instrumentation, which wraps the generate_groth16_proofs_c call. The small increase is within noise, suggesting that the A4 parallelization and D4 window tuning are not harming GPU performance — but also not helping much.

GPU total (wrapper timer): 44.2s vs 34s baseline (+10.2s, +30%). This is the critical discrepancy. The wrapper timer measures the entire generate_groth16_proofs_c function, including setup and teardown. The 8.6-second gap between the two timers points directly to B1: cudaHostRegister and cudaHostUnregister are called around each of the a, b, and c arrays for each circuit. With 10 circuits and 3 arrays each, that's 30 calls, each pinning approximately 4 GiB of host memory — 120 GiB total. The CUDA driver must walk the page tables, lock the pages, and register them with the GPU memory manager. At roughly 50-100 milliseconds per 4 GiB call, the math adds up.

The Reasoning: Why Each Optimization Made Sense

What makes this message so instructive is that none of these optimizations were frivolous. Each was grounded in real profiling data:

The LC Indexer's Vec<(usize, Scalar)> allocations were identified in earlier micro-optimization analysis as a major CPU hotpath, with 780 million heap allocations per partition causing measurable malloc contention. SmallVec was the textbook fix: stack-allocate the common case (4 or fewer entries) and fall back to heap only for outliers.

The pre-sizing optimization addressed a genuine inefficiency: during synthesis, the ProvingAssignment vectors grow incrementally via push, triggering repeated capacity-doubling reallocations. For a vector that ends up at 131 million elements, this means approximately 27 reallocations, each copying the entire existing data. Pre-sizing to the known final capacity eliminates this copy overhead entirely — on paper.

The B1 pinning optimization targeted a known bottleneck in GPU data transfer. Without cudaHostRegister, the a, b, and c arrays reside in pageable host memory. When the GPU kernel accesses them via direct memory access (DMA), the CUDA driver must pin and unpin the pages on the fly, adding latency to every memory access. Pre-registering the memory should, in theory, eliminate this per-access overhead.

The flaw is not in the individual analyses but in the assumption that these optimizations compose cleanly. Pre-sizing at 328 GiB creates memory pressure that dwarfs the reallocation savings. Pinning 120 GiB of host memory incurs a setup cost that exceeds the per-access savings for a single proof. The optimizations interact in ways the isolated analyses could not predict.

Assumptions and Their Failure Modes

The message reveals several implicit assumptions that proved incorrect:

Assumption 1: Upfront allocation is always better than incremental. This holds when memory is plentiful and the allocation fits in the TLB. At 328 GiB, it does not. The Linux virtual memory subsystem must fault in 328 GiB of anonymous pages, each requiring a physical page allocation, zero-fill, and TLB entry. With 10 threads competing for page table locks, the system enters a pathological regime where allocation dominates computation.

Assumption 2: cudaHostRegister overhead is negligible. The CUDA documentation describes cudaHostRegister as "moderate cost," but "moderate" is relative. At 30 calls × 4 GiB, the cost is measured in seconds, not milliseconds. The optimization assumes a single proof generation; in a continuous proving pipeline, the pinning cost would be amortized across many proofs. But for a single benchmark, it dominates.

Assumption 3: Optimizations can be validated independently and then combined. The Phase 4 wave implements five optimizations simultaneously, then benchmarks the aggregate. This conflates the signal: if A1 helps by 2 seconds, A2 hurts by 9 seconds, and B1 hurts by 8 seconds, the net regression masks any individual improvement. The assistant's subsequent actions — reverting A2 and adding detailed phase-level instrumentation — directly address this methodological flaw.

The Thinking Process: From Surprise to Action

The assistant's reasoning in [msg 864] follows a clear arc: surprise, decomposition, hypothesis formation, and decisive action.

The surprise is evident in the tone: "Interesting findings" is diplomatic, but the numbers speak for themselves. The assistant expected improvement, not regression. The decomposition follows immediately, breaking the 106-second total into synthesis (61.6s), GPU prove (35.6s), and GPU wrapper overhead (44.2s). The discrepancy between the two GPU timers is the key insight — it isolates the B1 pinning overhead as a separate, measurable cost.

The hypothesis for the synthesis regression is quantitative and specific: "131M × 32B = ~4 GiB upfront for each of the 8 vectors = ~32 GiB upfront per circuit × 10 circuits = 320 GiB." This is not speculation; it is arithmetic derived from the known constraint count and data structure sizes. The assistant then connects this arithmetic to the observed behavior: "massive upfront allocation may be causing TLB pressure or memory contention when all 10 circuits try to allocate simultaneously via rayon."

The decision to revert A2 is made in this message, with the assistant reading pipeline.rs to find the call sites. But the revert is surgical: the _with_hint API is kept available but unused. The assistant preserves the optimization's infrastructure while disabling its application, allowing future re-evaluation under different conditions (e.g., with a single circuit, or with memory pre-warming).

Input Knowledge Required

To understand this message, one needs familiarity with several domains:

Output Knowledge Created

This message produces several forms of knowledge:

Negative knowledge: The A2 pre-sizing optimization, in its current form, is harmful at the 10-circuit scale. The B1 pinning optimization incurs overhead that exceeds its benefit for single-proof generation. These are not permanent verdicts — the assistant preserves the code for future re-evaluation — but they are actionable findings for the current hardware and workload.

Diagnostic methodology: The decomposition of total time into synthesis, GPU prove, and GPU wrapper overhead establishes a template for future benchmarks. The insight that timer discrepancies reveal hidden costs (the 8.6-second gap pointing to pinning) is a general technique applicable to any system with layered instrumentation.

Decision record: The message documents why A2 is being reverted and what the expected impact is. This creates an audit trail: future readers can see that the optimization was tried, measured, and found harmful, and can re-evaluate if conditions change (e.g., with a different memory allocator, or with fewer circuits per batch).

The Broader Lesson

Message [msg 864] exemplifies a fundamental truth about systems optimization: the gap between theory and practice is bridged by measurement, not reasoning. Each of the five optimizations was theoretically sound. Each addressed a real bottleneck identified through profiling. Yet the aggregate result was a 19% regression. The assumptions that held in isolation — that pre-sizing avoids reallocation cost, that pinning avoids DMA overhead — broke down when composed at scale.

The assistant's response is exemplary: no denial, no hand-waving, no blame. Just measurement, decomposition, hypothesis, and action. The revert of A2 is immediate and surgical. The plan to add detailed CUDA timing instrumentation (executed in subsequent messages) will enable precise A/B testing of each optimization individually. This is how professional optimization works: not by guessing, but by measuring, iterating, and letting the data guide the decisions.

The message also underscores the value of keeping optimization infrastructure even when the optimization itself is disabled. The synthesize_circuits_batch_with_hint API remains in the codebase, ready to be re-enabled if the memory pressure problem is solved — perhaps by using a different allocator (e.g., jemalloc with transparent huge pages), or by pre-warming the memory pool before synthesis begins. The code is preserved; only the call sites are changed. This is the mark of a mature engineering approach: optimizations are experiments, not commitments.