Systematic Regression Diagnosis: How Instrumentation, Microbenchmarks, and the Discipline to Revert Saved the Phase 4 Optimization Campaign

Introduction

In performance engineering, the most dangerous moment is not when a system is slow—it is when a carefully planned set of optimizations makes things worse. The natural response is panic: revert everything, abandon the branch, start over. But the disciplined engineer knows that a regression is not a setback—it is data. Every millisecond of slowdown carries information about the system's behavior, and the goal of diagnosis is to decode that information into actionable insight.

This article chronicles the systematic diagnosis of a 19% performance regression encountered during Phase 4 of the cuzk project, a pipelined SNARK proving engine for Filecoin proof generation. What began as a confusing slowdown—106 seconds versus an 88.9-second baseline—was transformed through rigorous instrumentation, targeted microbenchmarking, and the intellectual discipline to revert beloved changes into a clear understanding of which optimizations helped and which hurt. The journey took the assistant through three distinct phases: consolidating knowledge into a comprehensive diagnostic state dump, reverting two suspected harmful changes (A2 and B1), and ultimately using a synth-only microbenchmark to identify the surprising culprit behind the remaining synthesis regression.

The Diagnostic State Dump: Consolidating Understanding Before Acting

The first critical action was not a tool call or a code edit—it was a 2,000+ word diagnostic state dump in message 885 ([msg 885]). This document, examined in depth in [1], served as the foundation for the entire investigation. It formalized the regression into five testable hypotheses, each with a mechanistic explanation:

  1. A2 (Pre-size vectors): Allocating ~328 GiB upfront via rayon caused a page-fault storm and TLB pressure
  2. B1 (cudaHostRegister): Pinning ~120 GiB of host memory touched every page via mlock, adding overhead
  3. D4 (Per-MSM window): Creating separate msm_t objects might add GPU memory allocation overhead
  4. A4 (Parallel B_G2): For single-circuit tests, this fell through to the original path (neutral)
  5. A1 (SmallVec): A pure stack optimization with no expected regression The state dump also constructed a decision tree: finish reverting A2, build with CUDA timing instrumentation, run a single-proof test to get a phase-level breakdown, then decide which optimizations to keep. This transformed an open-ended investigation into a concrete action plan with clear success criteria.

Step 1: Reverting A2 from the Remaining Call Site

The first concrete action was to complete the partial reversion of the A2 pre-sizing optimization. While the multi-sector synthesis path had already been restored to use synthesize_circuits_batch() (without the capacity hint), one remaining call site in pipeline.rs at approximately line 750 still used synthesize_circuits_batch_with_hint. This call site served the single-sector batch path, and its continued use of the hint meant that the pre-sizing optimization was still active for the most common test case.

The assistant edited pipeline.rs to replace the _with_hint call with the plain synthesize_circuits_batch() and cleaned up the now-unused imports (synthesize_circuits_batch_with_hint, SynthesisCapacityHint). The build succeeded, confirming the code compiled cleanly. However, a subtle build-system issue emerged: the CUDA source files (groth16_cuda.cu, groth16_srs.cuh) were not recompiled because the CUDA compilation artifacts are managed by build.rs and live outside the standard cargo output directory. Touching the source files and cleaning the supraseal-c2 package did not trigger a rebuild as intended—a nuance that would need to be addressed before the instrumented benchmark could run.

This step exemplifies the disciplined approach: reverting a suspected harmful change while preserving other optimizations, cleaning up the codebase, and carefully managing the build process to ensure accurate timing data in subsequent tests.

Step 2: Fixing CUDA Printf Buffering and Collecting the First CUZK_TIMING Breakdown

With A2 reverted, the next challenge was to collect the phase-level GPU timing data from the CUZK_TIMING printf's that had been added to the CUDA code. The assistant confirmed that the instrumentation was compiled into the daemon, but the printf output was lost—it was being buffered and never flushed to the log file.

The root cause was a classic Unix buffering pitfall: when stdout is redirected to a file, the C runtime switches from line-buffered to fully-buffered mode. The CUDA printf's output, being directed to stdout, was accumulating in an internal buffer that was never flushed before the program exited. The fix was to add fflush(stderr) after each timing print, ensuring that the output was written immediately.

This seemingly mundane fix unlocked the first successful collection of a phase-level GPU breakdown, and the data was immediately revealing. The CUZK_TIMING output identified B1 (cudaHostRegister) as the primary GPU culprit: pinning approximately 125 GiB of host memory added 5.7 seconds of overhead. This far exceeded the initial estimate of 150–300 ms, confirming that the mlock-equivalent call inside cudaHostRegister was touching every page of a massive memory region with severe performance consequences.

Step 3: Reverting B1 and Measuring the Impact

With the evidence in hand, the assistant reverted the B1 optimization—removing the cudaHostRegister and cudaHostUnregister calls from groth16_cuda.cu. The result was immediate and dramatic: total proof time dropped from 101.3 seconds to 94.4 seconds.

But 94.4 seconds was still 5.5 seconds above the 88.9-second baseline. The GPU time was now back to normal, but synthesis remained elevated at 60.3 seconds (versus the baseline of 54.7 seconds). This narrowed the investigation to a single suspect: the remaining synthesis regression had to be caused by one of the other Phase 4 changes—A1 (SmallVec), A4 (Parallel B_G2), or D4 (Per-MSM window).

Of these, A4 and D4 were considered unlikely to affect single-circuit synthesis time. A4 only activates for num_circuits > 1, and D4 affects GPU memory allocation, not CPU synthesis. That left A1—the SmallVec change that the assistant had initially described as "the highest-confidence change" with "no regression expected."

Step 4: Building the Synth-Only Microbenchmark

To isolate the synthesis regression without the confounding factors of GPU proving, SRS loading, and pipeline orchestration, the assistant built a dedicated synth-only microbenchmark subcommand in cuzk-bench. This microbenchmark performed only the CPU synthesis phase—loading the C1 output, constructing the proving assignments, and measuring the time taken—without ever touching the GPU.

The microbenchmark was designed for rapid A/B testing. It accepted a --variant parameter to select between the original Vec implementation and SmallVec with inline capacities of 1, 2, or 4. Each configuration was run for three iterations to account for variance, and the results were averaged.

This is a textbook example of the microbenchmarking methodology: isolate the component under test, eliminate confounding variables, run multiple iterations, and compare variants systematically. The synth-only microbenchmark stripped away the GPU proving (which added ~35 seconds), the SRS loading (which added ~5 seconds), and the pipeline overhead, leaving only the pure synthesis time.

Step 5: A/B Testing SmallVec Variants

The microbenchmark results were conclusive and surprising:

| Variant | Synthesis Time (avg of 3) | |---------|--------------------------| | Vec (original) | 54.5 s | | SmallVec cap=1 | 59.6 s | | SmallVec cap=2 | 60.0 s | | SmallVec cap=4 | 60.2 s |

Every SmallVec variant was approximately 5–6 seconds slower than the original Vec, regardless of the inline capacity. The regression was consistent across all three capacities, suggesting that the overhead was not related to spill-to-heap behavior (which would be more frequent with cap=1 than cap=4) but was instead a fundamental property of the SmallVec data structure on this system.

This result was deeply counterintuitive. SmallVec is designed to be faster than Vec for small collections by avoiding heap allocation. The LC Indexer typically holds only a handful of entries (the inline capacity of 4 was chosen based on this expectation), so the SmallVec should have been a pure optimization. Yet it was consistently slower.

The assistant's initial assumption—that A1 was "the highest-confidence change" with "no regression expected"—had been proven wrong by measurement. This is the essence of performance engineering: intuition, no matter how well-reasoned, must always yield to data.

Step 6: Preparing for Low-Level Analysis

With the SmallVec regression confirmed, the assistant began preparing to gather low-level perf stat hardware counters to understand why SmallVec is slower on the AMD Zen4 Threadripper PRO 7995WX system. The planned analysis includes measuring L1/L2/L3 cache misses, branch mispredictions, and instructions per cycle (IPC) on a single-partition run.

Several hypotheses could explain the regression:

  1. Branch misprediction: SmallVec uses an inline array with a branch to decide whether to spill to heap. If the branch predictor on Zen4 is poorly tuned for this pattern, the misprediction penalty could outweigh the heap allocation savings.
  2. Memory layout changes: SmallVec stores data inline within the struct, which changes the memory access pattern compared to Vec's heap-allocated buffer. This could cause cache line conflicts or TLB pressure.
  3. Compiler optimization differences: The Rust compiler may optimize Vec access patterns more aggressively than SmallVec access patterns, particularly around bounds checking and loop vectorization.
  4. Allocator interaction: The LC Indexer is used intensively during synthesis. If SmallVec reduces heap allocations but the allocator's free path is faster than its alloc path (due to cached memory), the net effect could be negative. The perf stat analysis will provide the data needed to distinguish these hypotheses and guide the fix—whether it's reverting A1 entirely, tuning the inline capacity, or finding an alternative optimization.

The Broader Narrative: Discipline and Rigor in Performance Engineering

What makes this investigation noteworthy is not the specific optimizations or the final results—it is the methodology. The assistant demonstrated several principles that are worth codifying:

1. Instrument Before You Speculate

The first action after detecting the regression was not to revert changes or guess at causes—it was to add instrumentation. The CUZK_TIMING printf's provided phase-level granularity that turned the regression from a single opaque number (106 seconds) into a structured set of measurements (synthesis: 61.6s, GPU wrapper: 44.2s, bellperson internal: 35.6s). This decomposition immediately revealed the 8.6-second gap between the two GPU timers, pointing directly at B1.

2. Build Microbenchmarks to Isolate Components

When the GPU regression was fixed but synthesis remained slow, the assistant did not try to optimize synthesis while it was still entangled with GPU proving. Instead, it built a dedicated microbenchmark that stripped away all confounding factors. This allowed rapid A/B testing of the SmallVec variants—four configurations tested in minutes rather than hours.

3. Trust Data Over Intuition

The SmallVec result was a humbling reminder that performance intuition is fallible. The assistant had described A1 as "the highest-confidence change" with "no regression expected." The data showed a 5–6 second regression. The correct response was not to defend the intuition but to accept the data and investigate further.

4. Have the Discipline to Revert

Perhaps the most important principle is the willingness to revert. Two optimizations were reverted in this investigation—A2 and B1—because the evidence showed they were harmful. The assistant did not become attached to the changes or try to salvage them with additional complexity. It reverted cleanly and moved on.

5. Know When to Go Deeper

The investigation did not stop at "SmallVec is slower." The assistant is now preparing perf stat analysis to understand the root cause. This is the difference between fixing a symptom and understanding a system. The perf stat data may reveal a fix that preserves the SmallVec optimization (e.g., a different inline capacity, a different SmallVec configuration, or a compiler flag) rather than simply reverting it.

Conclusion: The Regression as Teacher

The Phase 4 regression, which initially appeared as a discouraging 19% slowdown, ultimately taught several lessons about the cuzk system: