The Microbenchmark That Caught a Counterintuitive Regression

In the disciplined practice of performance engineering, there is a moment when a carefully constructed hypothesis meets the unforgiving reality of measurement. Message <msg id=1034> captures precisely such a moment: the execution of a synth-only microbenchmark testing the SmallVec optimization with an inline capacity of 2 elements, the third data point in a systematic A/B test designed to isolate a stubborn performance regression in the cuzk Groth16 proving pipeline.

The Context: A Regression After Five Optimizations

The cuzk project had just completed Phase 3 of a multi-phase optimization effort for Filecoin's SNARK proving pipeline, achieving a solid baseline of 88.9 seconds for a single 32 GiB PoRep proof. Phase 4 introduced five optimizations drawn from a detailed proposal document (c2-optimization-proposal-4.md): A1 (SmallVec for the LC Indexer), A2 (pre-sizing vectors for ProvingAssignment), A4 (parallelizing B_G2 CPU MSMs), B1 (pinning a/b/c vectors with cudaHostRegister), and D4 (per-MSM window tuning). The combined effect, however, was a regression to 106 seconds — a 17-second slowdown.

What followed was a textbook exercise in regression diagnosis. The assistant had already added CUDA timing instrumentation (CUZK_TIMING printf's) to the GPU code, discovered that printf output was being lost due to full buffering when stdout was redirected to a file, fixed it with fflush(stderr), and collected the first phase-level GPU breakdown. That data immediately identified B1 (cudaHostRegister) as the primary culprit: pinning approximately 125 GiB of host memory added 5.7 seconds of overhead, far exceeding the estimated 150–300 milliseconds. Reverting B1 brought the total proof time from 101.3 seconds down to 94.4 seconds, but this was still 5.5 seconds above the 88.9-second baseline, with synthesis (60.3 seconds) now the remaining regression.

Building the Microbenchmark

To isolate the synthesis slowdown without the confounding overhead of GPU proving and SRS loading, the assistant built a dedicated synth-only microbenchmark subcommand in cuzk-bench. This was a deliberate architectural decision: rather than continuing to run full end-to-end tests that took over 90 seconds each, a focused microbenchmark could iterate in under a minute, enabling rapid A/B testing of the A1 (SmallVec) change.

The microbenchmark directly invoked synthesize_porep_c2_batch from cuzk-core, the same function used in the production pipeline, but without the subsequent GPU proving phase. It loaded the C1 output from a JSON file (51,510,727 bytes), deserialized it, and ran the full 10-partition circuit synthesis for a 32 GiB PoRep proof. The implementation required adding cuzk-core as an optional dependency to cuzk-bench, creating a new SynthOnly command variant in the clap argument parser, and writing feature-gated handler code that timed just the synthesis phase.

The A/B Test Matrix

The assistant systematically tested four configurations of the LC Indexer data structure:

1. Vec (original baseline): The standard Vec<(Variable, Fr)> representation, which allocates heap memory for each linear combination term but benefits from jemalloc's fast thread-local caching on the AMD Zen4 Threadripper PRO 7995WX system.
2. SmallVec cap=1: A SmallVec<[(Variable, Fr); 1]> that can store one element inline without heap allocation, growing to the heap if more elements are needed. This was hypothesized to be optimal for the common case of single-term linear combinations, keeping the struct small (~56 bytes for 6 Indexers = ~336 bytes, fitting in ~5 L1 cache lines).
3. SmallVec cap=2 (the subject message): A SmallVec<[(Variable, Fr); 2]> with inline capacity for two elements, a middle ground that could handle more common cases inline while still being relatively compact.
4. SmallVec cap=4: A SmallVec<[(Variable, Fr); 4]> with the largest inline capacity, which the original proposal had recommended. Each Indexer would be ~170 bytes, and 6 Indexers per enforce() call would be ~1020 bytes — 16 cache lines of hot stack data. The message <msg id=1034> shows the cap=2 run being launched. The full content of the message is the bash invocation and its output:

[assistant] [bash] echo "=== SmallVec cap=2 ==="
FIL_PROOFS_PARAMETER_CACHE=/data/zk/params /home/theuser/curio/extern/cuzk/target/release/cuzk-bench synth-only --c1 /data/32gbench/c1.json --iterations 3 2>&1
=== SmallVec cap=2 ===
=== Synthesis Microbenchmark ===
c1:         /data/32gbench/c1.json
sector:     1 (miner 1000)
partition:  all
iterations: 3

c1 loaded:  51510727 bytes
[2026-02-17T23:24:42.200122Z INFO synthesize_porep_c2_batch{job_id="synth-bench-0"}
  cuzk_core::pipeline: building circuits for all partitions (parallel)
  num_partitions=10
[2026-02-17T23:24:42.206851Z INFO ...

The --iterations 3 flag runs three consecutive synthesis passes, providing multiple samples to distinguish consistent performance differences from measurement noise. The log line confirms that all 10 partitions are being synthesized in parallel, matching the production pipeline's behavior.

The Results: A Clear and Surprising Signal

The microbenchmark results were conclusive:

| Configuration | Synthesis Time | |---|---| | Vec (original) | 54.5 s | | SmallVec cap=1 | 59.6 s | | SmallVec cap=2 | 60.0 s | | SmallVec cap=4 | 60.2 s |

Every SmallVec variant was 5–6 seconds slower than the original Vec implementation. The inline capacity made almost no difference — cap=1, cap=2, and cap=4 all clustered tightly around 60 seconds. This was a counterintuitive result. The SmallVec optimization was designed to eliminate heap allocations for small linear combinations, which should have been a pure win. The LC Indexer in bellpepper-core is called millions of times during circuit synthesis, and each call constructs a small vector of constraint terms. Reducing allocation pressure should reduce memory bandwidth, cache pressure, and the overhead of the allocator.

Why SmallVec Might Be Slower on Zen4

The assistant had earlier noted that the AMD Zen4 Threadripper PRO 7995WX has excellent branch prediction and prefetching, and that jemalloc's thread-local cache on this architecture is very fast (~10–15 ns per allocation). The pointer chase to heap data might miss L1 but would likely hit L2 (512 KB per core) since the thread-local arena stays hot. Under these conditions, the Vec path's heap allocations are cheap enough that the overhead of SmallVec's branching logic — checking whether the inline storage is full, deciding whether to spill to the heap, and potentially copying data — could outweigh the allocation cost.

The SmallVec struct itself is larger than a simple Vec pointer pair. A Vec<(Variable, Fr)> is 24 bytes (pointer, length, capacity) on a 64-bit system. A SmallVec<[(Variable, Fr); 2]> is larger because it embeds the inline array directly in the struct. When these structs are stored in vectors or passed around by value, the increased size causes more register pressure and more cache misses. The enforce() function in bellpepper-core creates six Indexer objects per call, and if those objects are 50–150 bytes larger each, the working set of the hot loop expands significantly.

The Deeper Significance

This message is not merely a data point in a benchmark table. It represents a critical juncture in the performance engineering process: the moment when a theoretically sound optimization is empirically falsified. The SmallVec change had been implemented with the best intentions — reducing allocation overhead is a classic optimization technique — but on this particular architecture, with this particular allocator, and in this particular hot loop, it backfired.

The broader lesson is about the importance of measurement over intuition. The Phase 4 optimization proposal had recommended SmallVec based on a generic model of allocation costs (estimating ~15 ns per allocation from a general-purpose allocator). But the actual system used jemalloc with thread-local caching on a high-end AMD workstation, where allocation was already nearly free. The optimization that was supposed to eliminate allocation overhead instead introduced branching overhead and struct size bloat that the original code never had.

The assistant's next step, prompted by the user in the following message (<msg id=1035>: "Can you gather some lowlevel perf info, cache/branch stats?"), was to run perf stat to collect hardware counter data — L1/L2/L3 cache misses, branch mispredictions, instructions per cycle — to understand the root cause at the microarchitectural level. This is the final piece of the diagnostic puzzle: moving from timing-level observation to hardware-level explanation.

Input and Output Knowledge

To fully understand this message, one needs knowledge of: the cuzk project's architecture (a pipelined Groth16 prover for Filecoin PoRep), the Phase 4 optimization taxonomy (A1–D4), the role of the LC Indexer in circuit synthesis, the SmallVec library and its trade-offs, the AMD Zen4 microarchitecture and its cache hierarchy, and the jemalloc allocator's thread-local caching behavior. The message itself is a bash command invocation, but its meaning is embedded in the experimental protocol that preceded it: the reverting of B1, the building of the microbenchmark, the testing of Vec and SmallVec cap=1, and now cap=2.

The output knowledge created by this message is the third data point confirming that SmallVec causes a ~5–6 second regression in synthesis time regardless of inline capacity. Combined with the Vec baseline and the cap=1 and cap=4 results, it provides the statistical confidence needed to make a decision: the SmallVec optimization (A1) must be reverted. The regression is real, consistent, and not sensitive to the inline capacity parameter.

Conclusion

Message <msg id=1034> is a quiet but decisive moment in a performance engineering investigation. It is the execution of a carefully designed experiment that, together with its companion measurements, falsifies a well-intentioned optimization and forces a return to the original code. The true optimization in this story is not the SmallVec change itself, but the rigorous methodology — build a microbenchmark, test multiple variants, collect multiple samples, compare against a baseline — that prevented a harmful change from being committed to the codebase. In the end, the assistant would keep only the optimizations that survived empirical validation (likely A4 and D4), revert A1 and B1, and commit a clean, validated set of Phase 4 improvements that strictly improved upon the 88.9-second baseline.