The Build That Confirmed a Regression: SmallVec's Counterintuitive Slowdown on Zen4

[assistant] [bash] cargo build --release -p cuzk-bench --features synth-bench 2>&1 | tail -5
   Compiling filecoin-proofs v19.0.1
   Compiling filecoin-proofs-api v19.0.0
   Compiling cuzk-core v0.1.0 (/home/theuser/curio/extern/cuzk/cuzk-core)
   Compiling cuzk-bench v0.1.0 (/home/theuser/curio/extern/cuzk/cuzk-bench)
    Finished `release` profile [optimized] target(s) in 15.39s

At first glance, message [msg 1028] appears to be nothing more than a routine build log — the kind of output a developer sees dozens of times per session and scrolls past without a second thought. Five lines of compiler output, a 15-second build, a clean Finished status. Yet in the context of the broader investigation, this message represents a critical inflection point in a performance regression hunt that had consumed the better part of a day's work. It is the bridge between a hypothesis and its experimental validation, the moment when the code is ready to be tested, and the quiet before the data speaks.

The Performance Regression That Drove Everything

To understand why this build matters, one must understand the crisis that precipitated it. The cuzk project — a CUDA-accelerated SNARK proving engine for Filecoin's Proof-of-Replication (PoRep) — had successfully completed Phases 0 through 3, establishing a solid baseline of 88.9 seconds for a single 32 GiB PoRep proof. Phase 4 introduced a suite of five micro-optimizations (A1 through D4) intended to squeeze additional performance from the pipeline. Instead, the first integration test returned a shocking result: 106 seconds, a regression of over 17 seconds from the baseline.

The systematic diagnosis that followed is a textbook case of disciplined performance engineering. The assistant used CUDA timing instrumentation (CUZK_TIMING printf's) to obtain a phase-level breakdown, which immediately identified B1 (cudaHostRegister) as the primary culprit: pinning approximately 125 GiB of host memory added 5.7 seconds of overhead. Reverting B1 brought the total down to 94.4 seconds, but the synthesis phase remained 5.5 seconds above baseline at 60.3 seconds.

This left A1 (SmallVec) as the prime suspect. The optimization had replaced Vec<(usize, Scalar)> with SmallVec<[(usize, Scalar); 4]> in the Indexer struct of bellpepper-core, with the intention of eliminating heap allocations for the majority of linear combinations in Filecoin's SHA-256-heavy circuits. The assumption was that most linear combinations have 1–3 terms, so an inline capacity of 4 would cover ~99% of cases without heap allocation, reducing allocation overhead and improving cache locality.

Building a Microbenchmark to Isolate the Signal

The challenge was that measuring synthesis time through the full proving pipeline was slow and noisy — each test required SRS loading, GPU initialization, and the proving phase itself. To iterate faster, the assistant built a dedicated synth-only microbenchmark subcommand in cuzk-bench ([msg 1009]–[msg 1012]). This subcommand loaded the C1 output, synthesized all 10 partitions of a PoRep C2 circuit, and timed only the CPU-bound synthesis phase, bypassing the daemon, GPU, and SRS entirely.

The first microbenchmark run with SmallVec cap=1 (the assistant had already changed the inline capacity from 4 to 1 in [msg 993] to test a cache-line alignment hypothesis) yielded 59.5–59.9 seconds across three iterations ([msg 1021]). The original Vec baseline, tested next ([msg 1025]), returned approximately 54.5 seconds. The conclusion was stark: SmallVec, regardless of inline capacity, introduced a ~5–6 second regression in synthesis time.

The Subject Message: Switching to cap=4

Message [msg 1028] is the build that follows the assistant's decision to test SmallVec cap=4 — the original inline capacity from the optimization proposal. After seeing the Vec baseline outperform cap=1, the natural question was whether the regression scaled with inline capacity. Perhaps cap=1 was too small, causing frequent spills to the heap for 2–3 term linear combinations, and the original cap=4 would perform better.

The assistant had just applied two edits to bellpepper-core/src/lc.rs ([msg 1026]–[msg 1027]), changing the INDEXER_INLINE_CAP constant back to 4. Now, with this build command, the assistant compiles the change and prepares to run the cap=4 benchmark.

The build output reveals several things. First, the compilation is fast — 15.39 seconds — because only four packages needed recompilation: filecoin-proofs, filecoin-proofs-api, cuzk-core, and cuzk-bench. The dependency chain from bellpepper-core → bellperson → storage-proofs-core → filecoin-proofs → cuzk-core → cuzk-bench propagated efficiently, with Rust's incremental compilation reusing all intermediate artifacts. The --features synth-bench flag enables the microbenchmark feature in cuzk-bench, which pulls in cuzk-core as an optional dependency.

What the Build Doesn't Say

The build log is silent about what the assistant is thinking at this moment. The assistant has already seen two data points — Vec at 54.5s and SmallVec cap=1 at 59.6s — and is now testing cap=4 to complete the picture. There is a quiet tension here: the SmallVec optimization was supposed to be a clear win, and its failure is counterintuitive. On AMD Zen4 (Threadripper PRO 7995WX), with its generous L1d cache (32 KB per core) and excellent branch prediction, the Vec path through jemalloc's thread-local cache is apparently very fast — perhaps 10–15 nanoseconds per allocation. The SmallVec, by contrast, increases the struct size from ~24 bytes (Vec: pointer + length + capacity) to ~170 bytes (SmallVec<4>: inline storage + discriminant + length), which means each Indexer struct is larger, consuming more stack space and potentially causing more L1 cache pressure.

The assistant's earlier reasoning in [msg 992]–[msg 993] had explored this tradeoff extensively, considering cache line alignment, Zen3+ cache characteristics, and the allocation patterns of SHA-256 gadgets. The hypothesis was that cap=1 would minimize struct size while still avoiding heap allocations for single-term linear combinations. But the data contradicted this hypothesis — cap=1 was slower than Vec.

The Result That Followed

The cap=4 benchmark ran immediately after this build ([msg 1029]) and returned approximately 60.2 seconds — essentially identical to cap=1 and cap=2. The regression was not sensitive to inline capacity. SmallVec was consistently 5–6 seconds slower than Vec, regardless of configuration.

This result forced a deeper investigation. The assistant pivoted to gathering low-level perf stat hardware counters — L1/L2/L3 cache misses, branch mispredicts, instructions per cycle — to understand why SmallVec was slower on this particular microarchitecture. The answer would ultimately require examining the generated assembly and the interaction between SmallVec's branching logic (checking whether data is inline or heap-allocated on every access) and the CPU's branch predictor.

The Deeper Significance

Message [msg 1028] is a reminder that performance engineering is not about intuition — it is about measurement. The SmallVec optimization had seemed obviously beneficial on paper: fewer heap allocations, better cache locality, lower memory fragmentation. But on a modern AMD Zen4 processor with a world-class allocator (jemalloc) and aggressive prefetching, the heap allocation path is so fast that the overhead of SmallVec's inline storage management dominates. The optimization that was supposed to be a free lunch turned out to have hidden costs: larger struct sizes, more register pressure, and additional branching on every element access.

This message also illustrates the value of building the right measurement infrastructure. Without the synth-only microbenchmark, testing each SmallVec variant would have required a full 90+ second E2E proof run, making A/B testing prohibitively slow. With the microbenchmark, each configuration could be tested in under a minute, enabling rapid iteration.

In the end, the SmallVec optimization was reverted, and the Phase 4 commit would proceed with only the optimizations that passed rigorous A/B testing: A4 (parallel B_G2 CPU MSMs) and D4 (per-MSM window tuning). The A1 change, despite its theoretical appeal, was discarded — a testament to the discipline of letting data, not assumptions, drive engineering decisions.