The Third Data Point: How a Simple Benchmark Run Conclusively Proved SmallVec Was the Culprit

In the disciplined practice of performance engineering, no single measurement tells the whole story. It is the pattern across multiple data points—the systematic variation of one parameter while holding everything else constant—that reveals the truth. Message [msg 1029] captures precisely such a moment: the third and final benchmark run in a controlled A/B comparison that definitively identified the source of a 5–6 second synthesis regression in the cuzk Groth16 proof generation pipeline for Filecoin PoRep.

The Message

The message is a single bash command that executes the synth-only microbenchmark subcommand of cuzk-bench with the SmallVec inline capacity set to 4:

[assistant] [bash] echo "=== SmallVec cap=4 ==="
FIL_PROOFS_PARAMETER_CACHE=/data/zk/params /home/theuser/curio/extern/cuzk/target/release/cuzk-bench synth-only --c1 /data/32gbench/c1.json --iterations 3 2>&1
=== SmallVec cap=4 ===
=== Synthesis Microbenchmark ===
c1:         /data/32gbench/c1.json
sector:     1 (miner 1000)
partition:  all
iterations: 3

c1 loaded:  51510727 bytes
2026-02-17T23:17:57.285393Z  INFO synthesize_porep_c2_batch{job_id="synth-bench-0"}: building circuits for all partitions (parallel) num_partitions=10
2026-02-17T23:17:57.294402Z  INFO ...

On its face, this is unremarkable: a benchmark invocation, a log line, a truncated output. But to understand why this message matters, we must understand the journey that led to it.

The Context: A Performance Regression Under Investigation

The cuzk project is a pipelined, GPU-accelerated proving engine for Filecoin's Proof-of-Replication (PoRep) protocol. By message [msg 1029], the project had successfully completed Phases 0 through 3, establishing a strong baseline of 88.9 seconds for a single 32 GiB PoRep proof. Phase 4 was intended to apply a suite of compute-level optimizations to improve throughput further.

Five optimizations were implemented in Phase 4, Wave 1:

The Systematic Diagnosis

The assistant and user embarked on a methodical root-cause analysis. Using CUDA timing instrumentation (CUZK_TIMING printf's) added to the GPU code, they first identified B1 (cudaHostRegister) as the primary culprit: pinning approximately 125 GiB of host memory added 5.7 seconds of overhead, far exceeding the estimated 150–300 milliseconds. Reverting B1 brought the total proof time down to 94.4 seconds.

But 94.4s was still 5.5s above the 88.9s baseline, and synthesis (the CPU phase) was now the slowest component at 60.3 seconds—versus approximately 54.5 seconds in the baseline. The remaining regression pointed squarely at A1 (SmallVec).

To isolate synthesis from the noise of GPU proving and SRS loading, the assistant built a dedicated synth-only microbenchmark subcommand in cuzk-bench (see [msg 1001] through [msg 1012]). This tool loaded a C1 proof file, ran only the CPU circuit synthesis for all 10 partitions of a 32 GiB sector, and reported the elapsed time. Crucially, it allowed rapid A/B testing: change one constant in bellpepper-core/src/lc.rs, rebuild (approximately 15 seconds), and run three iterations (approximately 3 minutes each).

The A/B Test Design

The test was designed to isolate the effect of the SmallVec inline capacity parameter. Four configurations were benchmarked:

  1. Vec (original): The baseline using standard Vec<(usize, Scalar)>—heap-allocated, pointer-chase access pattern.
  2. SmallVec cap=1: Inline capacity of 1 element (40 bytes inline + overhead), fitting in a single 64-byte cache line.
  3. SmallVec cap=2: Inline capacity of 2 elements (80 bytes), crossing a cache line boundary.
  4. SmallVec cap=4: Inline capacity of 4 elements (160 bytes), spanning 3 cache lines—the original proposed value. The first two data points had already been collected before [msg 1029]. The Vec baseline ran at approximately 54.5 seconds. SmallVec cap=1 ran at approximately 59.6 seconds—a clear 5-second regression. The question was: does the inline capacity matter? Would a smaller capacity (cap=1) reduce the regression by keeping the struct small enough to fit in fewer cache lines, or was the problem inherent to SmallVec itself?

The Third Data Point

Message [msg 1029] provides the answer. By testing cap=4—the originally proposed value from the optimization proposal—the assistant completed the comparison matrix. The results (visible in subsequent messages) showed cap=4 at approximately 60.2 seconds, cap=2 at approximately 60.0 seconds, and cap=1 at approximately 59.6 seconds. The Vec baseline was approximately 54.5 seconds.

The pattern was unmistakable: every SmallVec variant, regardless of inline capacity, caused a 5–6 second regression in synthesis time. The inline capacity made almost no difference—cap=1, cap=2, and cap=4 all clustered within 0.6 seconds of each other, while Vec was over 5 seconds faster than all of them.

Why This Matters: The Counterintuitive Result

This result was deeply counterintuitive. The original optimization proposal (documented in the project's c2-optimization-proposal-4.md) had argued that SmallVec would improve performance by eliminating heap allocations for the common case. In Filecoin PoRep circuits, most linear combinations in SHA-256 gadgets have exactly 1 term—a single variable. For these, SmallVec with inline storage should avoid a heap allocation (approximately 10–15 nanoseconds in jemalloc) and a pointer chase to heap memory (which misses L1 cache). The theory was sound.

The reality was the opposite. The assistant's earlier reasoning (see [msg 992]) had explored several hypotheses:

Assumptions Made and Lessons Learned

Several assumptions were challenged by this result:

Assumption 1: Heap allocations are expensive. The optimization proposal assumed that avoiding heap allocations would yield a measurable speedup. In practice, on an AMD Zen4 Threadripper PRO 7995WX with jemalloc's thread-local caching, a heap allocation for a 40-byte Vec<(usize, Scalar)> takes approximately 10–15 nanoseconds—effectively free in the context of a 54-second synthesis. The pointer chase to access heap data is equally fast because the thread-local arena remains hot in L2 cache (512 KB per core on Zen3, 1 MB on Zen4).

Assumption 2: Inline storage is always faster. The conventional wisdom that "no allocation is faster than allocation" fails when the allocation is tiny and the allocator is highly optimized. SmallVec trades allocation cost for branch overhead, struct size, and code bloat. On modern CPUs with excellent branch prediction and fast allocators, the tradeoff can be negative.

Assumption 3: The bottleneck is allocation rate. The proposal assumed that the Indexer was created and destroyed frequently enough that allocation dominated. In reality, the synthesis hotpath is dominated by constraint evaluation logic (field arithmetic, gadget evaluation), not by data structure construction. The SmallVec overhead, while small per operation, is amplified by being on the critical path of every enforce() call.

Assumption 4: Zen4 behaves like the generic model. The original analysis used a generic CPU model. Zen4's massive L3 cache (up to 384 MB on Threadripper PRO 7995WX) and excellent prefetchers make heap access patterns much cheaper than on smaller, older CPUs.

Input Knowledge Required

To understand this message, one needs:

  1. The cuzk project architecture: A pipelined Groth16 proving engine for Filecoin PoRep, with CPU synthesis and GPU proving phases.
  2. The Phase 4 optimization suite: Five optimizations (A1, A2, A4, B1, D4) applied in Wave 1, causing a net regression.
  3. The diagnostic process: B1 was already identified and reverted; A2 was partially reverted; synthesis remained 5.5s above baseline.
  4. The SmallVec proposal: The intent to replace Vec<(usize, Scalar)> with SmallVec in the Indexer struct to eliminate heap allocations for single-term linear combinations.
  5. The microbenchmark tool: The synth-only subcommand built specifically for this diagnosis, loading a C1 file and running only CPU synthesis.
  6. Previous benchmark results: Vec at ~54.5s, SmallVec cap=1 at ~59.6s.

Output Knowledge Created

This message produced:

  1. The cap=4 timing data: Completing the four-way comparison matrix.
  2. The conclusive pattern: All SmallVec variants are 5–6 seconds slower than Vec, regardless of inline capacity.
  3. The refutation of the cache-line hypothesis: Since cap=1 (single cache line) is as slow as cap=4 (three cache lines), the cause is not struct size or stack pressure.
  4. The implication for next steps: The investigation must shift from "which capacity minimizes the regression" to "why is SmallVec fundamentally slower on this architecture," requiring low-level hardware counter analysis (perf stat for cache misses, branch mispredicts, IPC).
  5. The decision to revert A1: The optimization must be abandoned for this architecture, or fundamentally rethought.

The Thinking Process Visible in the Reasoning

The assistant's reasoning leading up to this message (visible in [msg 992] and [msg 995]) shows a sophisticated understanding of CPU microarchitecture. The assistant correctly identified:

Conclusion

Message [msg 1029] is, on its surface, a mundane benchmark run. But it represents the culmination of a disciplined, multi-step diagnostic process: identify the regression, build instrumentation, isolate the suspect, design a controlled experiment, and collect the data that definitively rules out a class of hypotheses. The message's true significance lies not in the numbers it produced but in the pattern it completed—the third data point that turned a correlation into a conclusion.

The lesson is universal in performance engineering: elegant theories and careful reasoning must always yield to measurement. SmallVec should have been faster on paper. The heap allocation it avoided should have been the bottleneck. But on this CPU, with this allocator, on this workload, the opposite was true. Only by running the benchmark—by actually measuring—could the team discover that their model was wrong and adjust course accordingly. That is the essence of disciplined optimization: not assuming you know the answer, but letting the data tell you where to look next.