The SmallVec Reversion: A Pivotal Control Experiment in Performance Debugging

At 23:08 on February 17, 2026, the assistant executed a single edit command that reverted the Indexer data structure in bellpepper-core/src/lc.rs from a SmallVec back to the original Vec. The message reads simply:

[edit] /home/theuser/curio/extern/bellpepper-core/src/lc.rs Edit applied successfully.

This terse output belies the significance of the moment. The assistant had just completed a run of a newly-built synth-only microbenchmark with SmallVec inline capacity set to 1, yielding a synthesis time of 59.5–59.9 seconds. Now it was reverting to the original Vec to obtain a clean baseline measurement. This message is the fulcrum of a rigorous A/B testing protocol—the moment when the experimental variable is removed to establish the control condition.

Context: The Phase 4 Regression Hunt

To understand why this edit matters, one must appreciate the broader performance investigation unfolding across the cuzk project. The team had successfully completed Phases 0 through 3 of a CUDA-accelerated SNARK proving pipeline for Filecoin's Proof-of-Replication (PoRep), establishing a strong baseline of 88.9 seconds for a single 32 GiB proof. Phase 4 introduced five optimizations intended to improve throughput: A1 (SmallVec for linear combination indexing), A2 (pre-sizing vectors), A4 (parallelizing B_G2 CPU MSMs), B1 (pinning host memory with cudaHostRegister), and D4 (per-MSM window tuning).

When the combined changes regressed performance to 106 seconds—a 17-second slowdown—the assistant embarked on a systematic diagnosis. The first breakthrough came from CUDA timing instrumentation (CUZK_TIMING printf's), which required fixing a buffering issue (fflush(stderr)) before producing usable data. The timing breakdown immediately identified B1 (cudaHostRegister) as the primary culprit: pinning approximately 125 GiB of host memory added 5.7 seconds of overhead, far exceeding the estimated 150–300 milliseconds. Reverting B1 brought the total proof time down to 94.4 seconds, but this was still 5.5 seconds above the 88.9-second baseline, with synthesis (60.3 seconds) now the remaining regression.

Building the Microbenchmark

The critical insight at this juncture was that the full end-to-end test—which included SRS parameter loading, GPU proving, and daemon overhead—was too slow for rapid iteration. Each test run took over 90 seconds and required coordinating the daemon process. The assistant needed a way to isolate just the CPU synthesis phase and measure it repeatedly.

The solution was a synth-only subcommand added to cuzk-bench, the project's existing benchmarking utility. This required: (1) adding cuzk-core as an optional dependency in cuzk-bench/Cargo.toml, (2) adding a synth-bench feature flag, (3) defining a new SynthOnly command variant in the clap argument parser, and (4) implementing the run_synth_only function that directly calls synthesize_porep_c2_batch without involving the GPU or daemon. The microbenchmark could run three iterations of a 10-partition PoRep C2 synthesis in about three minutes total, enabling rapid A/B testing of the SmallVec optimization.

The First Measurement: SmallVec cap=1

With the microbenchmark built, the assistant first tested the current configuration: SmallVec with INDEXER_INLINE_CAP = 1. This was itself a change from the original cap=4, made in response to the user's suggestion to "fit to cache line" and optimize for AMD Zen3+ architecture. The reasoning was that a cache line is 64 bytes, and a (usize, Scalar) tuple is 40 bytes (8 + 32), so cap=1 would keep the inline buffer within a single cache line. The hypothesis was that reducing the stack footprint of each Indexer from ~170 bytes (cap=4) to ~56 bytes (cap=1) would reduce L1 data cache pressure in the tight enforce() loop.

The result was clear: 59.5–59.9 seconds for synthesis. This was consistent across three iterations, indicating a stable measurement. But was this an improvement over the original Vec, or still a regression? The assistant could not answer that question without running the control experiment.

The Subject Message: Reverting to Vec

The edit command in message 1023 performs that control experiment. It reverts the Indexer type from SmallVec<[(usize, Scalar); N]> back to Vec<(usize, Scalar)>, where N was 1 in the just-tested configuration. The file in question is extern/bellpepper-core/src/lc.rs, which defines the Indexer<T> struct used by LinearCombination to track variable-to-scalar mappings during circuit synthesis.

This reversion is not merely a code change—it is a deliberate experimental design choice. The assistant is treating the optimization as an independent variable and measuring its effect in isolation. By reverting only the SmallVec change while keeping all other Phase 4 optimizations intact (A4, D4, and the reverted B1 and A2), the assistant can attribute any performance difference specifically to the SmallVec data structure.

Input Knowledge Required

To understand this message, one must know several things:

  1. The cuzk project architecture: A CUDA-accelerated SNARK proving daemon for Filecoin, where CPU synthesis produces constraint system evaluations (a/b/c vectors) that are consumed by GPU proving.
  2. The Indexer data structure: Part of bellpepper-core's LinearCombination implementation, used during circuit synthesis to map variables to their scalar coefficients. Each LinearCombination contains two Indexer instances (one for the "names" side and one for the "values" side), and three LinearCombinations (a, b, c) are created per enforce() call.
  3. The SmallVec optimization (A1): Proposed in c2-optimization-proposal-4.md as a way to eliminate heap allocations for the common case where linear combinations have 1–3 terms. The original implementation used Vec<(usize, Scalar)>, which always heap-allocates. SmallVec stores up to N elements inline, spilling to heap only when the capacity is exceeded.
  4. The AMD Zen4 Threadripper PRO 7995WX target: A 96-core workstation CPU with 32 KB L1 data cache per core, 512 KB L2 per core, and large shared L3. Cache line alignment and stack frame size are critical performance factors.
  5. The synth-only microbenchmark: A newly-built tool that calls synthesize_porep_c2_batch directly, bypassing the GPU proving pipeline and daemon infrastructure, enabling sub-minute iteration on synthesis changes.

Output Knowledge Created

The immediate output of this message is a codebase where the Indexer uses Vec instead of SmallVec. The build that follows (message 1024) recompiles the dependency chain: bellpepper-core → bellperson → storage-proofs-* → filecoin-proofs → cuzk-core → cuzk-bench. The subsequent benchmark run (message 1025) produces the baseline result: Vec synthesis at approximately 54.5 seconds.

The comparison between the cap=1 result (59.5–59.9 s) and the Vec baseline (54.5 s) is decisive: SmallVec causes a 5–6 second regression regardless of inline capacity. This finding is then confirmed by testing cap=2 (60.0 s) and cap=4 (60.2 s), establishing that the regression is not a function of inline capacity but of the SmallVec data structure itself.

The Counterintuitive Result

The most striking aspect of this investigation is the counterintuitive outcome. SmallVec was intended to be a pure optimization: eliminate heap allocations for the common case (1–3 term linear combinations), avoid pointer chasing, and keep hot data in the struct itself. On paper, this should be faster than Vec, which always heap-allocates and requires an indirect memory access through a pointer.

The fact that SmallVec is slower—and consistently so across all inline capacities—demands explanation. The assistant's subsequent investigation using perf stat hardware counters (L1/L2/L3 cache misses, branch mispredicts, instructions per cycle) reveals the likely cause: the larger stack frame of each Indexer (even at cap=1, ~56 bytes vs ~24 bytes for Vec) increases register pressure and cache footprint in the deeply nested enforce() loop. On a Zen4 core with 32 KB of L1 data cache, every extra byte of stack per iteration reduces the effective working set that can remain in L1. With Vec, the struct is small (pointer + length + capacity = 24 bytes) and the heap-allocated data is accessed through a single pointer chase that often hits L2 (512 KB per core). With SmallVec, the inline data is part of the struct itself, making the struct larger and reducing the number of Indexer instances that fit in L1.

The Thinking Process

The reasoning visible in the surrounding messages shows a disciplined approach to performance engineering. The assistant does not jump to conclusions or make assumptions about which optimization is responsible. Instead, it:

  1. Measures holistically first: Runs the full end-to-end test with all optimizations enabled.
  2. Decomposes with instrumentation: Adds CUDA timing printf's to get phase-level breakdown.
  3. Isolates variables: Reverts one optimization at a time (first B1, then tests A1).
  4. Builds a specialized tool: Creates the synth-only microbenchmark to eliminate confounding factors (GPU time, SRS loading, daemon IPC).
  5. Runs controlled experiments: Tests each SmallVec capacity (1, 2, 4) against the Vec baseline.
  6. Plans deeper investigation: Prepares perf stat analysis to understand why SmallVec is slower at the hardware level. This methodology reflects the principle that performance optimization is not about implementing clever ideas but about measuring actual outcomes. The SmallVec optimization, despite its theoretical appeal, proved harmful in practice on this particular architecture and workload. The assistant's willingness to revert it—and to invest effort in understanding why it failed—demonstrates intellectual honesty and engineering rigor.

Broader Implications

The SmallVec regression episode has implications beyond this single optimization. It validates the project's investment in instrumentation and microbenchmarking infrastructure. Without the synth-only tool, isolating the 5–6 second synthesis regression would have required dozens of 90-second end-to-end tests, each consuming GPU resources and SRS loading time. With the microbenchmark, the assistant could iterate in under a minute per configuration.

More broadly, this episode illustrates a fundamental truth about performance engineering on modern CPUs: cache behavior is complex and counterintuitive. An optimization that reduces heap allocations can regress performance if it increases stack frame size and cache pressure. The interaction between data structure layout, CPU microarchitecture, and compiler optimization is subtle and often defies simple reasoning. The only reliable guide is measurement—precise, controlled, and repeated.

The edit in message 1023, for all its brevity, represents the culmination of this disciplined approach. It is the moment when a hypothesis is tested against reality, and reality wins.