The Cache Line Detective: Diagnosing a SmallVec Regression on AMD Zen3+

Introduction

In the high-stakes world of Filecoin proof generation, every second counts. A 32 GiB PoRep proof that takes 88.9 seconds to generate represents a carefully optimized baseline—the result of months of work across Phases 0 through 3 of the cuzk project. When Phase 4's first wave of optimizations was applied, the team expected improvements. Instead, they got a regression: 106 seconds, a 17-second slowdown. This article examines a single message in the debugging session that followed—a message where the assistant, guided by a user's brief comment to "Optimize for AMD Zen3+," performs a masterful piece of performance detective work, reasoning about CPU cache lines, stack frame sizes, and the subtle ways that an optimization intended to eliminate heap allocations can backfire on modern microarchitecture.

The Scene: A Performance Regression Under the Microscope

The subject message ([msg 995]) arrives at a critical juncture. The assistant has already completed several rounds of diagnosis. The B1 optimization (cudaHostRegister, which pinned ~125 GiB of host memory) has been identified as a primary culprit, adding 5.7 seconds of overhead, and has been reverted. The total proof time has dropped from 101.3s to 94.4s, but this is still 5.5s above the 88.9s baseline. Synthesis—the CPU-bound phase that constructs the circuit constraints—is stuck at 60.3s, compared to 54.7s in the baseline. That's a 10.5% regression.

The only synthesis change still in play is A1: replacing Vec<(usize, Scalar)> with SmallVec<[(usize, Scalar); 4]> in the Indexer struct of bellpepper-core's linear combination (LC) module. The optimization was intended to eliminate heap allocations for the vast majority of linear combinations, which in Filecoin PoRep circuits (dominated by SHA-256 gadgets) typically have 1–3 terms. With an inline capacity of 4, the SmallVec should have avoided ~99% of allocations. Yet somehow, it was making things slower.

The assistant has just finished reading the current A1 code and is preparing to test a hypothesis about cache line alignment when the user interjects with a four-word comment: "Optimize for AMD Zen3+."

The Message: Reasoning About Microarchitecture

The assistant's response ([msg 995]) is a model of disciplined performance engineering. It begins by acknowledging the user's point and immediately grounding the analysis in concrete microarchitectural facts about the target platform:

Good point — this is running on AMD Zen3+ (EPYC or Threadripper). Key Zen3+ cache characteristics: - L1d: 32 KB per core, 8-way, 64-byte lines - L2: 512 KB per core (Zen3) or 1 MB (Zen4) - L3: shared per CCD (up to 32 MB per 8-core CCD) - Stack access pattern: hot stack frames stay in L1d if they fit

This grounding is crucial. The assistant is not reasoning in the abstract about "SmallVec is good" or "Vec is slow." It is reasoning about a specific processor microarchitecture—the AMD Zen3+ Threadripper PRO 7995WX—and asking: what does this particular CPU's cache hierarchy mean for the data layout choices we made?

The analysis proceeds in two parts. First, the assistant quantifies the stack pressure problem with the current cap=4 configuration:

With cap=4, each Indexer is ~170 bytes. 6 Indexers per enforce() = ~1020 bytes of hot stack data. That's 16 cache lines in L1d — not terrible but it thrashes the L1d working set when combined with the enforce() logic accessing constraint system vectors.

The numbers are precise. A (usize, Scalar) tuple is 40 bytes (8 + 32 for the BLS12-381 Fr field element). SmallVec<4> stores 4 such tuples inline: 4 × 40 = 160 bytes, plus length and discriminant overhead (~10 bytes) = ~170 bytes per Indexer. Each enforce() call creates 3 LinearCombinations (a, b, c), each with 2 Indexers (terms and values), totaling 6 Indexers. That's 6 × 170 = ~1020 bytes of inline stack data per enforce() invocation.

In the parallel synthesis loop, each rayon worker thread calls enforce() millions of times. With 1020 bytes of hot stack data per call, the L1d cache (32 KB per core) can hold only about 31 such frames before eviction begins. But the enforce() function also accesses the constraint system's internal vectors—large heap-allocated arrays that are cold in L1. The combination of a large stack frame with scattered heap accesses creates a working set that exceeds L1d capacity, causing cache misses that stall the pipeline.

The Cache Line Insight

The assistant then pivots to the alternative: cap=1. With a single inline slot, each Indexer shrinks to ~56 bytes (40 bytes for the inline tuple + ~16 bytes metadata). Six Indexers = ~336 bytes = ~5 cache lines. This is a dramatically smaller footprint—roughly one-third of the cap=4 configuration. The entire enforce() stack frame would fit in L1d with room to spare for the constraint system accesses.

But the most interesting part of the message comes next. The assistant reconsiders the baseline assumption that Vec is slow:

But actually, let me reconsider: Zen3+ has excellent branch prediction and prefetching. The Vec path with heap allocation goes through jemalloc's thread-local cache, which on Zen3+ is very fast (~10-15ns per alloc). The pointer chase to heap data would miss L1 but hit L2 (512KB) since the thread-local arena is hot. So Vec on Zen3+ may not be as slow as the proposal assumed (which estimated 15ns from a generic allocator model).

This is a critical moment of intellectual honesty. The original Phase 4 proposal (documented in c2-optimization-proposal-4.md) assumed that heap allocations for small Vecs were costly—15ns per allocation from a generic allocator model. But on Zen3+, with jemalloc's thread-local caching, allocations are ~10-15ns, and the subsequent pointer chase from the Vec's heap-allocated buffer hits L2 cache (512 KB per core) rather than main memory, because the thread-local arena remains hot. The Vec path may not have been as slow as the proposal assumed.

This realization undermines the entire premise of the A1 optimization. If Vec allocations are cheap on Zen3+, and the SmallVec's larger stack frame causes L1 cache pressure, then SmallVec could easily be a net loss—which is exactly what the 5–6s regression suggests.

The Decision: Test cap=1

The assistant does not get stuck in analysis paralysis. It formulates a testable hypothesis and acts:

SmallVec with cap=1 is still worth testing — eliminates the alloc entirely for single-term LCs while keeping the struct small. Let me build and test.

The command that follows—killing the daemon, rebuilding with cargo build --release -p cuzk-daemon—is the practical payoff of the reasoning. The assistant has moved from "SmallVec cap=4 is causing a regression" to a specific, microarchitecture-grounded hypothesis: cap=1 will reduce stack pressure enough to recover the lost performance, while still eliminating heap allocations for the common single-term case.

What This Message Reveals About Performance Engineering

The subject message is a masterclass in several dimensions of performance engineering:

1. Grounding in microarchitecture. The assistant doesn't reason about "SmallVec is good" in the abstract. It asks: what does this specific CPU (Zen3+ Threadripper) do with this specific data layout? The answer depends on L1d size (32 KB), cache line size (64 bytes), L2 size (512 KB–1 MB), and the prefetching characteristics of the core.

2. Quantifying stack pressure. The assistant converts data structure sizes into cache line counts. 170 bytes per Indexer → 3 cache lines. 6 Indexers → 16 cache lines. 1020 bytes of stack data → exceeds L1d working set when combined with other accesses. This quantitative reasoning transforms a vague "maybe it's slower" into a specific, testable hypothesis.

3. Reconsidering assumptions. The most impressive moment is the reconsideration of Vec's cost. The original proposal assumed Vec allocations were expensive based on a generic allocator model. But on Zen3+ with jemalloc, the thread-local cache makes small allocations extremely fast (~10-15ns), and the pointer chase hits L2 rather than main memory. The assistant is willing to question its own earlier assumptions in light of new evidence.

4. The scientific method. The message follows a clear cycle: observe (synthesis is 10.5% slower), hypothesize (SmallVec's stack frame causes L1 pressure), predict (cap=1 will reduce stack pressure), test (rebuild and benchmark). This is disciplined performance engineering.

Input Knowledge Required

To fully understand this message, one needs:

Output Knowledge Created

This message produces several valuable insights:

  1. SmallVec cap=4 causes ~1020 bytes of stack data per enforce() call, likely exceeding L1d capacity and causing cache misses
  2. SmallVec cap=1 reduces this to ~336 bytes, fitting comfortably in L1d
  3. Vec on Zen3+ may not be slow due to jemalloc's thread-local cache and L2 cache hits
  4. A testable hypothesis: cap=1 should recover the regression while still eliminating allocations for single-term LCs
  5. A concrete test plan: rebuild with cap=1, run the single-proof benchmark, compare synthesis time

Broader Implications

The lesson extends far beyond this specific bug. It illustrates a fundamental tension in low-level optimization: reducing heap allocations by inlining data into stack frames is not an unqualified win. The tradeoff depends on:

Conclusion

The subject message captures a beautiful moment in the debugging process: the shift from "our optimization is causing a regression" to "I understand why it's causing a regression, and I have a specific fix to test." The assistant's reasoning about cache lines, stack frames, and jemalloc's thread-local caching on Zen3+ transforms a confusing performance anomaly into a testable hypothesis. The rebuild command at the end is not just a build step—it's the culmination of a rigorous analytical process that respects both the science of performance measurement and the art of microarchitectural reasoning.