The Zen4 Perf Event Puzzle: Debugging a SmallVec Regression at the Hardware Level

In the middle of an intensive optimization session for the SUPRASEAL_C2 Groth16 proof generation pipeline, a single bash command reveals a critical moment of debugging adaptation. Message [msg 1063] is deceptively simple:

[bash] perf list 2>&1 | grep -i 'l3_lookup\|ls_any_fills\|ls_dmnd_fills'

This is not a random query. It is the culmination of a frustrating back-and-forth with the Linux perf tool on an AMD Zen4 architecture, where the assistant is desperately trying to find the right hardware counter event names to diagnose a performance regression that has been puzzling the team for several rounds of iteration.

The Context: A Performance Regression That Defies Explanation

To understand why this message matters, we must step back into the broader narrative. The team has been implementing Phase 4 compute-level optimizations for the cuzk proving engine, a high-performance Groth16 prover used in Filecoin's Proof-of-Replication (PoRep) pipeline. Four optimizations were implemented in parallel:

• A1: SmallVec optimization for the LinearCombination Indexer, replacing Vec with SmallVec<[_; 2]> to reduce heap allocations
• A2: Pre-sizing for ProvingAssignment (later reverted)
• A4: Parallelization of B_G2 CPU MSMs
• D4: Per-MSM window size tuning
• B1: Pinning a/b/c vectors with cudaHostRegister (later reverted) The A1 optimization—replacing Vec with SmallVec—was expected to be a clear win. SmallVec is a well-known Rust crate that stores elements inline on the stack for small capacities (up to the inline capacity) and spills to heap only when exceeded. For the LinearCombination Indexer, which typically holds 1–2 elements, this should reduce allocation pressure and improve cache locality. Yet microbenchmarks showed the opposite: SmallVec was slower than Vec on the Zen4 test system across all configurations (cap=1, cap=2, cap=4). This was deeply puzzling. SmallVec is widely used in production Rust code and generally performs well. The regression suggested something specific to the Zen4 microarchitecture—perhaps cache pressure from the larger inline storage, or branch misprediction from the SmallVec's dynamic dispatch between inline and heap storage.

The perf stat Odyssey

The assistant's natural next step was to collect hardware performance counter data using perf stat to compare Vec vs SmallVec at the microarchitectural level. The goal was to measure:

• Instructions retired
• Cycles
• IPC (instructions per cycle)
• Cache misses (L1, L2, L3)
• Branch mispredictions
• TLB misses This data would reveal why SmallVec was slower—whether from increased cache pressure, worse instruction-level parallelism, or branch misprediction overhead from the inline-vs-heap check. But the assistant immediately hit a wall. The initial perf stat command (in [msg 1055]) used event names that work on Intel architectures:

perf stat -e instructions,cycles,cache-references,cache-misses,branch-instructions,branch-misses,l2_cache_req_stat.dc_access_in_l2,l2_cache_req_stat.dc_hit_in_l2,l2_cache_req_stat.dc_miss_in_l2,ls_l1_d_tlb_miss.all_l2_miss ...

This failed with:

event syntax error: '.._hit_in_l2,l2_cache_req_stat.dc_miss_in_l2,ls_l1_d_tlb..'
                                  \___ Bad event name

The problem is that AMD Zen4 uses a completely different set of event names than Intel. The l2_cache_req_stat.dc_miss_in_l2 event does not exist on this architecture. The assistant then spent several messages (1056–1062) probing for the correct names:

• Checking which l2_cache_req_stat.* events exist (1056)
• Searching for L3 cache events (1057)
• Trying l3_cache_accesses and l3_misses (1058–1059), which appeared in perf list but failed with "Bad event name"
• Attempting tracepoint syntax l3_cache:l3_cache_accesses (1061), which failed with "unknown tracepoint"
• Trying metric groups with perf stat -M l3_misses (1062), which also failed Each failure taught the assistant something about the Zen4 perf event interface, but none provided the working command needed to collect the data.

The Subject Message: A Strategic Pivot

Message [msg 1063] represents a strategic pivot. Instead of guessing event names, the assistant searches the full perf list output for events containing l3_lookup, ls_any_fills, or ls_dmnd_fills—three categories that might reveal L3 cache behavior and memory access patterns.

The output reveals a rich set of available events:

ls_any_fills_from_sys.all
ls_any_fills_from_sys.all_dram_io
ls_any_fills_from_sys.alternate_memories
ls_any_fills_from_sys.dram_io_all
ls_any_fills_from_sys.dram_io_far
ls_any_fills_from_sys.dram_io_near
ls_any_fills_from_sys.far_all
ls_any_fills_from_sys.far_cache
ls_any_fills_from_sys.local_all
ls_any_fills_from_sys.local_ccx
ls_any_fills_from_sys.local_l2
ls_any_fills_from_sys.near_cache
ls_any_fills_from_sys.remote_cache
ls_dmnd_fills_from_sys.all
ls_dmnd_fills_from_sys...

These are AMD Zen4-specific events for tracking demand fills from the system (memory subsystem). The ls_dmnd_fills_from_sys events track demand-driven cache line fills—precisely the kind of data needed to understand whether SmallVec is causing more cache pressure than Vec. The ls_any_fills_from_sys events include both demand and prefetch fills, providing a broader picture.

What This Message Reveals About the Debugging Process

The message is a window into the assistant's reasoning process, even though it contains no explicit reasoning text. The choice of grep pattern is itself a hypothesis:

1. l3_lookup: The assistant was looking for L3 cache lookup events. On Intel, these are named l3_lookup_state.*. The fact that none matched suggests AMD uses different naming entirely.
2. ls_any_fills: This targets "load/store any fills" — events tracking when the CPU fills cache lines from memory. The from_sys suffix indicates fills that come from outside the core (L2, L3, or DRAM). These are critical for understanding whether SmallVec's larger inline storage is causing more cache misses.
3. ls_dmnd_fills: "Load/store demand fills" — only fills triggered by actual load/store instructions, excluding prefetches. This is a purer measure of cache miss pressure. The assistant's thinking is visible in the structure of the grep: they are looking for events that can distinguish between cache levels (L2 hit vs L3 hit vs DRAM) and between demand vs prefetch traffic. The local_l2, local_ccx, near_cache, far_cache, remote_cache suffixes in the output reveal a NUMA-aware hierarchy (local CCX, near/far cache, remote cache) that is characteristic of AMD's chiplet-based Zen architecture.

Assumptions and Their Consequences

Several assumptions are embedded in this debugging journey:

Assumption 1: That perf on AMD Zen4 would have similar event names to Intel. This was the initial trap. The assistant's first perf stat command used Intel-style event names, which failed. This is a common pitfall—perf abstracts many things, but raw hardware events are architecture-specific.

Assumption 2: That events listed in perf list would work directly as perf stat -e arguments. The l3_cache_accesses and l3_misses events appeared in perf list output but failed when used. This suggests these events require special syntax or kernel configuration that wasn't available.

Assumption 3: That the regression would be visible in standard cache miss counters. This is reasonable, but the difficulty in accessing those counters means the assistant may need to fall back to higher-level metrics (instructions, cycles, IPC) or use different tools entirely (like valgrind --tool=cachegrind or likwid-perfctr).

The Deeper Significance

This message is not just about finding perf event names. It represents a critical inflection point in the optimization workflow. The team has hit a wall where their usual tooling (SmallVec as a drop-in optimization) has backfired, and the standard diagnostic tools (perf) are fighting them on an unfamiliar architecture.

The regression itself is fascinating: SmallVec should reduce allocations, which should improve performance. The fact that it doesn't on Zen4 suggests something subtle about the microarchitecture—perhaps the larger inline storage (two (usize, T) tuples plus a discriminant = ~48 bytes) is causing L1 cache line pressure in a hot loop, or the branch on inline-vs-heap is mispredicting.

The assistant's response—methodically probing the perf interface, adapting to the Zen4 event naming, and searching for alternative counters—demonstrates a disciplined approach to performance debugging. Rather than guessing or reverting blindly, they are seeking empirical evidence at the hardware level.

Input and Output Knowledge

Input knowledge required to understand this message includes: familiarity with the Linux perf tool and its event syntax; awareness that AMD and Intel use different hardware event names; understanding of the SmallVec optimization and why it might cause cache pressure; knowledge of the Zen4 microarchitecture's cache hierarchy (L1, L2, L3, CCX, NUMA domains); and context about the broader optimization effort (Phase 4, the A1 regression, the synth-only microbenchmark).

Output knowledge created by this message is the set of available ls_any_fills_from_sys.* and ls_dmnd_fills_from_sys.* events on this Zen4 system. This knowledge directly enables the next step: constructing a perf stat command that uses these events to measure cache fill behavior under SmallVec vs Vec. The assistant now knows that demand fills can be tracked with ls_dmnd_fills_from_sys.all and that the source of fills (L2, local CCX, remote cache, DRAM) can be distinguished using the suffixed variants.

Conclusion

Message [msg 1063] captures a moment of debugging adaptation—the pivot from guessing Intel-style event names to discovering the Zen4-native perf events. It is a small command that encapsulates a larger story: the challenge of cross-architecture performance optimization, the importance of empirical measurement, and the disciplined process of iterating toward a working diagnostic tool.

The assistant will go on to use these events to build a proper perf stat command, collect the SmallVec vs Vec comparison data, and ultimately confirm that the regression is real. But this message is where the diagnostic path becomes clear. It is the turning point where frustration yields to discovery, and where the team gains the tool they need to understand why an optimization that should work doesn't—at least not on Zen4.