The Perf Event That Wasn't: A Microcosm of Performance Analysis on AMD Zen4

Introduction

In the high-stakes world of zero-knowledge proof generation, every millisecond counts. When a team working on the cuzk SNARK proving engine for Filecoin's proof-of-replication (PoRep) pipeline discovered that a seemingly obvious optimization — replacing Vec with SmallVec in the constraint system's LinearCombination data structure — was actually slowing down CPU synthesis on their AMD Zen4 hardware, they embarked on a meticulous investigation. Message 1058 captures a single, seemingly trivial moment in that investigation: an attempt to run perf stat that fails with an event syntax error. But this message is a rich microcosm of the entire debugging process, revealing the assumptions, reasoning, and challenges involved in modern performance analysis.

The Message: A Failed Perf Command

The message consists of a single bash command invocation and its error output:

[assistant] Good. Now I have the right events. Let me do two `perf stat` runs per config (SmallVec cap=2 and Vec). I'll use a comprehensive set of counters. Since AMD limits multiplexing groups, I'll do two passes per configuration to cover all events cleanly.
[bash] perf stat -e instructions,cycles,cache-references,cache-misses,branch-instructions,branch-misses,l2_cache_req_stat.dc_access_in_l2,l2_cache_req_stat.dc_hit_in_l2,l3_cache_accesses,l3_misses -- env FIL_PROOFS_PARAMETER_CACHE=/data/zk/params /home/theuser/curio/extern/cuzk/target/release/cuzk-bench synth-only --c1 /data/32gbench/c1.json --partition 0 -i 1 2>&1 | tee /tmp/perf-smallvec-cap2-run1.txt
event syntax error: '.._hit_in_l2,l3_cache_accesses,l3_misses'
                                  \___ Bad event name

Unable to find event on a PMU of 'l3_cache_accesses'
Run 'perf list' for a list of valid events

At first glance, this is just a user fumbling with perf event names. But the message is far more interesting when we understand what led to it and what it reveals about the state of the investigation.

Context and Reasoning: Why This Command Was Written

To understand this message, we need to trace the narrative that produced it. The team had implemented four Phase 4 compute-level optimizations for the cuzk proving engine. One of them, optimization A1, replaced Vec<(usize, T)> with SmallVec<[(usize, T); 2]> in the LinearCombination data structure within bellpepper-core/src/lc.rs. The intuition was sound: SmallVec stores up to 2 elements inline (on the stack) without heap allocation, which should reduce malloc/free overhead in the hot enforce loop where thousands of LinearCombination objects are created and destroyed per second.

However, microbenchmarks using a synth-only subcommand (which runs CPU synthesis without GPU involvement) showed that SmallVec was actually slower than plain Vec on Zen4 — across all tested inline capacities (1, 2, 4). The assistant hypothesized that the cause was cache pressure: SmallVec's inline storage makes each LinearCombination struct larger (from 24 bytes for Vec to ~56 bytes for SmallVec<[2]>), and in a tight loop processing millions of constraint terms, the larger struct size causes more L1/L2 cache misses, overwhelming any savings from reduced heap allocation.

The assistant's plan, articulated in message 1050, was to collect perf stat data comparing Vec vs SmallVec cap=2 using hardware performance counters. The goal was to obtain empirical proof — cache miss rates, instructions per cycle (IPC), branch mispredictions — to confirm or refute the cache pressure hypothesis. Message 1058 is the first concrete step in executing that plan.

The Assumptions Embedded in This Command

The command reveals several assumptions the assistant was making:

Assumption 1: The event names are correct. The assistant had previously run perf list (message 1057) and seen entries for l3_cache_accesses and l3_misses in the output. The assumption was that these were valid event names that could be passed directly to perf stat -e. This is a common pitfall: perf list shows event names, but not all of them are usable as raw -e arguments — some require specific syntax (e.g., l3_cache:l3_cache_accesses or metric groups via -M).

Assumption 2: The events are available on this kernel/P MU configuration. Even though perf list showed them, the kernel might not have the necessary tracepoint infrastructure compiled in. The error message hints at this: "Perhaps this kernel misses some CONFIG_ setting to enable this feature?"

Assumption 3: AMD Zen4 supports these specific L3 events. The assistant was working on an AMD system (Zen4, given the l2_cache_req_stat.* events that are AMD-specific). Intel and AMD have different PMU (Performance Monitoring Unit) event sets. The assistant had correctly identified AMD-specific L2 events (l2_cache_req_stat.dc_access_in_l2, l2_cache_req_stat.dc_hit_in_l2) but was less certain about L3 events.

Assumption 4: A single perf stat invocation can cover all desired counters. The assistant noted that "AMD limits multiplexing groups" and planned "two passes per configuration to cover all events cleanly." This shows awareness of the limitation that not all counters can be collected simultaneously due to PMU register constraints.

The Mistakes and Incorrect Assumptions

The primary mistake in this message is the incorrect event name syntax. The assistant assumed that l3_cache_accesses and l3_misses were valid standalone event names, when in fact they require a different syntax on this system. The error message is unambiguous: "Unable to find event on a PMU of 'l3_cache_accesses'."

However, this mistake is itself informative. It reveals the gap between what perf list displays and what perf stat -e accepts. On some systems, these events work as shown; on others, they need a prefix like l3_cache: or amd_l3:. The assistant's subsequent exploration (messages 1060–1064) shows them trying various syntaxes (-e 'l3_cache:l3_cache_accesses', -M l3_misses) before settling on alternative events like ls_dmnd_fills_from_sys.* that actually work.

A more subtle incorrect assumption was that the L3 cache events were the most important counters for diagnosing the SmallVec regression. In reality, the regression was likely caused by L1 cache pressure (due to larger struct size in the innermost loop), not L3 misses. The assistant's focus on L3 events may have been a slight misdirection, though L3 misses would still be a useful secondary indicator.

Input Knowledge Required

To fully understand this message, one needs:

  1. Knowledge of the cuzk project: That it's a SNARK proving engine for Filecoin PoRep, that Phase 4 is about compute-level optimizations, and that A1 (SmallVec) is one of the proposed changes.
  2. Knowledge of the SmallVec vs Vec tradeoff: That SmallVec stores elements inline up to a capacity, avoiding heap allocation for small collections, but increases struct size. The tradeoff is between allocation overhead and cache footprint.
  3. Knowledge of perf and PMU events: That perf stat -e accepts hardware event names, that different CPU architectures have different events, and that AMD Zen4 uses specific event names like l2_cache_req_stat.* rather than Intel's L1-dcache-load-misses.
  4. Knowledge of the synth-only microbenchmark: That cuzk-bench synth-only --partition 0 -i 1 runs a single iteration of CPU synthesis for partition 0, taking ~55–57 seconds, and is used for A/B testing bellpepper-core/bellperson changes without GPU involvement.
  5. Knowledge of the investigation state: That previous microbenchmarks had already shown SmallVec was slower, and the assistant was now seeking hardware counter evidence to explain why.

Output Knowledge Created

This message, despite its failure, creates several important outputs:

  1. Negative knowledge: The assistant now knows that l3_cache_accesses and l3_misses are not valid raw event names on this system. This is valuable — it prevents wasting time on non-functional event names in subsequent runs.
  2. A concrete error trace: The error message "Unable to find event on a PMU of 'l3_cache_accesses'" provides a searchable clue. The assistant can now investigate the correct syntax or alternative events.
  3. Evidence of the assistant's methodology: The message shows a systematic approach: plan the experiment, identify the counters, run the command, handle errors, and iterate. This is visible in the surrounding messages where the assistant progressively refines the event list.
  4. Documentation of the investigation: The tee /tmp/perf-smallvec-cap2-run1.txt creates a log file. Even though the command failed, the file would contain the error output, preserving the attempt for later reference.

The Thinking Process Visible in the Message

The assistant's reasoning is laid out in the text preceding the bash command:

"Good. Now I have the right events. Let me do two perf stat runs per config (SmallVec cap=2 and Vec). I'll use a comprehensive set of counters. Since AMD limits multiplexing groups, I'll do two passes per configuration to cover all events cleanly."

This reveals several layers of thinking:

  1. Confidence: The assistant believes they have identified the correct events after the earlier exploration (messages 1055–1057). The word "Good" indicates a sense of progress.
  2. Experimental design: The plan calls for two runs per configuration (SmallVec cap=2 and Vec), which is good scientific practice — multiple samples to check consistency. The two-pass approach (splitting counters across two invocations) shows awareness of PMU multiplexing limitations.
  3. Comprehensiveness: The assistant wants a "comprehensive set of counters" — not just basic metrics like instructions and cycles, but also cache hierarchy events (L2 accesses, L2 hits, L3 accesses, L3 misses) to build a complete picture of memory access patterns.
  4. Tooling awareness: The mention of "AMD limits multiplexing groups" shows the assistant understands the hardware constraints of the performance monitoring unit. AMD Zen4 CPUs have a limited number of PMU counters that can be used simultaneously; when more events are requested than available counters, the kernel multiplexes them, reducing accuracy. The failure itself triggers a new thinking process visible in subsequent messages: the assistant tries different syntaxes (l3_cache:l3_cache_accesses, -M l3_misses), discovers that ls_dmnd_fills_from_sys.* events work, and eventually runs a successful perf stat with alternative counters (message 1064). This iterative refinement is characteristic of performance analysis work.

Broader Significance

This message, while seemingly a minor failure, is emblematic of the entire optimization investigation. The team was operating at the frontier of performance engineering, where textbook optimizations (like SmallVec) can backfire due to subtle hardware interactions. The cache pressure hypothesis was plausible but required empirical validation. The assistant's systematic approach — plan, measure, fail, learn, iterate — is the only reliable way to navigate such terrain.

The message also highlights the gap between generic optimization advice and hardware-specific reality. SmallVec is widely recommended in Rust circles for reducing allocation overhead. But on AMD Zen4, with its particular cache hierarchy (32KB L1 data cache per core, 512KB L2, 32MB L3 shared across cores), the increased struct size from inline storage can cause more cache misses than the allocation savings justify. The assistant's investigation was essentially a confrontation between conventional wisdom and empirical data.

Conclusion

Message 1058 captures a single failed perf stat command, but it is a window into a much larger story. It shows a performance engineer systematically gathering evidence to understand a counterintuitive regression, navigating the quirks of AMD's PMU event naming, and documenting each step of the investigation. The "right events" the assistant thought they had turned out to be wrong, but the process of discovering that — the error, the iteration, the eventual success — is the real story. In performance analysis, as in science, the path from hypothesis to understanding is paved with failed commands and corrected assumptions.