The Art of the Minimal Test: Debugging perf Event Names on AMD Zen4

In the midst of an intensive optimization campaign targeting the SUPRASEAL_C2 Groth16 proof generation pipeline for Filecoin PoRep, a single bash command appears that, at first glance, seems almost trivial:

[assistant] [bash] perf stat -e l3_cache_accesses -e l3_misses -- sleep 0.1 2>&1
event syntax error: 'l3_cache_accesses'
                     \___ Bad event name

Unable to find event on a PMU of 'l3_cache_accesses'
Run 'perf list' for a list of valid events

 Usage: perf stat [<options>] [<command>]

    -e, --event <event>   event selector. use 'perf list' to list available events

A failed perf stat invocation targeting a sleep 0.1 command. Yet this tiny failure is a microcosm of the systematic, hypothesis-driven methodology that characterizes the entire optimization effort. To understand why this message exists, we must trace the reasoning chain that led to it, the assumptions it tests, and what its failure reveals about the gap between documentation and reality on modern AMD hardware.

The Optimization Context

The message belongs to Segment 14 of a deep-dive investigation into the cuzk proving engine, specifically Phase 4 of a multi-phase optimization project. The team had already achieved substantial throughput improvements through architectural changes — Phase 2's async overlap pipeline between synthesis and GPU proving delivered a 1.27× speedup, and Phase 3's cross-sector batching pushed that to 1.46×. Phase 4 aimed for compute-level micro-optimizations within the CPU synthesis hotpath.

Among the candidate optimizations was A1: replacing Vec&lt;(usize, T)&gt; with SmallVec&lt;[(usize, T); 2]&gt; in the LinearCombination indexer, a change intended to reduce heap allocation pressure by storing small vectors inline. The hypothesis was sound on paper — SmallVec avoids heap allocation for small collections, and most LinearCombination objects in the Groth16 synthesis have only one or two terms. However, microbenchmarks on the AMD Zen4 target machine showed that SmallVec was actually slower than plain Vec in every configuration tested.

This counterintuitive result demanded explanation. The team hypothesized that SmallVec's inline storage was increasing cache pressure — the larger structure size (due to the inline array capacity) might be causing more L1/L2 cache misses, or the additional branching in SmallVec's access paths might be hurting instruction-level parallelism. To test this, they needed hardware performance counter data.

The Perf Event Discovery Chain

The subject message is the fourth attempt in a rapid sequence of perf event discovery. In [msg 1055], the assistant attempted a compound event string with l2_cache_req_stat.dc_miss_in_l2 and ls_l1_d_tlb_miss.all_l2_miss, only to discover that dc_miss_in_l2 is not a valid event name on this system. In [msg 1056], they verified which l2_cache_req_stat.* events actually exist, finding dc_access_in_l2 and dc_hit_in_l2 but no dc_miss_in_l2. In [msg 1057], they searched for L3 cache events and found l3_cache_accesses and l3_misses listed in perf list output. In [msg 1058], they tried combining these with other events and got a syntax error — but the error message blamed the compound syntax, not the individual event names.

This brings us to the subject message. The assistant now makes a critical methodological choice: instead of continuing to debug compound event strings against the expensive synthesis benchmark (~6 seconds per run), they isolate the problem by testing the individual events against the simplest possible target: sleep 0.1. This is a textbook debugging technique — reduce the variable space to its minimum before scaling up.

The Assumption and Its Failure

The assumption embedded in this command is clear: because l3_cache_accesses and l3_misses appeared in perf list output ([msg 1059]), they should be valid event names for perf stat -e. The perf list command is the canonical way to discover available events on a Linux system. If an event appears there, it should be usable.

The failure reveals a subtle truth about AMD Zen4's PMU (Performance Monitoring Unit) event encoding. The events l3_cache_accesses and l3_misses are listed by perf list, but they cannot be used directly with perf stat -e. This could be because:

  1. They require a different syntax: On AMD systems, some events need to be specified with a amd_l3_ prefix or through the amd_l3 PMU domain explicitly.
  2. They are raw event codes: The listed names might be human-readable aliases for raw hex event codes that must be specified differently.
  3. They belong to a different event group: AMD Zen4 organizes PMU events into groups (e.g., l2_cache_req_stat.* works, but l3_cache_* might need a different PMU selector).
  4. They are uncore events: L3 cache events on AMD Zen4 are "uncore" events that may require different permissions or a different collection mechanism than core events. The error message "Unable to find event on a PMU of 'l3_cache_accesses'" is particularly telling — it suggests perf is looking for a PMU (Performance Monitoring Unit) named l3_cache_accesses rather than finding an event within a known PMU. This points toward a naming convention mismatch: the event might need to be qualified as amd_l3/l3_cache_accesses or similar.

The Broader Methodology

What makes this message significant beyond its immediate failure is what it represents about the optimization methodology at work. The assistant is not throwing random event names at the wall. There is a clear, structured process:

  1. Hypothesis formation: SmallVec causes cache pressure → we need cache miss counters.
  2. Event discovery: Use perf list to find candidate events.
  3. Isolation testing: Test individual events against a trivial workload before running the real benchmark.
  4. Incremental complexity: Start with sleep 0.1, then scale to the synthesis benchmark once events are confirmed working. This mirrors the broader Phase 4 methodology: each optimization is tested in isolation, regressions are reverted, and the root cause is chased down through increasingly precise measurements. The same pattern appears in the SmallVec investigation itself — when the microbenchmark showed a slowdown, the team didn't just revert and move on. They dug deeper, seeking hardware-level proof of the cache pressure hypothesis.

What This Message Creates

The output knowledge from this message is negative but valuable: l3_cache_accesses and l3_misses are not directly usable as perf stat -e arguments on this AMD Zen4 system, despite appearing in perf list. This negative result forces a pivot — either find the correct syntax for L3 events, or fall back to L2 events (l2_cache_req_stat.dc_access_in_l2, l2_cache_req_stat.dc_hit_in_l2) and the generic cache-misses event (which measures last-level cache misses on this architecture).

The input knowledge required to understand this message is substantial: one must know that perf stat is being used to collect hardware counter data, that l3_cache_accesses and l3_misses were previously identified as candidate events, that the synthesis benchmark is too expensive to use for event name debugging, and that the ultimate goal is to compare SmallVec vs Vec cache behavior on AMD Zen4.

Conclusion

A single failed perf stat command against sleep 0.1 might seem like a trivial moment in a coding session. But it encapsulates a rigorous engineering methodology: form a hypothesis, discover the tools needed to test it, isolate variables, and verify each component before assembling the full experiment. The failure is not a setback — it is data. And in the high-stakes world of optimizing a ~200 GiB memory, ~90-second proof generation pipeline for Filecoin, every piece of data, even a negative one, moves the project forward.