The Zen4 PMU Gap: A Microcosm of Hardware-Aware Optimization

The Message

In the midst of an intensive performance optimization campaign for the cuzk Groth16 proving engine, the assistant issued a single, seemingly trivial command:

perf stat -M l3_misses -- sleep 0.1 2>&1

The response was an error:

event syntax error: '{l3_lookup_state.l3_miss/metric-id=l3_lookup_state.l3_mi..'
                      \___ Bad event or PMU

Unable to find PMU or event on a PMU of 'l3_lookup_state.l3_miss'

This message ([msg 1062]) is the culmination of a seven-message struggle to measure L3 cache misses on an AMD Zen4 processor using Linux's perf tool. On its surface, it is a failed command — a dead end. But beneath that surface lies a rich story about the challenges of hardware-level performance analysis, the assumptions we bring to cross-platform tooling, and the iterative, hypothesis-driven nature of optimization work.

Context: The Optimization Campaign

To understand why this message was written, we must step back into the broader investigation. The assistant was deep in Phase 4 of a multi-phase effort to optimize the SUPRASEAL_C2 Groth16 proof generation pipeline for Filecoin's Proof-of-Replication (PoRep) protocol. This pipeline, spanning Go, Rust, C++, and CUDA code, had a peak memory footprint of ~200 GiB and involved massive multi-scalar multiplication (MSM) and number-theoretic transform (NTT) operations on GPU, coupled with CPU-intensive R1CS constraint synthesis.

One of the optimization proposals — codenamed A1 — was to replace Vec with SmallVec (a stack-allocated, inline-capacity vector from the smallvec crate) in the LinearCombination data structure used throughout the constraint synthesis hot path. The hypothesis was that SmallVec would reduce heap allocation pressure and improve cache locality, since many linear combinations in practice contain only 1–2 terms. Previous microbenchmarks on other architectures had shown SmallVec to be beneficial for similar workloads.

However, when the assistant benchmarked this change on their AMD Zen4 system, the results were counterintuitive: SmallVec was slower than Vec in every configuration tested (inline capacities of 1, 2, and 4). This contradicted both the hypothesis and prior experience. The assistant needed to understand why.

Why This Message Was Written

The message was written as part of a systematic effort to collect hardware performance counter data — specifically cache miss rates, instructions per cycle (IPC), and branch misprediction statistics — to diagnose the SmallVec regression. The assistant had already formulated a hypothesis: perhaps Zen4's cache hierarchy reacts differently to SmallVec's inline storage, causing more L1 or L2 cache pressure than the heap-allocated Vec approach.

Across messages [msg 1055] through [msg 1061], the assistant had been probing the available PMU (Performance Monitoring Unit) events on the Zen4 system, encountering repeated syntax errors. The perf list output revealed that Zen4 uses different event names than the Intel systems the assistant was presumably more familiar with. Events like l2_cache_req_stat.dc_miss_in_l2 and l3_cache_accesses that exist on Intel simply don't exist on AMD, or have different names.

By message [msg 1062], the assistant had exhausted the straightforward event-name approach and was trying the -M (metric) flag, which allows perf stat to compute derived metrics from raw events. The l3_misses metric is a predefined metric group that should resolve to the correct PMU events for the detected architecture. This was a reasonable fallback: instead of manually specifying event names that might not exist, delegate the event resolution to perf's built-in metric tables.

Assumptions and Their Failure

The message reveals several assumptions, most of which turned out to be incorrect:

  1. That perf's metric system (-M) would work transparently across architectures. The l3_misses metric is defined in perf's metric JSON files, but those files are architecture-specific. On this Zen4 system, the metric referenced l3_lookup_state.l3_miss — a PMU event that either doesn't exist or isn't exposed by the kernel on this particular configuration. The assumption that perf would gracefully abstract over PMU differences was broken.
  2. That the kernel had the necessary PMU support enabled. The error "Unable to find PMU or event on a PMU of 'l3_lookup_state.l3_miss'" suggests the PMU itself might be present but the kernel lacks the event table entry, or the event is gated behind a kernel configuration option that wasn't enabled. The Zen4 PMU is well-supported in modern kernels (5.19+), but the system might be running an older kernel or have restricted access.
  3. That L3 cache miss data was essential for the diagnosis. The assistant was pursuing L3 metrics specifically, but the real bottleneck might have been at L1 or L2 level, or might not be cache-related at all. The fixation on L3 misses reflects a common pattern: developers gravitate toward LLC (last-level cache) metrics because they're the most familiar, even though L1/L2 behavior often matters more for tight loops.
  4. That a single ~6-second microbenchmark run would yield statistically significant counter data. The synth-only --partition 0 -i 1 subcommand runs a single iteration of synthesis for one partition, taking about 6 seconds. For stable hardware counter measurements, especially on a modern out-of-order CPU with turbo boost and frequency scaling, multiple runs are typically needed to average out noise.

Input Knowledge Required

To understand this message, a reader needs knowledge spanning several domains:

Output Knowledge Created

Even though the command failed, this message created valuable knowledge:

  1. Negative knowledge: The -M l3_misses metric is not available on this system. This eliminates one avenue of investigation and forces the assistant to find alternative approaches — perhaps using raw AMD-specific events like l2_pf_miss_l2_hit_l3 or falling back to the generic cache-misses event (which on AMD typically measures L3 misses).
  2. Architecture documentation: The error message itself documents that this Zen4 system exposes l3_lookup_state.l3_miss as a known event name (it appears in the error), but the PMU cannot resolve it. This is useful for anyone else profiling on similar hardware.
  3. Process boundary: The message marks a transition point. After this failure, the assistant would need to pivot — either to different events, a different measurement approach (like perf record with call-graph sampling), or a different diagnostic strategy entirely.

The Thinking Process

The assistant's reasoning is visible across the sequence of messages leading to [msg 1062]. There's a clear scientific method at work:

  1. Hypothesis formation ([msg 1050]): "SmallVec is slower than Vec on Zen4 — let's collect hardware counter proof (L1/L2/L3 cache misses, IPC, branch mispredicts) to confirm the Zen4 cache pressure hypothesis."
  2. Tool reconnaissance ([msg 1051]): Check that the binary exists and list available hardware events.
  3. Event discovery ([msg 1053]): Find that Zen4 exposes l2_cache_req_stat.* events but not Intel-style L1-dcache or LLC events.
  4. First attempt ([msg 1055]): Try a comprehensive event list including l2_cache_req_stat.dc_miss_in_l2 — fails because that specific event doesn't exist.
  5. Verification ([msg 1056]): List all l2_cache_req_stat.* events to confirm which are available.
  6. Second attempt ([msg 1058]): Try l3_cache_accesses and l3_misses as raw event names — fails.
  7. Syntax exploration ([msg 1059]): Check if the events exist in perf list output — they do appear, but as standalone names, not under a tracepoint group.
  8. Third attempt ([msg 1060]): Try the events directly — fails again.
  9. Tracepoint syntax ([msg 1061]): Try l3_cache:l3_cache_accesses tracepoint syntax — fails because no tracepoint exists.
  10. Metric fallback ([msg 1062]): Try -M l3_misses — fails because the underlying PMU event isn't accessible. This progression shows a methodical, narrowing search: from broad event lists to specific events to metric groups, each attempt eliminating one approach and informing the next. The assistant is not randomly trying commands — each step is a logical response to the error from the previous step.

Broader Significance

This message, for all its apparent triviality, illuminates a fundamental challenge in cross-platform performance engineering. The SmallVec optimization that worked on one architecture failed on another, and the tools to diagnose why are themselves architecture-dependent. The assistant's struggle to measure L3 cache misses on Zen4 is not a failure of competence — it is the normal experience of anyone doing low-level performance work outside the Intel/x86 mainstream.

The story also highlights a deeper truth about optimization: the measurement infrastructure is often the bottleneck before the code itself is. The assistant spent seven messages just trying to get perf stat to produce meaningful counter data. This investment in measurement capability is essential — without it, the SmallVec regression would remain a mysterious anomaly rather than a solvable problem.

In the subsequent messages ([msg 1063] onward), the assistant would pivot to a different profiling approach, ultimately discovering that the real bottleneck was not cache behavior at all, but rather the allocation patterns of temporary LinearCombination objects inside closure callbacks — a finding that led to a completely different optimization strategy (in-place add_to_lc/sub_from_lc methods) that yielded substantial improvements. The failed L3 measurement was a necessary detour on the path to that discovery.