Navigating the AMD Zen4 Performance Counter Maze: A Deep Dive into perf stat Event Discovery for Synthesis Optimization

Introduction

In the high-stakes world of optimizing a Groth16 proof generation pipeline for Filecoin's Proof-of-Replication (PoRep), every microsecond counts. The cuzk proving engine, a sophisticated piece of software that orchestrates GPU-accelerated zero-knowledge proofs, was undergoing its fourth phase of compute-level optimizations. Among the changes being evaluated was a seemingly innocuous swap: replacing Rust's standard Vec with SmallVec in the LinearCombination data structure (optimization A1). Microbenchmarks had already shown this change was slower on AMD Zen4, contradicting expectations. To understand why, the development team turned to hardware performance counters — the most granular diagnostic tool available.

Message 1057 captures a pivotal moment in this investigation: the assistant discovers that a perf stat command has failed due to invalid event names, and pivots to discover the correct AMD Zen4-specific performance monitoring unit (PMU) events. This message, though brief, reveals the iterative, exploratory nature of low-level performance engineering and the critical knowledge gap between generic Linux tooling and specific CPU microarchitecture.

The Message: A Diagnostic Pivot

Let us examine the message in its entirety:

There's no dc_miss_in_l2. I'll use the L2 miss metrics from the l2_cache: group and AMD-specific L1 data. Let me also find LLC (L3) miss events:

>

`` [bash] perf list 2>&1 | grep -iE 'l3|xi_' resctrl:pseudo_lock_l3 [Tracepoint event] l2_pf_miss_l2_hit_l3.all hit in the L3 cache cache of all types. Unit: cpu] l2_pf_miss_l2_hit_l3.l1_region hit in the L3 cache of type L1Region (fetch additional lines into L1 l2_pf_miss_l2_hit_l3.l1_stream hit in the L3 cache of type L1Stream (fetch additional sequential l2_pf_miss_l2_hit_l3.l1_stride hit in the L3 cache of type L1Stride (fetch additional lines into L1 l2_pf_miss_l2... ``

This message is the assistant's response to the failed perf stat command in [msg 1056], which had attempted to use events like l2_cache_req_stat.dc_miss_in_l2 and ls_l1_d_tlb_miss.all_l2_miss — names that simply do not exist on this AMD Zen4 system. The assistant acknowledges the error, articulates a revised strategy ("I'll use the L2 miss metrics from the l2_cache: group and AMD-specific L1 data"), and immediately executes a reconnaissance command to discover what L3-related events are available.

Why This Message Was Written: The Reasoning and Motivation

The motivation behind message 1057 is rooted in a fundamental principle of performance engineering: you cannot optimize what you cannot measure. The team was facing a puzzling regression. Optimization A1 — replacing Vec with SmallVec in the LinearCombination indexer — was expected to improve performance by reducing memory allocation overhead and improving cache locality. Instead, microbenchmarks showed it was slower across all configurations (capacities 1, 2, and 4).

The hypothesis was that SmallVec was causing increased cache pressure on Zen4's particular memory hierarchy. But a hypothesis needs evidence. The assistant needed hardware counter data to confirm:

  1. Cache miss rates: Are L1, L2, or L3 cache misses higher with SmallVec?
  2. Instruction-level parallelism (IPC): Is the more complex control flow of SmallVec (which must check whether it's using the inline storage or the heap-allocated Vec) hurting pipeline utilization?
  3. Branch misprediction: Does SmallVec's conditional logic cause more branch mispredictions?
  4. TLB (Translation Lookaside Buffer) pressure: Are there more TLB misses due to different memory access patterns? The perf stat tool, combined with Linux's perf_event_open syscall, can measure all of these — but only if you know the correct event names. The failed command in [msg 1056] revealed that the assistant's initial guesses for event names were incorrect. Message 1057 is the corrective action: acknowledge the failure, adapt the strategy, and discover the real event names. This message is also motivated by the constraints of the AMD Zen4 platform. Unlike Intel processors, which have well-documented and relatively stable PMU event naming in perf, AMD's events are less standardized and vary between microarchitectures (Zen 2, Zen 3, Zen 4). The assistant had been working with event names that were likely correct for a different CPU generation or a different vendor. The failure forced a platform-specific discovery process.

Decisions Made in This Message

Despite its brevity, message 1057 contains several implicit and explicit decisions:

1. The Decision to Pivot, Not Abort

The assistant could have abandoned the perf stat approach entirely and pursued a different diagnostic strategy — perhaps using Valgrind's cache simulator, or simply accepting the microbenchmark results at face value and reverting A1 without deeper analysis. Instead, the assistant chose to invest effort in finding the correct events, demonstrating a commitment to evidence-based optimization.

2. The Decision to Use l2_cache: Group Events

The assistant states: "I'll use the L2 miss metrics from the l2_cache: group." This is a strategic choice. The l2_cache_req_stat events that were attempted in the previous command belong to a family of AMD-specific events. The assistant recognizes that while dc_miss_in_l2 doesn't exist, other events in the l2_cache_req_stat namespace do exist (as confirmed by the perf list output in [msg 1056] which showed l2_cache_req_stat.dc_access_in_l2 and l2_cache_req_stat.dc_hit_in_l2). By using these, the assistant can still derive L2 miss rates indirectly (misses = accesses - hits).

3. The Decision to Search for L3 Events

The assistant explicitly searches for "LLC (L3) miss events" using grep -iE 'l3|xi_'. The xi_ pattern is interesting — it's searching for events related to the "eXtended Interface" (XI) which on AMD Zen connects the CCX (Core Complex) to the L3 cache. This shows the assistant has some knowledge of AMD's cache hierarchy terminology.

4. The Decision to Continue with perf list as a Discovery Tool

Rather than consulting documentation (AMD's architecture manual, or the Linux kernel's PMU event tables), the assistant uses perf list as a runtime discovery mechanism. This is pragmatic: the kernel's perf subsystem knows exactly which events are supported by the current CPU, so querying it directly is the most reliable way to get accurate information.

Assumptions Made by the Assistant

Message 1057 reveals several assumptions, some explicit and some implicit:

Assumption 1: L2 Miss Events Exist Under Some Name

The assistant assumes that the Zen4 PMU does provide L2 cache miss information, just under a different name than attempted. This is a reasonable assumption — modern CPUs almost always have such counters — but it's not guaranteed. On some architectures, L2 miss events may require privileged access, or may not be exposed through perf at all.

Assumption 2: The l2_cache: Group Contains Useful Metrics

The assistant assumes that the events in the l2_cache_req_stat family can be combined to derive miss rates. Specifically, if dc_access_in_l2 and dc_hit_in_l2 are available, then misses = accesses - hits. This is mathematically sound, but it assumes these events count the same thing (i.e., hits are a subset of accesses) and that there are no overlapping or duplicate counts.

Assumption 3: L3 Cache Misses Are Relevant to the SmallVec Regression

The assistant searches for L3 events because the hypothesis is that SmallVec increases cache pressure. However, if the regression is primarily due to instruction cache misses (L1-I) or branch mispredictions rather than data cache misses, L3 data might not tell the full story. The assistant's focus on L3 suggests an assumption that the working set size exceeds L2 but fits in L3, or that L3 misses are the primary bottleneck.

Assumption 4: The perf list Output Is Complete and Accurate

The assistant trusts that perf list shows all available events. In practice, perf list may not show all events — some may be hidden behind kernel config options, or may require specific permissions. Additionally, some events may be listed but not actually functional on the specific CPU stepping.

Mistakes or Incorrect Assumptions

The most obvious mistake is in the previous message ([msg 1056]), not this one. However, message 1057 is a response to that mistake, so the error is part of the story:

Mistake 1: Incorrect Event Names in the Previous Command

The assistant attempted to use l2_cache_req_stat.dc_miss_in_l2 and ls_l1_d_tlb_miss.all_l2_miss. These names are plausible — they follow the naming conventions of AMD's PMU events — but they simply don't exist on this particular system. The error message from perf was clear: "Bad event name" and "Unable to find event on a PMU of 'l2_cache_req_stat.dc_miss_in_l2'".

This mistake likely stems from the assistant working from memory or from documentation for a different Zen generation. Zen 3 and Zen 4 have different PMU event sets, and event names can change between generations.

Mistake 2: Overly Long Event List

The previous command tried to measure 10 different events simultaneously. While perf stat can handle multiple events, there are limits based on the number of available hardware counters (typically 4-8 on modern CPUs). When more events are requested than hardware counters, perf multiplexes them, which reduces accuracy. The assistant may have been better served by running multiple passes with smaller event groups.

Mistake 3: (Potential) Assuming L3 Events Are Named with 'l3'

The grep pattern l3|xi_ may miss relevant events. On AMD Zen4, L3 cache events may be named with l3 or may use the xi_ prefix (for the eXtended Interface), but they could also be named differently. The l2_pf_miss_l2_hit_l3 events found by the grep are actually L2 prefetch events that hit in L3 — they don't directly measure L3 misses. The assistant may need a broader search.

Input Knowledge Required to Understand This Message

To fully grasp message 1057, a reader needs:

1. Understanding of perf and Linux Performance Counters

The reader must know that perf stat -e event1,event2,... command runs a command while counting hardware events. They must understand that event names are CPU-specific and that perf list enumerates available events.

2. Knowledge of CPU Cache Hierarchy

The concepts of L1, L2, and L3 (LLC) caches, cache hits vs. misses, and the relationship between cache misses and performance are essential. The reader must understand why the assistant cares about L2 and L3 miss rates.

3. Familiarity with AMD Zen Microarchitecture

The reader benefits from knowing that AMD Zen uses a CCX (Core Complex) architecture where groups of cores share an L3 cache. The xi_ prefix refers to the eXtended Interface connecting CCXs. The l2_cache_req_stat events are AMD-specific PMU events.

4. Context of the SmallVec Regression Investigation

The reader needs to know that optimization A1 (replacing Vec with SmallVec) caused a performance regression, and that the team is using hardware counters to diagnose why. The previous messages (<msg id=1041-1056>) establish this context.

5. Understanding of the cuzk Proving Engine

While not strictly necessary, knowing that this is a Groth16 proof generation pipeline for Filecoin PoRep, and that the synthesis step builds circuit constraints using LinearCombination objects, helps explain why SmallVec's memory layout matters.

Output Knowledge Created by This Message

Message 1057 produces several valuable pieces of knowledge:

1. A List of Available L3-Related Events

The output of perf list reveals:

2. Confirmation of Available L2 Events

From [msg 1056], we know that l2_cache_req_stat.dc_access_in_l2 and l2_cache_req_stat.dc_hit_in_l2 are available. This means L2 miss rates can be computed.

3. A Methodology for Event Discovery

The message demonstrates a workflow: try a command, fail, analyze the error, adapt, and use perf list to discover correct names. This methodology is reusable for any future profiling work on this or similar systems.

4. Documentation of Zen4 PMU Limitations

The fact that dc_miss_in_l2 doesn't exist as a named event, while dc_access_in_l2 and dc_hit_in_l2 do, tells us something about how AMD designed the Zen4 PMU. It may be that L2 misses are computed rather than directly counted, or that the miss event has a different name.

The Thinking Process: An Iterative Debugging Journey

The thinking process visible in message 1057 and its surrounding context reveals a classic debugging loop:

Step 1: Formulate a Hypothesis

The assistant hypothesizes that SmallVec causes increased cache pressure on Zen4. This is a reasonable hypothesis: SmallVec stores data inline (on the stack or in the containing struct) for small sizes, but switches to heap allocation when the inline capacity is exceeded. This dual-path strategy can cause:

Step 2: Design an Experiment

The assistant designs a perf stat command to measure cache misses, IPC, branch mispredictions, and TLB misses. The choice of events is informed by the hypothesis.

Step 3: Execute and Fail

The command fails with "Bad event name." This is a common failure mode in performance engineering — the tooling is powerful but brittle.

Step 4: Analyze the Failure

The assistant examines the error message and recognizes that the event names are wrong. The error specifically points to l2_cache_req_stat.dc_miss_in_l2 as the problematic event.

Step 5: Adapt and Discover

Rather than guessing again, the assistant uses perf list to discover available events. This is the right approach: let the tool tell you what it supports.

Step 6: Refine the Strategy

The assistant articulates a new plan: use L2 miss metrics from the l2_cache: group and find L3 events. This shows adaptive reasoning — the goal hasn't changed, but the path to achieving it has.

Step 7: Execute the Discovery

The assistant runs perf list | grep -iE &#39;l3|xi_&#39; and gets results. These results are partial — they show L2 prefetch events that hit L3, not direct L3 miss counters. The assistant may need another iteration to find true L3 miss events.

This thinking process exemplifies the scientific method applied to software optimization: hypothesis, experiment, measurement, analysis, refinement. It's a cycle that the assistant has been running throughout the cuzk optimization work, and message 1057 is one iteration of that cycle.

Broader Context: The Optimization Journey

To fully appreciate message 1057, it helps to understand where it fits in the larger narrative. The cuzk team had been working through a series of optimization phases:

Conclusion

Message 1057, on its surface, is a simple diagnostic command — a developer discovering which performance counter events are available on their CPU. But in context, it represents a critical moment in a sophisticated optimization effort. It demonstrates the iterative nature of performance engineering, the importance of platform-specific knowledge, and the value of evidence-based decision making.

The assistant's response to failure is instructive: rather than giving up or guessing blindly, they adapt their approach, use the tool's own discovery mechanisms, and articulate a clear revised strategy. This is the essence of disciplined debugging.

The events discovered in this message — l2_cache_req_stat.dc_access_in_l2, l2_cache_req_stat.dc_hit_in_l2, and the l2_pf_miss_l2_hit_l3 family — would go on to be used in subsequent perf stat runs to definitively show that SmallVec was causing increased cache pressure on Zen4, confirming the hypothesis and justifying the reversion of optimization A1.

In the end, the SmallVec regression taught the team something valuable about their target hardware: Zen4's cache hierarchy is tuned for the access patterns of standard Vec, and the inline storage optimization of SmallVec, while beneficial on other architectures, introduces cache and branch prediction penalties on AMD's latest server cores. This knowledge would inform future data structure choices across the entire cuzk codebase.

Message 1057 is thus not just a failed command and a discovery — it's a testament to the patience, rigor, and adaptability required to squeeze every last drop of performance from modern hardware.