The Elusive L3 Cache Event: A Case Study in Hardware Profiling on AMD Zen4

Message Overview

The subject message is a single tool invocation by the AI assistant — a bash command attempting to measure L3 cache performance counters using perf stat on an AMD Zen4 system:

perf stat -e 'l3_cache:l3_cache_accesses' -e 'l3_cache:l3_misses' -- sleep 0.1 2>&1

The output reveals an error: the kernel lacks the necessary tracepoint configuration (CONFIG_ setting), and the l3_cache group events are unavailable. The command itself is trivial (sleep 0.1), serving only as a test probe to verify whether the event syntax is valid.

Context and Motivation: Why This Message Was Written

This message sits within a much larger optimization campaign (Phase 4 of the cuzk proving engine) targeting the Groth16 proof generation pipeline for Filecoin's Proof-of-Replication (PoRep). The assistant and user had been systematically working through a list of micro-optimizations drawn from a prior analysis document, c2-optimization-proposal-4.md. One of those optimizations — labeled A1 — replaced Vec<(usize, T)> with SmallVec<[(usize, T); 2]> in the LinearCombination indexer, hoping to reduce heap allocation overhead by storing small vectors inline.

The problem was that A1 made synthesis slower, not faster. Microbenchmarks showed a 5–6 second regression (approximately 10%) on the Zen4 test system. The assistant's earlier analysis (messages 1049–1060) had hypothesized that the root cause was cache pressure: SmallVec's inline storage increases the struct size from 24 bytes (Vec: pointer, length, capacity) to roughly 48–64 bytes (SmallVec: inline array of 2 elements + discriminant + length). On Zen4's 32 KB L1 data cache, this larger footprint could cause more cache line evictions in the hot enforce loop, where thousands of LinearCombination objects are created and destroyed per constraint.

To confirm this hypothesis, the assistant needed hardware counter data — specifically L1 cache misses, L2 cache misses, L3 cache misses, and instruction-level metrics like IPC (instructions per cycle) and branch mispredict rates. The plan was to run perf stat on the synth-only microbenchmark twice: once with SmallVec (current code) and once after reverting to Vec, then compare the counter deltas. If the SmallVec run showed significantly more L1/L2/L3 misses, the cache-pressure hypothesis would be confirmed.

The subject message represents a critical infrastructure step in that measurement plan: verifying that the AMD-specific L3 cache events are accessible via perf. Without these counters, the comparison would lack data on last-level cache behavior, which is essential for understanding whether the regression is purely L1-local or propagates to the shared L3.

How Decisions Were Made

The assistant's decision-making process is visible across the preceding messages (1051–1060). Several key decisions led to this specific command:

  1. Choosing perf stat over other profilers: The assistant opted for Linux's perf tool because it provides precise, low-overhead hardware counter access without requiring special kernel modules or debug builds. This is the standard choice for micro-architecture analysis on Linux.
  2. Selecting specific counters: The assistant initially tried a broad set of events in message 1055: instructions,cycles,cache-references,cache-misses,branch-instructions,branch-misses,l2_cache_req_stat.dc_access_in_l2,l2_cache_req_stat.dc_hit_in_l2,l2_cache_req_stat.dc_miss_in_l2,ls_l1_d_tlb_miss.all_l2_miss. This shows an intention to capture a comprehensive profile — instruction counts, cycle counts, cache hierarchy behavior (L1 TLB, L2 data cache), and branch prediction quality.
  3. Iterating on event syntax: When the initial command failed with "Bad event name" for l2_cache_req_stat.dc_miss_in_l2, the assistant didn't give up. It ran perf list to discover available events (message 1056), found that dc_miss_in_l2 doesn't exist on this kernel/PMU, and pivoted to a different set of counters in message 1058: l2_cache_req_stat.dc_access_in_l2,l2_cache_req_stat.dc_hit_in_l2,l3_cache_accesses,l3_misses. When that also failed (this time for l3_cache_accesses), it checked again with perf list in message 1059, confirmed the events are listed, but then found in message 1060 that even a trivial test (sleep 0.1) fails with "Bad event name" for l3_cache_accesses.
  4. Trying the tracepoint syntax: The final attempt in the subject message tries prefixing the events with l3_cache: — the standard syntax for accessing tracepoint events from a specific subsystem. This is a reasonable next step when raw event names fail.

Assumptions Made

Several assumptions underpin this message and its predecessors:

Mistakes and Incorrect Assumptions

The most visible mistake is the failure to verify perf capabilities early. The assistant spent several rounds (messages 1055–1061) iterating on event syntax before discovering that the L3 tracepoint group is fundamentally unavailable. A more efficient approach would have been to run perf stat -e l3_cache_accesses sleep 0.1 as the very first test — a minimal probe that would have immediately revealed the kernel limitation.

A subtler mistake is the over-reliance on L3 counters for a hypothesis that is primarily about L1/L2 cache behavior. The SmallVec-to-Vec struct size difference (24 bytes vs ~48–64 bytes) is most impactful at the L1 level, where 32 KB of capacity means the difference between fitting ~530 Vec objects versus ~260 SmallVec objects. L3 (typically 16–32 MB on Zen4) is unlikely to be a bottleneck for this workload. The assistant could have proceeded with just the L2 counters that were available (l2_cache_req_stat.dc_access_in_l2 and l2_cache_req_stat.dc_hit_in_l2) and inferred L1 misses from the difference between total accesses and L2 hits.

There is also a tactical error in command construction: the subject message uses sleep 0.1 as the test command, which is too short to gather meaningful statistics. perf stat multiplexes events across time slices when more events are specified than hardware counters exist, and a 100 ms window may not allow all events to be sampled. A longer command (e.g., sleep 1) would have produced more reliable diagnostic output.

Input Knowledge Required

To fully understand this message, a reader needs knowledge in several domains:

  1. Linux perf and hardware counters: Understanding that perf stat -e <event> measures CPU performance monitoring unit (PMU) events, that events are organized into groups, and that some events require kernel tracepoint support (CONFIG_ settings).
  2. AMD Zen4 microarchitecture: Familiarity with the cache hierarchy (32 KB L1D, 512 KB L2, 16–32 MB L3 per CCD), the naming conventions for AMD-specific PMU events (l2_cache_req_stat.*, l3_cache:*), and the fact that Zen4 uses a distributed L3 with directory-based coherence.
  3. SmallVec vs Vec tradeoffs: Understanding that SmallVec stores elements inline up to a capacity threshold (here, 2 elements), avoiding heap allocation for small vectors but increasing the struct's stack footprint. This is a classic space-for-speed tradeoff that can backfire when working set size exceeds cache capacity.
  4. The cuzk proving engine context: Knowing that the LinearCombination type is used pervasively in the R1CS constraint synthesis loop (enforce), where thousands of temporary objects are created, populated with 1–3 terms, and destroyed per constraint. This makes the Vec/SmallVec choice a hot-path decision.
  5. The broader optimization campaign: Understanding that this perf stat attempt is part of Phase 4 Wave 1, which includes optimizations A1 (SmallVec), A4 (parallel B_G2 MSM), D4 (per-MSM window tuning), and max_num_circuits=30. The assistant is systematically evaluating each optimization's impact.

Output Knowledge Created

This message produces negative knowledge: it establishes that the L3 cache tracepoint events are unavailable on this kernel, which constrains the profiling methodology. Specifically:

The Thinking Process Visible in Reasoning Parts

The assistant's chain of reasoning is visible across the message sequence leading to this point:

  1. Problem diagnosis (messages 1049–1050): The assistant identifies that SmallVec (A1) causes a 5–6s regression and hypothesizes cache pressure as the root cause. It formulates a plan to collect perf stat data.
  2. Tool discovery (messages 1051–1054): The assistant checks whether perf is available, what events are listed, and whether the synth-only binary is built. It reads the --help output to confirm the benchmark options.
  3. First attempt (message 1055): The assistant tries a comprehensive event list but hits syntax errors for AMD-specific events. The error message reveals that l2_cache_req_stat.dc_miss_in_l2 doesn't exist.
  4. Event enumeration (message 1056): The assistant runs perf list to discover which l2_cache_req_stat.* events are actually available. It finds dc_access_in_l2 and dc_hit_in_l2 but no dc_miss_in_l2.
  5. Second attempt (message 1058): The assistant constructs a new event list using the available L2 events plus l3_cache_accesses,l3_misses. This fails for the L3 events.
  6. Verification (messages 1059–1060): The assistant checks that l3_cache_accesses appears in perf list, then tests it with sleep 0.1 to isolate the syntax issue. It fails with "Bad event name."
  7. Tracepoint syntax attempt (message 1061, the subject): The assistant tries the l3_cache: prefix syntax, which also fails — this time revealing the root cause: a missing kernel CONFIG_ setting. The reasoning is methodical and iterative: each failure narrows the search space. The assistant doesn't panic or guess randomly — it systematically tests hypotheses about event naming, availability, and syntax. However, there is a notable gap in the reasoning: the assistant never checks /sys/kernel/tracing/events/ to see which tracepoint groups are actually present, nor does it run perf list --tracepoint to filter for available events. The error message in the subject message explicitly points to the missing file path, which would have been a useful diagnostic clue earlier.

Broader Significance

This message, while seemingly a trivial failed command, is a microcosm of the challenges in performance engineering on modern hardware. The assistant is operating at the intersection of several complex systems: a custom Rust proving engine (cuzk), the Linux perf subsystem, AMD's Zen4 PMU, and the kernel's tracepoint infrastructure. Each layer has its own quirks and failure modes.

The message also illustrates an important principle in optimization work: measurement infrastructure is itself a bottleneck. Before you can optimize, you must be able to measure. And before you can measure, you must understand what your tools can and cannot do on the target system. The assistant's investment in getting perf stat working — even though it ultimately failed for L3 events — is a necessary cost of doing rigorous performance analysis.

In the larger narrative of the cuzk optimization campaign, this message marks a pivot point. The assistant will likely abandon L3-focused analysis and concentrate on the available L1/L2 counters and instruction-level metrics. The SmallVec regression will eventually be understood through other means — perhaps through the perf data that is collectible, or through the subsequent discovery (as noted in the chunk summaries) that the real bottleneck was not SmallVec at all, but temporary LinearCombination allocations inside closures. That later insight would render the cache-pressure hypothesis moot and lead to the more impactful add_to_lc/sub_from_lc optimization.

The message thus serves as a reminder that performance analysis is a journey of successive approximation, where dead ends and failed measurements are not failures but data points that refine the search.