The Data Point That Reshaped a Performance Investigation: Measuring DRAM Fills in the Synthesis Hot Path

Introduction

In the middle of an intensive optimization campaign targeting the SUPRASEAL_C2 Groth16 proof generation pipeline for Filecoin PoRep, a single perf stat command—message [msg 1119]—served as the decisive data point that reshaped the entire investigation. The message is deceptively simple: a bash invocation running a hardware performance counter on a synthesis microbenchmark, capturing memory subsystem fill events. But within its output lies the answer to a critical question that had been nagging the assistant across multiple rounds of optimization: Is the synthesis hot path memory-bound or compute-bound?

This article examines that single message in depth: why it was written, what assumptions it tested, what knowledge it produced, and how its output redirected the optimization strategy from cache-oriented prefetch tricks toward a pure instruction-throughput bottleneck.

Context: The Optimization Campaign So Far

To understand message [msg 1119], we must first understand the arc of the investigation it belongs to. The assistant had been working on Phase 4 of the cuzk proving engine—a series of compute-level optimizations intended to reduce the ~55 seconds spent on CPU synthesis and the ~34 seconds spent on GPU proving for a single 32 GiB Filecoin PoRep proof. Earlier rounds had implemented and then reverted several optimizations (SmallVec for the LC Indexer, pre-sizing for ProvingAssignment, cudaHostRegister for pinned memory) after discovering regressions. The final Phase 4 configuration retained only two changes: parallelized B_G2 MSMs (A4) and per-MSM window tuning (D4).

The E2E benchmark in [msg 1098] had delivered disappointing news: the Phase 4 configuration was 4.8% slower than the Phase 2/3 baseline (93.2s vs 88.9s). Synthesis had regressed from 54.7s to 55.8s, and GPU time had jumped from 34.0s to 37.2s. The assistant had spent several messages investigating whether the GPU regression was caused by the D4 split-MSM objects or by the parallel B_G2 thread pool startup, but the user redirected the investigation in [msg 1112]: instead of chasing the GPU regression, the user wanted a deep analysis of prefetch, cache, and instruction-level parallelism (ILP) optimization opportunities in the synthesis hot path.

This set the stage for message [msg 1119]. The assistant had already collected some perf stat data in [msg 1118]—measuring instructions, cycles, L1 cache loads/misses, and dTLB misses. That data showed an IPC of 2.60 (modest for a Zen4 core capable of ~6 IPC) and an L1D miss rate of 2.0%. But the assistant needed one more piece of information to complete the picture: how deep into the memory hierarchy did the misses go?

The Message Itself: A Targeted Measurement

Message [msg 1119] is a single bash command followed by its output:

perf stat -e instructions,cycles,ls_dmnd_fills_from_sys.all,\
  ls_dmnd_fills_from_sys.local_l2,ls_dmnd_fills_from_sys.local_ccx,\
  ls_dmnd_fills_from_sys.dram_io_near \
  -- env FIL_PROOFS_PARAMETER_CACHE=/data/zk/params \
  /home/theuser/curio/extern/cuzk/target/release/cuzk-bench synth-only \
  --c1 /data/32gbench/c1.json --partition 0 -i 1 \
  2>&1 | grep -E 'synth=|instructions|cycles|ls_dmnd|time elapsed'

The output shows:

  [1] wall=55.6s  synth=55.4s  circuits=1  constraints=130278869
   593,855,205,435      instructions:u                              (83.35%)
   229,115,504,565      cycles:u                                    (83.34%)
     2,382,738,419      ls_dmnd_fills_from_sys.all:u                (83.35%)
     2,269,321,489      ls_dmnd_fills_from_sys.local_l2:u            (83.35%)
       109,435...

(The output is truncated in the conversation data, but the full results are analyzed in the subsequent message [msg 1120].)

The key counters here are the ls_dmnd_fills_from_sys family—AMD Zen4-specific performance events that track demand (non-prefetch) cache line fills from the memory system. They break down by source: fills that came from the local L2 cache, fills that came from the local CCX (the L3 cache within the same core complex), and fills that came from DRAM (near memory).

Why This Specific Measurement Was Needed

The assistant already knew from [msg 1118] that the synthesis workload executed 247 billion L1D loads with 4.86 billion misses—a 2.0% miss rate. But that number alone is ambiguous. A 2.0% L1 miss rate could mean either:

  1. Most misses hit L2 or L3, in which case the penalty is modest (~12 cycles for L2, ~40 cycles for L3 on Zen4), and the workload is primarily compute-bound.
  2. Many misses go to DRAM, in which case the penalty is severe (~200+ cycles), and the workload is memory-latency-bound. These two scenarios demand completely different optimization strategies. If the workload is DRAM-bound, software prefetch and memory-level parallelism (MLP) tricks would be the primary lever. If the workload is compute-bound with good cache behavior, then the optimization focus should shift to reducing instruction count, improving IPC, and eliminating allocation overhead. The assistant needed to distinguish these cases. The ls_dmnd_fills_from_sys counters provide exactly that distinction: they reveal the source of each cache line that was brought in to satisfy a demand load.

The Knowledge Produced: A Surprising Result

The output, when fully interpreted in [msg 1120], revealed something remarkable:

| Fill Source | Count | Per Constraint | Interpretation | |---|---|---|---| | Total fills | 2.38B | 18.3/constraint | Every cache line fill | | From L2 | 2.27B | 17.4/constraint | 95.2% of fills hit L2 | | From L3/CCX | 109M | 0.84/constraint | 4.6% of fills hit L3 | | From DRAM | 2.8M | 0.02/constraint | 0.12% of fills go to DRAM |

The DRAM fill count—approximately 2.8 million across 130 million constraints—is negligible. For every 50 constraints processed, only a single cache line is fetched from DRAM. The L3 fill rate is also low: less than one per constraint. The vast majority of L1 misses (95%) are satisfied by the L2 cache.

This is a profoundly important result. It means the synthesis hot path is not memory-latency-bound. The data working set—primarily the aux_assignment array of ~13 million BLS12-381 field elements (416 MB per partition)—is accessed with sufficient spatial locality that the hardware prefetcher successfully brings most data into L2 before it is needed. The L2 cache on Zen4 is 1 MiB per core, which is large enough to hold a significant window of the assignment array.

The implication for optimization strategy is decisive: software prefetch would provide minimal benefit. The hardware prefetcher on Zen4 is already doing an excellent job. Adding _mm_prefetch intrinsics to the eval loop (as the assistant had been considering) would at best save a few percent, and at worst add instruction overhead that degrades IPC.

The Assumptions That Were Tested and Refuted

Message [msg 1119] tested several implicit assumptions that had been carried forward from earlier in the investigation:

Assumption 1: The synthesis workload is memory-latency-bound. This was a natural hypothesis given the 416 MB working set and the random-access pattern of the aux_assignment lookups in the eval loop. Many performance engineers would assume that a 416 MB data structure accessed randomly would generate frequent DRAM misses. The data proved this assumption wrong: the access pattern is not truly random—the LC terms tend to reference indices that are close together (a property the last_inserted optimization in the Indexer exploits), and the hardware prefetcher captures this strided pattern effectively.

Assumption 2: Software prefetch would yield a 2-5% gain. The assistant had estimated this in the subsequent analysis, but the DRAM fill data made it clear that prefetch would be attacking a problem that barely exists. With only 0.02 DRAM fills per constraint, there is almost no DRAM latency to hide. The small gain from prefetch would come from reducing L2→L1 latency, which is already only ~12 cycles—a tiny fraction of the ~423ns per constraint.

Assumption 3: The 34% allocation overhead identified earlier was the dominant bottleneck. This assumption was actually confirmed by the DRAM fill data, not refuted. The low DRAM fill rate means the bottleneck must be elsewhere—and the earlier perf stat data showing 594 billion instructions for 130 million constraints (4,560 instructions per constraint) pointed squarely at instruction count. The allocation/deallocation operations from jemalloc were consuming a huge fraction of those instructions.

The Thinking Process Visible in the Message Sequence

The assistant's reasoning is visible in the progression from [msg 1118] to [msg 1119] to [msg 1120]. In [msg 1118], the assistant ran a perf stat focused on L1 cache and TLB metrics. The output showed an IPC of 2.60 and an L1D miss rate of 2.0%, but left the deeper memory hierarchy unexplored. The assistant then, in [msg 1119], deliberately chose a different set of counters to probe exactly one level deeper: the ls_dmnd_fills_from_sys family.

The choice of counters reveals the assistant's mental model. Rather than running a generic perf stat with dozens of counters (which would increase measurement overhead and risk multiplexing skew), the assistant selected a targeted set of six events: instructions, cycles, and four fill-source breakdown counters. This is the mark of an experienced performance engineer who knows exactly which question to ask next.

The timing of the measurement is also significant. The assistant could have run this perf stat earlier—before implementing the Vec recycling pool, before the interleaved eval experiment, before any of the Phase 4 changes. But the assistant chose to gather this data after those experiments, in response to the user's specific request for prefetch/ILP analysis. The measurement was demand-driven, not speculative.

Input Knowledge Required to Interpret This Message

Understanding message [msg 1119] requires substantial domain knowledge:

  1. AMD Zen4 microarchitecture: The ls_dmnd_fills_from_sys counters are specific to AMD's Zen family. An Intel engineer would need to find equivalent MEM_LOAD_RETIRED events. The distinction between local_l2, local_ccx, and dram_io_near reflects the Zen4 cache hierarchy: each core has a private L1 (32 KB data) and L2 (1 MB), while L3 is shared within a CCX (core complex) of 8 cores.
  2. The synthesis hot path: The reader must know that synth-only runs the circuit synthesis for a single partition of a 32 GiB PoRep proof, processing ~130 million constraints. Each constraint involves building three LinearCombination objects (A, B, C) and evaluating them against the assignment.
  3. The perf stat tool: The -e flag specifies raw event names (not the usual perf event aliases like cache-misses). The 2>&1 redirection and grep filtering show the assistant extracting specific lines from mixed stdout/stderr output.
  4. The optimization context: The reader needs to know that this measurement was taken after the user explicitly asked for prefetch/ILP analysis (in the question at [msg 1112]), and that the assistant had already implemented and reverted several optimizations in the preceding rounds.

Output Knowledge Created

This message produced several distinct pieces of knowledge:

  1. Quantitative confirmation that DRAM is not the bottleneck: With only 2.8M DRAM fills across 130M constraints, the synthesis workload's memory access pattern is almost entirely serviced by L2 cache. This definitively rules out memory latency as a primary concern.
  2. Validation of the hardware prefetcher's effectiveness: The 95.2% L2 hit rate for L1 misses indicates that Zen4's hardware prefetcher is successfully predicting the access pattern of the LC eval loop. This is a non-trivial finding—many workloads with seemingly random access patterns defeat the prefetcher.
  3. Redirection of optimization strategy: The data implies that software prefetch (which the user had specifically asked about) is not worth pursuing. The assistant's subsequent analysis in [msg 1120] correctly downgrades prefetch to a "small win" (2-5%) and elevates the arena allocator to the top priority (15-25%).
  4. A baseline for future comparison: If subsequent optimizations change the memory access pattern (e.g., by batching constraints or restructuring the LC data), this DRAM fill measurement provides a baseline to detect regressions.

Mistakes and Incorrect Assumptions

The message itself is a measurement, so it contains no mistakes per se. However, the interpretation of the data in the subsequent message reveals a subtle point: the assistant estimated that software prefetch could yield 2-5%, but this estimate may still be too optimistic. With only 0.02 DRAM fills per constraint, the maximum latency that prefetch could hide is the L2→L1 latency (~12 cycles), which represents only ~3% of the per-constraint cycle budget (1,750 cycles). The upper bound for prefetch gain is therefore ~3%, and the actual gain after accounting for prefetch instruction overhead would likely be lower.

The assistant also assumed that the "arena allocator" optimization would yield 15-25% by eliminating jemalloc overhead. This assumption was later tested in the next chunk ([chunk 14.1]), where perf profiling revealed that the arena allocator (Vec recycling pool) had minimal effect because the real allocation bottleneck was temporary LinearCombination objects created inside closures via Boolean::lc()—allocations that the recycling pool didn't address. The DRAM fill data was correct, but the assistant's hypothesis about which allocations were the problem was refined only after further profiling.

Conclusion

Message [msg 1119] is a masterclass in targeted performance measurement. In a single command, the assistant gathered the data needed to distinguish between two fundamentally different optimization paths. The output—showing negligible DRAM traffic—redirected the investigation away from cache-oriented prefetch tricks and toward instruction-count reduction. The message demonstrates that the most valuable performance insights often come not from complex instrumentation frameworks, but from asking the right question and selecting the right hardware counter to answer it.

The DRAM fill measurement stands as a permanent record: for this workload, on this architecture, the hardware prefetcher is doing its job. The bottleneck lies elsewhere—in the 4,560 instructions per constraint, the 6 alloc/dealloc pairs per constraint, and the IPC of 2.60 that leaves headroom for improvement. The path forward was clear: stop trying to feed the CPU data faster, and start trying to make the CPU do less work.