The Rigor of Measurement: Validating Boolean::add_to_lc with perf stat

In the high-stakes world of Filecoin proof generation, where a single 32 GiB PoRep C2 proof consumes ~200 GiB of memory and takes over a minute to produce, every millisecond counts. The optimization journey documented across the cuzk proving engine's Phase 4 is a masterclass in disciplined performance engineering — and message [msg 1209] captures a pivotal moment: the rigorous validation of a critical optimization through hardware performance counter analysis.

The Message

The message is deceptively simple — a single bash command execution:

FIL_PROOFS_PARAMETER_CACHE=/data/zk/params perf stat -e cycles,instructions,cache-references,cache-misses,L1-dcache-loads,L1-dcache-load-misses,LLC-loads,LLC-load-misses,branches,branch-misses,task-clock -- extern/cuzk/target/release/cuzk-bench synth-only --c1 /data/32gbench/c1.json --partition 0 -i 1 2>&1 | tee /tmp/perf-add-to-lc-run1.txt

The output shows the benchmark completing successfully with a synthesis time of 51.22 seconds, with the full perf stat hardware counter data saved to /tmp/perf-add-to-lc-run1.txt for later analysis.

Why This Message Was Written

To understand why this message exists, we must trace the optimization arc that led here. The assistant had been deep in Phase 4 of the cuzk project — a systematic campaign to optimize the CPU-side synthesis phase of Groth16 proof generation. Earlier profiling with perf record had produced a detailed breakdown of the ~55-second synthesis time, revealing that LinearCombination construction accounted for 23.52% of execution time (approximately 12.9 seconds), with Boolean::lc() alone consuming 6.51%.

The root cause was identified: every call to Boolean::lc() created a fresh LinearCombination::zero() — a heap allocation — even when wrapping just 1-2 scalar terms. In tight loops like UInt32::addmany, this method was called 32 times per operand, generating a storm of tiny, short-lived allocations.

The fix was elegant: add Boolean::add_to_lc() and Boolean::sub_from_lc() methods that directly add or subtract a boolean term into an existing LinearCombination, bypassing the temporary allocation entirely. The assistant had implemented this optimization across six call sites in bellpepper-core and bellperson, patching UInt32::addmany, Num::add_bool_with_coeff, Boolean::enforce_equal, Boolean::sha256_ch, Boolean::sha256_maj, and the lookup table gadgets.

In the immediately preceding message ([msg 1207]), the assistant ran a three-iteration synth-only microbenchmark and observed synthesis time drop from 55.4s to 50.9s — an 8.1% improvement. But a single timing number, however promising, is not enough. The user had explicitly requested rigorous measurement after a previous regression (the SmallVec A1 optimization that hurt performance despite reducing cache misses). The lesson was seared in: measure twice, optimize once, and always understand why at the hardware level.

Message [msg 1209] is the direct consequence of that lesson. It is the assistant executing the second half of the validation protocol: collecting hardware performance counter data via perf stat to enable a detailed A/B comparison against the Vec baseline.

How Decisions Were Made

The choice of perf stat events reveals a sophisticated understanding of what matters on the Zen4 microarchitecture. The assistant selected:

• cycles, instructions, task-clock: For IPC (instructions per cycle) analysis — the single most important metric for understanding whether an optimization improves CPU utilization or just shifts bottlenecks.
• cache-references, cache-misses, L1-dcache-loads, L1-dcache-load-misses, LLC-loads, LLC-load-misses: To characterize the memory hierarchy behavior. This was especially relevant because the earlier SmallVec experiment had reduced cache misses but lowered IPC — a counterintuitive result that taught the team that simpler instruction streams can outperform "cache-friendly" data structures on Zen4's powerful out-of-order engine.
• branches, branch-misses: To detect whether the optimization changed control flow patterns, which could affect branch predictor behavior. The decision to run a single iteration (-i 1) rather than three is also deliberate. perf stat itself adds measurement overhead, and the goal here is not to get a precise timing average but to capture hardware counter data under realistic conditions. The timing from the three-iteration run ([msg 1207]) already established the average; this run is purely for the counter profile. The use of tee to save output to /tmp/perf-add-to-lc-run1.txt follows an established pattern from earlier experiments — the Vec baseline data was saved to /tmp/perf-vec-run1.txt and /tmp/perf-vec-run2.txt, and the SmallVec data to /tmp/perf-smallvec-cap2-run1.txt. This systematic file naming enables easy comparison later.

Assumptions Made

The assistant makes several implicit assumptions in this message:

1. Reproducibility: That a single-iteration perf stat run will produce representative counter data comparable to the earlier Vec baseline runs. Given the deterministic nature of synthesis (same C1 input, same circuit), this is reasonable.
2. Measurement validity: That perf stat on this Zen4 system accurately captures the selected hardware events without significant interference from other processes. The benchmark machine is a dedicated 96-core Threadripper PRO 7995WX, likely running minimal background load.
3. The optimization is correct: The assistant assumes the Boolean::add_to_lc patch is functionally correct — that the optimized code produces identical circuit constraints to the original. This was implicitly validated by the successful benchmark run (no assertion failures, no crashes).
4. The perf stat events selected are sufficient: The chosen set of counters (cycles, instructions, cache misses, branches) covers the most relevant dimensions for understanding the optimization's impact, but it's not exhaustive — there's no measurement of TLB misses, for instance, or front-end stall cycles.

Input Knowledge Required

To fully understand this message, a reader needs:

1. Knowledge of the cuzk project architecture: That synthesis is the CPU-intensive phase of Groth16 proof generation where the circuit constraints are evaluated to produce the a, b, and c vectors. This phase runs entirely on the CPU and is the current bottleneck.
2. Understanding of the Boolean::lc() allocation problem: That Boolean::lc() creates a temporary LinearCombination (heap-allocated Vec) on every call, and that in tight gadget loops this generates millions of short-lived allocations that pressure the allocator and pollute cache.
3. Familiarity with perf stat and hardware performance counters: What each event measures and why it matters. For example, L1-dcache-load-misses indicates data not found in L1 cache, requiring a fetch from L2/L3/DRAM; branch-misses indicates mispredictions that flush the pipeline.
4. The history of earlier optimizations: Particularly the SmallVec A1 experiment where cache misses decreased but IPC dropped, teaching that simpler code often outperforms "optimized" data structures on modern CPUs.
5. The Zen4 microarchitecture characteristics: That Zen4 has a large, powerful out-of-order engine that can hide memory latency if the instruction stream is simple enough, but is sensitive to branch prediction and code size.

Output Knowledge Created

This message produces several valuable outputs:

1. A hardware counter profile of the add_to_lc optimization, saved to /tmp/perf-add-to-lc-run1.txt. This file contains the raw event counts that can be compared against the Vec baseline data in /tmp/perf-vec-run1.txt.
2. Confirmation of timing consistency: The single-iteration time of 51.22s is consistent with the three-iteration average of 50.9s from [msg 1207], confirming the measurement is stable.
3. A documented checkpoint in the optimization workflow: The file naming convention (perf-add-to-lc-run1.txt) creates a permanent record that can be revisited later, supporting the project's emphasis on reproducible measurement.
4. The foundation for a rigorous A/B comparison: With both baseline and optimized perf stat data, the assistant can compute deltas in instructions retired, cache miss rates, branch misprediction ratios, and IPC — transforming a simple "it's faster" claim into a detailed mechanistic explanation.

The Thinking Process Visible

The assistant's reasoning is visible in the structure of the command itself. The careful selection of perf events tells a story: the assistant is looking for specific signatures of the optimization.

If Boolean::add_to_lc works as expected, the perf stat data should show:

• Fewer instructions retired: Because each call site no longer allocates a Vec, pushes terms, and then drops it. The add_to_lc path is a simple field addition into an existing structure.
• Fewer cache misses: Because the allocation storm is reduced, there's less pressure on the allocator's free lists and less cache pollution from short-lived objects.
• Fewer branches: The allocation path involves internal branching in the allocator; eliminating allocations removes those branches.
• Stable or improved IPC: Unlike SmallVec (which added enum discriminant checks and larger stack frames), add_to_lc simplifies the code path, which should help IPC. The assistant is also thinking about the broader narrative. The earlier perf stat data for SmallVec showed the danger of optimizing for the wrong metric (cache misses) at the expense of instruction stream quality. Now, with add_to_lc, the assistant expects to see improvement across all metrics — a genuine optimization that reduces work at every level. The decision to run perf stat after the three-iteration benchmark also reveals a two-phase validation strategy: first confirm the optimization works and measure the average improvement, then collect detailed hardware data to understand why. This is the mark of a mature optimization methodology.

Conclusion

Message [msg 1209] is a small but critical step in a larger journey. It represents the moment when a promising optimization — Boolean::add_to_lc, delivering 8.1% synthesis improvement — undergoes rigorous hardware-level validation. The assistant is not content to simply report "it's faster"; they dig into the hardware counters to build a mechanistic understanding of why.

This discipline was forged by earlier failures. The SmallVec A1 optimization taught a painful lesson: cache misses aren't everything, and simpler code often wins on modern CPUs. The B1 cudaHostRegister optimization showed that even a theoretically sound idea can add 5.7 seconds of overhead when it touches 125 GB of memory. These scars produced a rigorous measurement culture where every optimization must be validated at multiple levels — timing, hardware counters, and functional correctness.

The perf stat data collected in this message would later confirm that Boolean::add_to_lc reduced instruction count by 91 billion (15.3%) and branches by 18.6 billion (26.7%) — a clean win across all metrics. But at the moment of message [msg 1209], the assistant doesn't yet know those numbers. They are running the experiment, collecting the data, and trusting the process. That trust in measurement over intuition is the real story here.