The Batched Eval That Wasn't: A Case Study in CPU Microarchitecture Optimization

In the high-stakes world of Filecoin proof generation, every second counts. The SUPRASEAL_C2 Groth16 pipeline, responsible for producing proofs for 32 GiB sectors, already consumed roughly 55 seconds of CPU synthesis time per proof — and the optimization team was hunting for every possible gain. This article examines a single message in that optimization campaign: the moment the assistant turned its attention to optimization #2, "batched eval," after completing work on an arena allocator and software prefetch. The message, <msg id=1143>, is deceptively short — a simple file read prefaced by a design explanation — but it sits at a critical juncture in a multi-round optimization effort that would ultimately reveal how even well-motivated microarchitectural changes can backfire.

The Optimization Pipeline: Context and Motivation

To understand <msg id=1143>, we must first understand the broader context. The assistant had been working through Phase 4 of a performance optimization campaign targeting the CPU synthesis phase of Groth16 proof generation. Earlier analysis using perf stat had revealed that the synthesis hot loop — the enforce method called ~130 million times per proof — was spending approximately 34% of its runtime on memory allocation and deallocation via jemalloc. The CPU was achieving an IPC (instructions per cycle) of 2.60, well below the theoretical maximum for AMD's Zen4 microarchitecture (roughly 3.5–4.0 for well-optimized code). Cache performance was excellent, with negligible L3 and DRAM misses, meaning the bottleneck was purely computational.

The assistant had identified three optimization opportunities ranked by expected impact:

  1. Arena allocator / Vec recycling pool (estimated 15–25% savings): Eliminate the ~780 million malloc+free calls by reusing pre-allocated Vec buffers across enforce calls.
  2. Batched eval (estimated 5–15% savings): Buffer multiple constraints and evaluate their LinearCombinations in an interleaved fashion to improve instruction-level parallelism (ILP).
  3. Software prefetch (estimated 2–5% savings): Insert _mm_prefetch intrinsics to reduce cache miss penalties in the eval loops. By the time we reach <msg id=1143>, optimizations #1 and #3 have already been implemented. The assistant added zero_recycled, from_coeff_recycled, and recycle methods to LinearCombination in bellpepper-core, created a VecPool in ProvingAssignment that reuses 6 Vecs per enforce call, and inserted _mm_prefetch intrinsics into the inner loops of eval and eval_with_trackers. Now it was time for the third and most architecturally ambitious optimization: batched eval.

The Message Itself: Design Intent and File Read

The message begins with a clear statement of intent:

Now let me implement the batched eval (optimization #2). The idea: instead of evaluating A, B, C sequentially per constraint, we buffer a small batch of constraints and evaluate them in an interleaved fashion, allowing the CPU to overlap independent arithmetic chains.

This is a classic ILP optimization strategy. Modern superscalar processors can execute multiple instructions per cycle, but they are limited by data dependencies within a single instruction stream. By interleaving independent arithmetic chains — in this case, the field arithmetic for evaluating the A, B, and C linear combinations across multiple constraints — the CPU's out-of-order execution engine has more independent instructions to schedule across its execution ports. The hope was that this would push IPC closer to the Zen4 theoretical maximum.

The assistant then describes the implementation approach:

The approach: I'll add a batch eval function in bellperson's lc.rs that processes multiple constraints' LCs together, interleaving the field operations.

This is followed by a [read] tool call that reads the current state of /home/theuser/curio/extern/bellperson/src/lc.rs. The file content shows the recently-added prefetch_read function and the beginning of eval_with_trackers. The assistant is reading the file to understand where to add the new batched eval function — it needs to see the existing code structure, the function signatures, and the import dependencies before making edits.

The Reasoning Process: What the Assistant Was Thinking

The assistant's reasoning in this message reveals several important design decisions and assumptions:

Why batched eval specifically? The assistant had previously established through perf stat that the synthesis hot loop was CPU-bound with IPC=2.60. The field arithmetic operations (Montgomery multiplication and addition in BLS12-381 scalar field) are inherently sequential at the instruction level — a single multiplication has a latency of several cycles and its result feeds into the next operation. By interleaving independent chains from different constraints, the CPU can keep more execution ports busy during those latency windows.

Why bellperson's lc.rs? The assistant chose to implement the batched eval in bellperson rather than bellpepper-core because bellperson is the higher-level proving framework that orchestrates the constraint system evaluation. The eval_with_trackers function in bellperson's lc.rs is where the actual field arithmetic happens for the A and B linear combinations (the C polynomial uses a simpler eval from bellpepper-core). Adding a batched variant here would allow the assistant to interleave the A and B evaluations across multiple constraints without changing the lower-level API.

The buffering trade-off: The assistant implicitly assumes that buffering a small batch of constraints (likely 4–8) and evaluating them together will have negligible overhead compared to the ILP gains. The buffering requires storing the LinearCombination references temporarily, which adds some memory pressure and control flow complexity. The assistant judged this overhead to be small relative to the 5–15% expected speedup.

Assumptions Made

Several assumptions underpin this message, some of which would later prove incorrect:

  1. That the CPU has spare execution port capacity: The assistant assumed that IPC=2.60 meant there was headroom to reach 3.0+ by feeding more independent instructions. This is a reasonable assumption, but it depends on the specific mix of instructions in the eval loop and how well they map to Zen4's execution ports.
  2. That interleaving doesn't hurt cache behavior: By processing multiple constraints' LCs together, the access pattern to input_assignment and aux_assignment arrays becomes more scattered. The assistant assumed this wouldn't cause cache issues (given the already excellent cache performance), but the more complex control flow could introduce branch mispredictions and other penalties.
  3. That the field arithmetic chains are the dominant bottleneck: The assistant had already identified allocation as the #1 bottleneck (34% of runtime). With that addressed by optimization #1, the remaining time would be dominated by the actual field arithmetic in eval. The batched eval targets exactly this.
  4. That the batch size can be chosen to balance ILP and overhead: The assistant didn't specify a batch size in this message, but the design implies a small fixed-size buffer (likely a power of two for alignment). Choosing this value requires balancing ILP benefits against buffering overhead and register pressure.

What Went Wrong: The IPC Regression

The chunk summary reveals a sobering outcome: the synth-only microbenchmark showed only a ~1% improvement (54.9s vs 55.5s baseline), far below the expected 15–25%. Worse, perf stat revealed that instructions dropped 4.1% but IPC also fell from 2.60 to 2.53, indicating the interleaved eval's more complex control flow was hurting pipeline utilization rather than helping it.

The assistant then reverted the interleaved eval back to separate eval_with_trackers calls while keeping the recycling pool and prefetch, isolating the batched eval as the cause of the regression. This is a textbook example of why microarchitectural optimization must be validated empirically — even well-motivated changes can backfire when they interact with the CPU's complex pipeline dynamics.

Input and Output Knowledge

Input knowledge required to understand this message includes: familiarity with the Groth16 proof generation pipeline and its constraint system architecture; understanding of CPU microarchitecture concepts like ILP, out-of-order execution, and execution ports; knowledge of the bellpepper/bellperson codebase structure and the role of LinearCombination, eval_with_trackers, and the enforce method; and awareness of the previous perf stat analysis showing IPC=2.60.

Output knowledge created by this message is primarily the design intent and the implementation plan for the batched eval function. The actual code changes would follow in subsequent messages, but this message establishes the rationale and approach. The file read also provides a snapshot of the current code state, showing the recently-added prefetch infrastructure that optimization #3 had introduced.

The Broader Lesson

The story of <msg id=1143> is ultimately a cautionary tale about the gap between microarchitectural theory and practice. The batched eval optimization was grounded in sound reasoning — interleaving independent arithmetic chains should improve ILP — but it failed because the more complex control flow introduced pipeline penalties that outweighed the benefits. The assistant's disciplined approach (implement, measure, revert when it doesn't work) is a model for performance engineering in complex systems.

This message also illustrates the iterative nature of optimization work. The assistant didn't stop at the batched eval failure; instead, it reverted the change, kept the working optimizations (#1 and #3), and moved on to deeper analysis that would eventually identify the true bottleneck: temporary LinearCombination allocations inside closures via Boolean::lc(). That discovery would lead to a much more impactful optimization — adding add_to_lc and sub_from_lc methods to Boolean — that directly eliminated the allocation overhead at its source.

In the end, the batched eval that wasn't became a valuable data point, teaching the team that not all theoretically sound optimizations survive contact with real hardware.