The Moment the Model Saw the Code: How Reading `par_map(1, ...)` Reshaped a Memory-Bandwidth Optimization Strategy

In the middle of an intense optimization session for the SUPRASEAL_C2 Groth16 proof generation pipeline, a single message—message index 2711—serves as a pivotal moment of discovery. It is a message that looks, at first glance, like a routine code-reading exercise: the assistant traces through a C++ thread pool implementation to confirm how par_map behaves with a single item. But what it uncovers fundamentally reshapes the entire Phase 11 optimization plan, invalidating a core assumption that had been carried across multiple rounds of analysis and design.

The Context: A Pipeline Under Pressure

To understand why this message matters, one must understand the optimization journey that led to it. The cuzk SNARK proving engine, a high-performance Groth16 implementation for Filecoin's Proof-of-Replication (PoRep), had been the subject of a sustained optimization campaign spanning multiple phases. By Phase 9, the team had achieved significant throughput improvements through PCIe transfer optimization, but a new bottleneck had emerged: DDR5 memory bandwidth contention at high concurrency levels.

The waterfall timing analysis told a clear story. When running with synthesis_concurrency=15 and 15 concurrent proofs in flight, throughput plateaued at approximately 38 seconds per proof—a regression from the 32.1 seconds achievable in isolation. The GPU utilization remained high at 90.8%, but the CPU-side pipeline stages were inflating under load: synthesis times grew from 35 seconds to 54 seconds, and prep_msm (the preprocessing step for multi-scalar multiplication) inflated from 1.7 seconds to 2.7 seconds or more.

The root cause was not raw DDR5 bandwidth exhaustion—the measured demand of ~34 GB/s was far below the theoretical ~333 GB/s available. Instead, three subtle interference mechanisms were at play: TLB shootdown storms from concurrent munmap() calls during async deallocation, thread pool oversubscription as two separate pools (Rayon for synthesis and groth16_pool for preprocessing) both defaulted to 192 threads competing for 12 L3 cache domains, and temporal phase overlap where memory-heavy operations all fired simultaneously.

The Phase 11 Plan Takes Shape

The assistant had designed Phase 11 around three interventions. Intervention 1 would bound async deallocation to a single thread, eliminating the TLB shootdown storms. Intervention 2 would add a memory-phase semaphore interlock passed through the FFI (like the existing gpu_mtx), allowing at most N concurrent synthesis workers during prep_msm's window to reduce mutual interference. Intervention 3 would reduce the groth16_pool thread count and optionally pin threads to specific CCDs for spatial isolation.

The user had approved this plan, choosing "All 3, semaphore interlock" as the implementation scope. But then came a crucial caution: "We should be careful to not kill parallelism btw" ([msg 2704]). This prompted the assistant to pause and think carefully about where parallelism actually matters versus where it is wasteful.

The Critical Re-examination

In [msg 2705], the assistant performed a thoughtful analysis of parallelism trade-offs. It identified four areas where parallelism is critical and must not be throttled: the 10 concurrent partition syntheses that feed the GPU pipeline, the Rayon parallelism within each synthesis for witness generation, the prep_msm thread pool parallelism for bitmap classification and tail MSM population, and the NTT/MSM GPU overlaps essential for pipelining. It also identified three areas where parallelism is wasteful or harmful: async deallocation (pure cleanup, no impact on latency), the groth16_pool at 192 threads (excessive for single-circuit mode), and the simultaneous overlap of prep_msm with the full synthesis pack.

But this analysis contained an assumption: that prep_msm with num_circuits=1 (the per-partition pipeline mode) actually used the groth16_pool's thread parallelism. The assistant noted: "The actual parallelism within single-circuit prep_msm is in the bitmap loop, which iterates over bit_vector_size entries (~360K). Even 16-32 threads saturate this." This assumption seemed reasonable—the code called get_groth16_pool().par_map(num_circuits, ...), and with a pool of 192 threads, one would expect substantial parallelism.

To verify this, the assistant launched a research task ([msg 2705]) to check the prep_msm parallelism details. The task returned empty ([msg 2706]), so the assistant began reading the code directly.## The Discovery: Reading par_map(1, ...)

Message [msg 2711] begins with a moment of recognition: "Now I see." The assistant has just traced through the thread_pool_t::par_map implementation in /home/theuser/curio/extern/supraseal/deps/sppark/util/thread_pool_t.hpp. The logic is straightforward:

void par_map(size_t num_items, size_t stride, Workable work, size_t max_workers = 0)
{
    size_t num_steps = (num_items + stride - 1) / stride;
    size_t num_workers = std::min(size(), num_steps);
    if (max_workers > 0)
        num_workers = std::min(num_workers, max_workers);
    ...
}

When num_items = 1 (as it is for num_circuits=1 in the per-partition pipeline mode), num_steps = 1, and therefore num_workers = min(pool_size, 1) = 1. The lambda runs work(0) on exactly one pool thread. The entire prep_msm body—the bitmap classification, the tail MSM population, the loops over aux_size (~130 million iterations)—runs sequentially on a single thread.

The assistant's realization is stark: "This is critical: prep_msm with num_circuits=1 is single-threaded. The groth16_pool size is irrelevant for prep_msm—it only uses 1 thread regardless."

Why This Changes Everything

This discovery invalidates a central assumption of the Phase 11 plan. The entire Intervention 2—the memory-phase semaphore interlock—had been designed around the idea that prep_msm and synthesis workers were competing for memory bandwidth simultaneously, with prep_msm using the full groth16_pool of 192 threads to process its ~23 GiB of SRS data. The semaphore would throttle synthesis workers during prep_msm's window to give it more bandwidth.

But if prep_msm is single-threaded, its memory bandwidth demand is dramatically lower. A single thread reading ~23 GiB over 1.9 seconds achieves roughly 12 GB/s—a fraction of the DDR5 bandwidth. It is not competing significantly with the synthesis workers. The interference that was attributed to prep_msm must have a different cause.

The assistant immediately identifies the real implication: "The only place the pool size matters is b_g2_msm (line 575: mult_pippenger with &get_groth16_pool()), which DOES use the full pool for single-circuit Pippenger." The b_g2_msm phase—the G2 multi-scalar multiplication that runs after prep_msm completes—is the actual thread-pool-heavy operation. It runs for approximately 0.4 seconds but uses all 192 groth16_pool threads simultaneously, competing with the 192 Rayon synthesis threads for L3 cache and memory bandwidth.

The Thinking Process Revealed

The message shows a clear chain of reasoning. The assistant:

Reads the code: Traces par_map(1, ...) through the template to compute num_steps and num_workers.
Verifies the consequence: Confirms that with num_circuits=1, the lambda runs on exactly 1 pool thread.
Checks the loops: Notes that the prep_msm body contains sequential for loops over aux_size (~130M iterations), confirming no inner parallelism.
States the conclusion explicitly: "prep_msm with num_circuits=1 is single-threaded."
Identifies the corrected target: The groth16_pool size only matters for b_g2_msm, not prep_msm.
Extends the investigation: Immediately begins checking mult_pippenger to understand how b_g2_msm uses the pool. This is a textbook example of how reading source code can overturn assumptions that survived multiple rounds of high-level analysis. The assistant had spent hours designing a sophisticated semaphore-based interlock system, only to discover that the interference it was designed to mitigate didn't exist in the way assumed.

Input Knowledge Required

To understand this message, one needs familiarity with several concepts: the Groth16 proving pipeline structure (partition synthesis, prep_msm, b_g2_msm), the C++ thread pool abstraction (thread_pool_t with its par_map method), the per-partition dispatch mode where num_circuits=1, and the concept of multi-scalar multiplication (Pippenger's algorithm). One also needs context from the preceding optimization phases: the Phase 9 benchmark results showing throughput plateau, the waterfall timing analysis identifying memory contention, and the Phase 11 plan document with its three interventions.

Output Knowledge Created

This message produces several critical pieces of knowledge. First, it establishes definitively that prep_msm with num_circuits=1 is single-threaded, regardless of the groth16_pool size. Second, it redirects attention from prep_msm to b_g2_msm as the actual thread-pool-heavy phase. Third, it narrows the scope of Intervention 3 (reducing groth16_pool thread count): the pool size reduction will affect b_g2_msm performance, not prep_msm performance. Fourth, it implies that the memory-phase semaphore interlock (Intervention 2) may be unnecessary or should be redesigned to target the b_g2_msm window rather than the prep_msm window.

Assumptions and Corrections

The key assumption that was corrected here is that par_map(num_circuits, ...) with num_circuits=1 would distribute work across the thread pool. This is a natural assumption—the function is called par_map, it takes a thread pool reference, and the pool has 192 threads. But the implementation reveals that par_map partitions work by items, not by loop iterations within an item. When there is only one item, there is nothing to parallelize.

A subtler assumption was that prep_msm's bitmap loop (iterating over ~360K entries) would be internally parallelized. The code shows it is not—the loop runs sequentially within the single worker thread. The par_map call only parallelizes across circuits, not within a circuit's processing.

The Broader Significance

This message exemplifies a pattern that recurs throughout systems optimization: the gap between conceptual understanding and code-level reality. The assistant had a high-level mental model of the pipeline where prep_msm was a "memory-bandwidth-heavy phase" that "uses the groth16_pool." Both statements are true, but the second one masked a crucial detail: how it uses the pool. The par_map abstraction creates a boundary where parallelism is coarse-grained (across circuits) rather than fine-grained (within a circuit's processing loop).

The correction ripples through the optimization strategy. The semaphore interlock, which would have required passing a new synchronization primitive through the Rust-to-C++ FFI, can potentially be simplified or eliminated. The groth16_pool size reduction, previously aimed at reducing prep_msm's footprint, now targets b_g2_msm instead—a much shorter (0.4s) but more intensely parallel phase. The async deallocation bounding (Intervention 1) remains independently valuable.

In the subsequent messages, the assistant would go on to check mult_pippenger to understand b_g2_msm's thread usage, then redesign the plan accordingly. But message 2711 is the turning point—the moment when reading a few lines of C++ template code reshaped the trajectory of a multi-week optimization campaign. It is a reminder that in performance engineering, assumptions must be verified against the actual code, no matter how reasonable they seem.

The Moment the Model Saw the Code: How Reading par_map(1, ...) Reshaped a Memory-Bandwidth Optimization Strategy