The $0.02 Question: How Many Threads Does par_map Really Use?

"We should be careful to not kill parallelism btw." — User, msg 2704

In the middle of a deep optimization campaign targeting the cuzk SNARK proving engine's memory bandwidth contention, a single bash command reveals the entire engineering mindset at work. The message is deceptively small:

[assistant] [bash] grep -n "par_map" /home/theuser/curio/extern/supraseal/deps/sppark/util/thread_pool_t.hpp
175:    // pool.par_map(20, [&](size_t i) { std::cout << i << std::endl; });
177:    void par_map(size_t num_items, size_t stride, Workable work,
211:    void par_map(size_t num_items, Workable work, size_t max_workers = 0)
212:    {   par_map(num_items, 1, work, max_workers);   }

Three lines of output. A grep for a function name. Yet this single query sits at a critical juncture in a months-long optimization effort, and the answer it yields — or rather, the answer it sets up to be read in the very next message — determines whether a major optimization proposal lives or dies.

The Crisis That Led Here

To understand why this grep matters, we must understand the pressure that produced it. The cuzk proving engine, responsible for generating Groth16 proofs for Filecoin's Proof-of-Replication (PoRep), had just undergone a painful Phase 10 failure. The assistant had designed and implemented a sophisticated two-lock GPU interlock architecture intended to allow two GPU workers to overlap their PCIe transfer and compute phases. It failed catastrophically — 16 GB of VRAM couldn't accommodate pre-staged buffers from multiple workers, and CUDA's device-global synchronization APIs defeated the lock split's purpose entirely. The code was reverted to Phase 9's single-lock approach, and a comprehensive benchmark campaign followed.

That campaign revealed the true bottleneck: not GPU utilization (which hit 90.8%), but DDR5 memory bandwidth contention on the CPU side. At high concurrency (c=15–20), throughput plateaued at ~38 seconds per proof, with synthesis inflating from 35s to 54s and prep_msm inflating from 1.7s to 2.7s. The root cause was not raw bandwidth exhaustion — the system had ~333 GB/s theoretical bandwidth and only ~34 GB/s demand — but rather three specific pathologies: TLB shootdown storms from concurrent munmap() calls, thread pool oversubscription with two 192-thread pools competing for 12 L3 cache domains, and temporal overlap of memory-heavy phases.

The assistant designed Phase 11 with three interventions targeting these pathologies. Intervention 2 was the most nuanced: reduce the groth16_pool thread count from its default of 192 to something like 16–32 threads, to shrink the memory footprint and L3 cache competition when prep_msm and b_g2_msm run. But then the user injected a crucial caution: "We should be careful to not kill parallelism btw."

The Tension: Parallelism vs. Contention

The user's warning (msg 2704) forced the assistant to think carefully about where parallelism actually matters. The response (msg 2705) is a masterclass in systems thinking — a detailed breakdown of what must not be throttled versus what can safely be constrained:

Critical parallelism (DO NOT throttle): The 10 concurrent partition syntheses that feed the GPU pipeline; the rayon thread-level parallelism within each synthesis for witness generation; the prep_msm thread pool parallelism for bitmap classification; and the GPU-side NTT/MSM overlaps.

Wasteful or harmful parallelism: The async_dealloc threads (pure cleanup, no latency impact); the groth16_pool at 192 threads when num_circuits=1 (per-partition mode); and the simultaneous execution of prep_msm alongside the full pack of 10 synthesis workers.

But there was a problem. The assistant's analysis of where parallelism is wasteful rested on a critical assumption: that par_map(num_circuits, ...) with num_circuits=1 would use only a single thread, not the full pool of 192. If that assumption was wrong — if par_map still dispatched work across all 192 threads even for a single item — then reducing the pool size would directly hurt prep_msm performance, extending the GPU lock hold time and potentially regressing throughput.

The assistant needed to verify this assumption. A previous task tool (spawned at the end of msg 2705) had returned empty — the subagent produced no output. So the assistant switched to direct investigation, reading relevant sections of groth16_cuda.cu (msg 2706–2708) and then issuing the bash grep that is the subject of this article.## What the Grep Actually Revealed

The grep output shows three lines. The first is a comment showing example usage. The second and third are the function declarations for par_map:

void par_map(size_t num_items, size_t stride, Workable work, ...);
void par_map(size_t num_items, Workable work, size_t max_workers = 0)
{   par_map(num_items, 1, work, max_workers);   }

The second overload is a convenience wrapper that defaults stride=1 and delegates to the first. On its own, this output doesn't answer the question — it only shows the interface, not the implementation. But it achieves something critical: it confirms the exact location and signature of the function, enabling the assistant to read the full implementation in the next step (msg 2710).

The actual answer comes in msg 2711, where the assistant reads the implementation body and discovers:

With par_map(1, ...) (num_circuits=1): num_steps = 1, num_workers = min(pool_size, 1) = 1. The lambda runs work(0) on exactly 1 pool thread.

This confirmation is the output knowledge created by the chain of investigation. It validates the assistant's earlier assumption (stated in msg 2705) that prep_msm with num_circuits=1 is effectively single-threaded. But it also surfaces a critical nuance: while prep_msm uses only 1 thread, b_g2_msm (the G2 MSM computation) does use the full pool via mult_pippenger. This means reducing groth16_pool from 192 to 32 threads would have zero impact on prep_msm latency, but would directly affect b_g2_msm's performance. The assistant now has precise, verified knowledge to feed into the Phase 11 design.

The Thinking Process: A Case Study in Verification

The subject message is a textbook example of how rigorous engineering verification works under uncertainty. The chain of reasoning is worth examining step by step:

  1. Hypothesis formation (msg 2705): The assistant hypothesizes that groth16_pool at 192 threads is wasteful for num_circuits=1 because par_map with 1 item should use 1 thread. This is stated as an assumption, not a fact.
  2. First verification attempt (msg 2705, task tool): A subagent is spawned to research the code. It returns empty — a failure mode that can happen with complex codebase navigation tasks.
  3. Direct code reading (msg 2706–2707): The assistant pivots to reading groth16_cuda.cu directly, examining the call sites where par_map is invoked. This shows how the function is called but not how it works internally.
  4. File location (msg 2708): A find command locates the header file containing the par_map implementation.
  5. Function signature lookup (msg 2709, the subject): The grep confirms the function declarations and their exact file location.
  6. Implementation analysis (msg 2710–2711): Reading the function body reveals the critical num_workers = min(pool_size, num_steps) logic, confirming the assumption for the single-circuit case. This methodical escalation — from spawned subagent to direct read to grep to full implementation analysis — demonstrates a disciplined approach to uncertainty. Each step builds on the previous one, narrowing the gap between assumption and verified knowledge.

The Deeper Significance

At first glance, a grep for par_map seems trivial — three lines of output, a routine codebase query. But in the context of the Phase 11 optimization campaign, this message represents the moment when a critical assumption was put to the test. The user's caution ("We should be careful to not kill parallelism") created the pressure to verify rather than assume. The assistant responded not with defensive justification but with systematic investigation.

The message also reveals something about the nature of performance engineering at scale. The difference between 192 threads and 1 thread for a single-item par_map call is not a bug — both produce correct results. But understanding that difference is essential for making correct optimization decisions. A naive reading of "groth16_pool has 192 threads" might lead one to believe that prep_msm uses heavy parallelism. The grep disproves that intuition, enabling a more targeted intervention.

Moreover, the discovery that b_g2_msm does use the full pool via mult_pippenger adds crucial nuance. It means reducing groth16_pool thread count is not free — it trades prep_msm's (nonexistent) parallelism for b_g2_msm's real parallelism. The assistant now has the information needed to make that trade-off consciously, with the ability to benchmark both configurations.

Conclusion

The subject message — a simple bash grep — is a hinge point in the Phase 11 optimization effort. It transforms an assumption ("par_map with 1 item uses 1 thread") into a verified fact, while simultaneously uncovering a countervailing consideration (b_g2_msm's genuine need for pool parallelism). The user's caution about preserving parallelism forced this verification, and the assistant's methodical response exemplifies the engineering discipline required for high-stakes systems optimization.

In the end, the $0.02 question — "how many threads does par_map really use?" — had a $0.02 answer: one. But the process of finding that answer, and the nuanced understanding it produced, was worth far more.