The Elusive 7%: Diagnosing CPU Contention in Parallel SNARK Synthesis

In the high-stakes world of Filecoin proof generation, every second counts. The cuzk proving engine, designed to generate Groth16 proofs for Filecoin's Proof-of-Replication (PoRep) protocol, had just undergone a significant architectural transformation. After weeks of building a pipelined proving engine with cross-sector batching, Pre-Compiled Constraint Evaluators (PCE), and slotted partition proving, the team had identified a glaring inefficiency: the GPU was idle for 12–14 seconds between every proof. The culprit was strictly sequential CPU synthesis that took 39 seconds per proof while the GPU finished in only 27 seconds. The obvious fix—parallel synthesis—had been implemented using a tokio::sync::Semaphore to allow two syntheses to run concurrently. But when the first benchmark with synthesis_concurrency=2 and client concurrency -j 3 was run, the results were puzzling: GPU utilization soared to 99.3%, yet throughput improved by only 5%, from 45.3 to 43.0 seconds per proof. Something was wrong.

This article examines a single message in that debugging session—message index 1878—where the assistant runs a benchmark with a modified configuration to test a hypothesis about CPU resource contention. The message appears simple on the surface: a bash command that launches a benchmark with -j 2 client concurrency instead of -j 3. But behind that one-line command lies a sophisticated chain of reasoning about pipeline architecture, resource contention, and the law of diminishing returns in parallel systems.

The Message: A Benchmark with a Hypothesis

The target message, produced by the AI assistant in the cuzk coding session, reads as follows:

[bash] > /tmp/cuzk-parallel2.log
cd /home/theuser/curio/extern/cuzk && \
  ./target/release/cuzk-bench batch -t porep --c1 /data/32gbench/c1.json -j 2 -c 8 2>&1 | tee /tmp/cuzk-parallel2-bench.txt
2026-02-18T17:43:44.516521Z  INFO cuzk_bench: loading C1 output path=/data/32gbench/c1.json
=== Batch Benchmark ===
proof type:  porep
count:       8
concurrency: 2

  [1/8] COMPLETED — 67.8s (prove=28995 ms, queue=262 ms)
  [2/8] COMPLETED — 99.9s (prove=32018 ms, queue=20141 ms)
  [3/8] COMPLETED — 77.4s (prove=26574 ms, queue=2546 ms)
  [4/8] COMPLETED — 80.8s (prove=27162 ms, queue=771 ms)
  [5/8] COMPLETED — 76.2s (prove=28751 ms, q...

The output is truncated in the conversation, but the critical data is already visible. The benchmark is running 8 proofs (-c 8) with a client-side concurrency of 2 (-j 2), meaning the bench tool will keep at most 2 proof requests in-flight to the daemon at any time. The daemon itself is configured with synthesis_concurrency=2 (from the config file /tmp/cuzk-parallel2.toml written in the preceding message), allowing two CPU synthesis tasks to run in parallel.

The Reasoning Behind the Configuration Change

To understand why the assistant chose -j 2, we must trace the reasoning chain from the earlier messages. The first parallel synthesis test (msg 1871–1875) used -j 3—three in-flight client requests—and produced 99.3% GPU utilization. This should have been a victory: the GPU idle gap was essentially eliminated. But the throughput improvement was disappointing because both synthesis time and GPU time had inflated. Synthesis averaged 47.2 seconds (up from 39.3), and GPU time averaged 37.6 seconds (up from 26.7). The net effect was only 5% faster end-to-end.

The assistant correctly diagnosed the root cause: CPU resource contention. The machine has 96 CPU cores (an AMD Ryzen Threadripper PRO 7995WX). When two syntheses run concurrently, each one spawns rayon parallel work across the entire thread pool. Simultaneously, the GPU prover's b_g2_msm step—a multi-threaded Pippenger MSM that runs on the CPU during GPU proving—also competes for those same cores. With three in-flight proofs (-j 3), the system had two syntheses running plus one GPU proof's CPU work, all fighting for 96 cores. The result was that every component slowed down.

The assistant's hypothesis was that reducing client concurrency from 3 to 2 would reduce the pressure. With -j 2, at most two proofs would be in-flight simultaneously. Since the daemon's synthesis_concurrency=2 already limits synthesis to two concurrent tasks, the client concurrency of 2 would mean: at most two syntheses running, and when one finishes and moves to GPU, the client sends the next request. But critically, with only two in-flight proofs, there would be moments when only one synthesis is active (while the other proof is on the GPU), reducing the peak CPU demand.

This is the thinking that motivated message 1878. The assistant wasn't randomly tweaking parameters—it was systematically testing a model of how CPU contention scales with the number of concurrent proof requests.## What the Output Reveals

The benchmark output in message 1878 shows the first five of eight proofs completing. The per-proof completion times reveal a system under stress but not catastrophically so:

Proof 1: 67.8 seconds (prove time 28,995 ms, queue time 262 ms)
Proof 2: 99.9 seconds (prove time 32,018 ms, queue time 20,141 ms)
Proof 3: 77.4 seconds (prove time 26,574 ms, queue time 2,546 ms)
Proof 4: 80.8 seconds (prove time 27,162 ms, queue time 771 ms)
Proof 5: 76.2 seconds (prove time 28,751 ms, queue time...) The "prove" time is the GPU proving duration for that proof, and "queue" is the time the request spent waiting in the scheduler before being picked up. The first proof is always slowest (cold start, loading SRS parameters). Proof 2 shows a 20-second queue time—indicating that both synthesis slots were occupied when it arrived. But proofs 3–5 show much lower queue times (0.8–2.5 seconds), suggesting the pipeline had stabilized. The assistant's subsequent analysis (msg 1879) would compute the average: 42.2 seconds per proof, or 1.422 proofs per minute. This is a 7% improvement over the baseline of 45.3 seconds per proof, and notably better than the 43.0 seconds achieved with -j 3. The hypothesis was confirmed: reducing client concurrency from 3 to 2 reduced CPU contention enough to improve throughput. But the waterfall timeline analysis (also in msg 1879) revealed a more nuanced picture. While the first two proofs (P1 and P2) started synthesis simultaneously as expected, subsequent proofs fell back into a sequential pattern. With synthesis_concurrency=2 and only 2 in-flight requests, after P1 and P2 launched, the dispatcher couldn't submit P3 until one of the first two completed its GPU phase and freed a client slot. The GPU idle gaps stabilized around 10–16 seconds—better than the original 12–14 seconds, but far from the theoretical zero the assistant had hoped for.

The Deeper Insight: Client-Side Starvation

This led to a critical realization. The bottleneck wasn't just in the daemon's synthesis pipeline—it was also in the client-side concurrency model. The bench tool's -j parameter controls how many proof requests can be outstanding at once. With -j 2, even though the daemon could theoretically synthesize two proofs concurrently, the client couldn't submit the third proof until one of the first two had completed its entire lifecycle (synthesis + GPU). This created a structural starvation: the pipeline's depth was limited by the client, not the server.

The assistant immediately tested this theory in the next message (msg 1880) by running -j 4 to see if deeper client concurrency would keep the pipeline fuller. The result was catastrophic: 60.2 seconds per proof. With four in-flight requests, the daemon's two synthesis slots were constantly occupied, and the GPU's b_g2_msm step was starved for CPU cores, ballooning GPU time to as much as 118 seconds for a single proof. The system had crossed a tipping point where contention overwhelmed any benefit from parallelism.

Assumptions and Their Consequences

Several assumptions underpinned this experiment, and understanding them is crucial for interpreting the results.

Assumption 1: CPU cores are fungible. The assistant assumed that the 96-core machine could handle two concurrent syntheses plus GPU CPU work without significant slowdown. In practice, the rayon thread pool used by both synthesis and b_g2_msm is a shared resource, and when saturated, each task gets fewer cores, increasing wall-clock time. The 96 cores are not truly independent when they share memory bandwidth, cache hierarchy, and memory controller channels.

Assumption 2: The semaphore provides sufficient backpressure. The assistant implemented parallel synthesis using a tokio::sync::Semaphore with synthesis_concurrency permits. The assumption was that the semaphore would naturally throttle synthesis to match the GPU's consumption rate. But the semaphore only controls how many syntheses run concurrently—it doesn't coordinate with the GPU's actual readiness. When syntheses complete faster than the GPU can consume them (which didn't happen here), or when the client submits requests faster than the pipeline can drain (which did happen with -j 4), the semaphore alone is insufficient.

Assumption 3: Client concurrency and server concurrency are independent. The assistant initially treated -j (client concurrency) and synthesis_concurrency (server concurrency) as orthogonal parameters. The experiments revealed they are tightly coupled: the effective pipeline depth is min(client_concurrency, synthesis_concurrency + 1), because the GPU acts as a serial consumer. With -j 2 and synthesis_concurrency=2, the effective depth is 2—the client can't submit a third request until one completes, so the pipeline is always exactly full but never overstuffed.

Input Knowledge Required

To fully understand this message, one needs familiarity with several domains:

Groth16 proof generation: The message deals with the C2 phase of Filecoin's PoRep protocol, where a constraint system is synthesized (CPU-bound) and then proved via multi-scalar multiplication and number-theoretic transform on GPU. The b_g2_msm step is a multi-threaded Pippenger MSM that runs on CPU during GPU proving.
Tokio async runtime: The engine uses tokio::spawn for async tasks and tokio::sync::Semaphore for concurrency control. Understanding the difference between spawn_blocking (for CPU-heavy work) and tokio::spawn (for async I/O) is essential.
Rayon parallelism: Both synthesis and b_g2_msm use rayon's work-stealing thread pool. When multiple tasks compete for the same pool, work-stealing can cause tasks to interfere with each other's cache locality and memory bandwidth.
Pipeline architecture: The cuzk engine implements a two-stage pipeline (synthesis → GPU) with a bounded channel. The synthesis task pulls from a scheduler, runs CPU synthesis, and pushes the result to the GPU worker. Understanding this producer-consumer model is key to interpreting the waterfall timelines.
Benchmark methodology: The -j flag controls client-side concurrency (how many proofs are in-flight), while -c controls total proof count. The "queue" time in the output measures how long a request waited in the scheduler before synthesis began.

Output Knowledge Created

This message produced several valuable outputs:

A benchmark result showing that synthesis_concurrency=2 with -j 2 achieves 42.2 seconds per proof—a 7% improvement over the sequential baseline.
Evidence that CPU contention is the dominant bottleneck. The fact that reducing client concurrency from 3 to 2 improved throughput (from 43.0 to 42.2 s/proof) proves that the system is CPU-bound, not GPU-bound. Adding more parallelism to an already CPU-saturated system degrades performance.
A validated methodology for testing pipeline depth. The systematic variation of -j (2, 3, 4) with fixed synthesis_concurrency=2 created a response surface showing the optimal operating point. This is a textbook example of load testing a producer-consumer pipeline.
Confirmation that the waterfall timeline instrumentation works. The assistant could immediately parse the timeline events to understand why gaps appeared, validating the instrumentation investment from earlier in the session.

Mistakes and Incorrect Assumptions

The most significant mistake was the initial expectation that parallel synthesis would fully eliminate GPU idle gaps. The theoretical model predicted that with two concurrent syntheses, a new proof would be ready every ~20 seconds (39s / 2), which is less than the GPU's 27-second proving time, so the GPU would never wait. This model failed because it assumed syntheses would complete in their standalone time (39s) even when running concurrently. In reality, concurrent syntheses took 45–50 seconds due to CPU contention, and GPU time inflated to 29–33 seconds because b_g2_msm was also slowed. The theoretical model was correct in form but wrong in its parameters—it used uncontended timings that didn't hold under load.

A subtler error was treating the 96-core CPU as a homogeneous resource. The assistant initially thought "96 cores is plenty for 2 syntheses + GPU work." But the rayon thread pool doesn't partition cores cleanly—work-stealing means tasks can migrate between threads, causing cache misses and memory bandwidth contention. The Threadripper PRO 7995WX has 96 Zen4 cores across 12 CCDs (core complexes), each with its own L3 cache. When rayon tasks from different syntheses land on different CCDs, cross-CCD communication becomes expensive. A more sophisticated approach would pin each synthesis to a specific set of cores using rayon::ThreadPoolBuilder::num_threads() or core affinity.

The Broader Context

This message sits at a pivotal moment in the cuzk project. The team had spent weeks building increasingly sophisticated pipeline features—cross-sector batching, PCE pre-compilation, slotted partition proving—each delivering measurable improvements. But the parallel synthesis experiment revealed that the low-hanging fruit had been picked. The 7% improvement from parallel synthesis was real but modest, and further gains would require deeper changes: either reducing the absolute CPU time of synthesis (via specialized MatVec kernels or pre-sorted SRS, as proposed in Phase 5 Wave 2/3) or decoupling the CPU workloads entirely (e.g., moving b_g2_msm to GPU).

The assistant's summary message (msg 1884) captures this inflection point perfectly. It lists four options for next steps: (A) limit rayon threads for synthesis to leave headroom for b_g2_msm, (B) move b_g2_msm off CPU entirely, (C) reduce synthesis time so it's faster than GPU time, or (D) accept the 7% improvement and commit. Each option represents a different strategic bet on where the next bottleneck will appear.

Conclusion

Message 1878 is, on its surface, a simple benchmark command. But it is the product of a sophisticated diagnostic process that had already: instrumented the engine with waterfall timeline logging, implemented parallel synthesis via tokio semaphores, run multiple benchmarks with varying parameters, and built a mental model of CPU contention between synthesis and GPU proving. The choice of -j 2 was not arbitrary—it was a targeted test of the hypothesis that reducing client concurrency would reduce CPU contention and improve throughput.

The result—42.2 seconds per proof, a 7% improvement—validated the hypothesis but also revealed the fundamental challenge of parallel pipeline optimization: when you eliminate one bottleneck (GPU idle), another appears (CPU contention). The law of diminishing returns applies ruthlessly to parallel systems, and the 7% gain from parallel synthesis represents the last easy improvement before deeper architectural changes are required. The message stands as a testament to the importance of systematic benchmarking, instrumentation-driven diagnosis, and the willingness to accept modest gains when the data demands it.