Diagnosing CPU Contention: The partition_workers=30 Experiment in Phase 8

Introduction

In the ongoing optimization of the cuzk SNARK proving engine for Filecoin PoRep, Phase 8 had just delivered a breakthrough: the dual-worker GPU interlock, which eliminated GPU idle gaps by narrowing the C++ static mutex to cover only the CUDA kernel region. The results were impressive — single-proof GPU efficiency hit 100.0%, and multi-proof throughput improved 13–17% over Phase 7, reaching 44.0 seconds per proof with partition_workers=20 and concurrency=5, j=3. The architecture now ran two GPU workers per device, interleaving CPU preprocessing on one worker while the other ran CUDA kernels, achieving near-perfect GPU utilization.

But the partition_workers parameter — which controls how many CPU threads are spawned to synthesize circuit partitions in parallel — had not yet been systematically explored. The user, wanting to understand the trade-off space, issued a simple request: "Try with config partition_workers = 30" (see [msg 2233]). This seemed like a reasonable exploration: if 20 workers were good, would 30 be better? More parallelism should mean faster synthesis, right?

The Benchmark Result

The assistant dutifully updated the configuration, restarted the daemon, waited for SRS preload, and ran the standard 5-proof batch benchmark with concurrency 3. The result was unambiguous — and disappointing:

60.4s/proof — significantly worse than pw=20 (44.0s/proof). With 30 concurrent partition workers, synthesis contention is hurting. The first proof took 140s prove time — that's 30 partitions being synthesized simultaneously across 3 sectors competing hard for CPU and memory.

This single sentence in message [msg 2239] encapsulates the entire diagnosis. The throughput regressed by 37% — from 44.0s to 60.4s per proof. The first proof in the batch was especially pathological: 140 seconds of prove time, compared to ~65–70 seconds in the pw=20 configuration. The assistant immediately identified the root cause: CPU contention.

The Reasoning Behind the Diagnosis

The assistant's reasoning reveals a deep understanding of the system's resource constraints. The key insight is that partition_workers does not control GPU parallelism — it controls CPU synthesis parallelism. Each partition's synthesis phase (the CPU-bound computation that generates the circuit's a/b/c vectors) runs on a separate thread. With partition_workers=30, the system attempts to synthesize 30 partitions simultaneously. But when running 3 concurrent proofs (j=3), each with 10 partitions, the total demand is 30 concurrent synthesis threads — and on a 96-core machine, that might seem fine at first glance.

However, the assistant recognized that synthesis is not purely CPU-bound in isolation. It contends for memory bandwidth, cache, and crucially, it starves the GPU preprocessing threads that run on the CPU. In the Phase 8 architecture, each GPU worker's "GPU time" measurement includes CPU preprocessing work (roughly 3.2 seconds per partition) that runs before the CUDA kernel. When 30 synthesis threads saturate the CPU, the GPU workers cannot get CPU time to prepare their next partition's data, creating a pipeline stall that cascades into longer per-proof wall time.

The assistant also noted the specific symptom: the first proof took 140 seconds of prove time. This is the "cold start" effect where all 30 partitions from the first three concurrent proofs begin synthesis simultaneously, overwhelming the CPU before any GPU work has completed to free resources. In the pw=20 configuration, only 20 synthesis threads compete, leaving enough CPU headroom for the GPU preprocessing threads to keep the GPU fed.

The Timeline Investigation

To validate this diagnosis, the assistant immediately issued a diagnostic command — grepping the daemon's TIMELINE log for GPU_START and GPU_END events:

[bash] grep "^TIMELINE" /tmp/cuzk-phase8-pw30.log | grep "GPU_START\|GPU_END" | head -30

This command extracts the first 30 timeline entries showing when each GPU worker started and finished processing partitions. The output reveals the severity of the contention. The first partition (partition 8, worker 0) took 22,306 ms of GPU time — more than triple the ~6,400–6,700 ms seen in the pw=20 configuration. Similarly, partition 7 on worker 1 took 22,019 ms. These inflated GPU durations confirm that the GPU workers are being starved: they acquire the mutex and start their CUDA work, but the CPU-side preprocessing that should have completed before acquiring the mutex is delayed because synthesis threads are consuming all available CPU resources.

The timeline data also shows the gap between GPU_START events: worker 0 starts partition 8 at timestamp 110,869, but worker 1 doesn't start partition 7 until 132,237 — a 21-second gap where only one GPU worker is active. In the pw=20 configuration, the dual-worker interlock kept both workers continuously busy with sub-20ms gaps between their respective starts.

Input Knowledge Required

To fully understand this message, the reader needs several pieces of context:

  1. The Phase 8 architecture: Two GPU workers per device share a narrowed C++ mutex. CPU preprocessing (3.2s per partition) runs outside the lock, while CUDA kernels (NTT+MSM, ~3.3s) run inside it. Workers interleave: one does CPU work while the other runs CUDA.
  2. The partition model: Each PoRep proof has 10 partitions. Each partition goes through CPU synthesis (generating circuit values) followed by GPU proving (preprocessing + CUDA kernels). partition_workers controls how many partitions are synthesized concurrently.
  3. The benchmark setup: 5 proofs (c=5) with concurrency 3 (j=3), meaning up to 3 proofs in flight simultaneously. With 10 partitions per proof, that's up to 30 partitions competing for synthesis threads.
  4. The machine's resources: 96 CPU cores (192 hyperthreads with rayon). The assistant has previously established that this machine has ample cores but that contention manifests in memory bandwidth and cache pressure, not just core availability.
  5. The Phase 7 baseline: Before the dual-worker interlock, Phase 7 achieved 50.7s/proof with pw=20. Phase 8 improved to 44.0s/proof by overlapping CPU and GPU work. The pw=30 regression to 60.4s/proof is worse than even Phase 7's baseline.

Output Knowledge Created

This message creates several important pieces of knowledge:

  1. The optimal partition_workers range is bounded above by CPU contention. The experiment empirically demonstrates that pw=30 is too high for this 96-core machine, causing throughput regression. This establishes an upper bound for the sweep that follows in [msg 2240] (chunk 1), where pw=10–12 emerges as optimal at 43.5s/proof.
  2. Synthesis parallelism has diminishing returns and can become negative. More synthesis threads do not always mean faster synthesis — beyond a saturation point, they cannibalize CPU resources needed by GPU preprocessing threads, creating a bottleneck shift from GPU-idle to CPU-starvation.
  3. The first-proof effect: The 140-second prove time on the first proof reveals a cold-start dynamic where the initial burst of concurrent synthesis creates a CPU contention spike that delays all subsequent pipeline stages. This has implications for production deployment: steady-state throughput may differ significantly from first-proof latency.
  4. A diagnostic methodology: The assistant demonstrates a pattern for investigating performance regressions — state the quantitative result, hypothesize the root cause, then immediately instrument with timeline data to validate. The grep command is not just data collection; it's a hypothesis test.

Assumptions and Mistakes

The message itself does not contain mistakes — the diagnosis is correct and the timeline data supports it. However, the experiment reveals an implicit assumption that deserves examination: the assumption that more partition workers would improve throughput. This assumption was held by both the user (who requested pw=30) and likely by the assistant (who executed the request without prior prediction of the outcome). The fact that the result was worse is what makes this message valuable — it disproves the naive scaling assumption and reveals the system's true bottleneck dynamics.

One subtle point: the assistant attributes the regression to "synthesis contention" broadly, but the timeline data shows inflated GPU durations (22s vs 6.5s per partition). This suggests the bottleneck is not just synthesis competing with synthesis, but synthesis starving the GPU preprocessing threads that run on the CPU. The GPU workers cannot prepare their next batch of data because the CPU is saturated with synthesis work. This is a more precise diagnosis than "synthesis contention" alone — it's a cross-pipeline resource conflict where CPU-bound synthesis work preempts the CPU-bound preprocessing that feeds the GPU.

The Thinking Process

The message reveals a clear diagnostic thinking process:

  1. Observe the anomaly: 60.4s/proof is significantly worse than the 44.0s/proof baseline. This is not a minor fluctuation — it's a 37% regression that demands explanation.
  2. Formulate a hypothesis: The assistant immediately connects the regression to the changed parameter. "With 30 concurrent partition workers, synthesis contention is hurting." This is a causal hypothesis: the increase from pw=20 to pw=30 caused CPU contention that degraded throughput.
  3. Identify a specific symptom: The first proof's 140-second prove time is singled out as evidence. This is a specific, measurable symptom that supports the hypothesis — if contention were uniform, all proofs would be equally slow, but the first proof bears the brunt because all 30 partitions start synthesis simultaneously.
  4. Design a diagnostic experiment: The grep command is chosen to extract GPU_START and GPU_END timeline events. This directly measures whether GPU workers are experiencing delays. The inflated 22-second GPU durations confirm the hypothesis.
  5. Communicate the result: The assistant presents the finding concisely, with the key number (60.4s/proof) bolded, the comparison to baseline (44.0s/proof), the hypothesized cause, and the supporting evidence — all in a single sentence followed by the diagnostic command.

Conclusion

Message [msg 2239] is a masterclass in performance diagnosis under constraint. In just a few lines, the assistant receives a negative result (pw=30 regresses), immediately identifies the root cause (CPU contention starving GPU preprocessing), validates it with timeline data, and communicates the finding with precision. The message serves as the critical data point that bounds the partition_workers parameter space, informing the subsequent sweep (chunk 1) that identifies pw=10–12 as the optimal range.

More broadly, this message illustrates a fundamental truth about systems optimization: parallelism is not free, and adding more concurrent work can backfire when resources are shared across pipeline stages. The Phase 8 dual-worker interlock successfully eliminated GPU idle gaps, but it did so by shifting the bottleneck to CPU resource contention — a trade-off that only becomes visible through systematic parameter exploration like this pw=30 experiment.