The First Test of Parallel Synthesis: When Theory Meets CPU Contention

Introduction

In the high-stakes world of Filecoin proof generation, every second counts. The cuzk proving engine, a custom Groth16 prover for Filecoin's Proof-of-Replication (PoRep), had reached an impasse. After weeks of optimization—including Phase 5's Pre-Compiled Constraint Evaluator (PCE), Phase 6's slotted partition pipeline, and meticulous waterfall timeline instrumentation—a clear bottleneck had emerged: the GPU was idle for 12–14 seconds between every proof. The cause was architectural: CPU synthesis (39 seconds) ran strictly sequentially and took longer than GPU proving (27 seconds), leaving the GPU starved for work.

The solution seemed obvious: run multiple syntheses in parallel. If two CPU synthesis tasks could run concurrently, a new proof would be ready every ~20 seconds instead of every ~39 seconds, keeping the GPU continuously fed. The assistant had just implemented this parallel synthesis feature using a tokio::sync::Semaphore and a new synthesis_concurrency configuration parameter. Message 1871 records the very first benchmark run of this new capability—a moment of truth where theory collides with reality.

The Message: Running the First Parallel Synthesis Benchmark

The message itself is deceptively simple. It contains two bash commands and their output. The first command clears the timeline log file to ensure clean data. The second launches the benchmark suite:

cd /home/theuser/curio/extern/cuzk && \
  ./target/release/cuzk-bench batch -t porep --c1 /data/32gbench/c1.json -j 3 -c 5 2>&1 | tee /tmp/cuzk-parallel-bench.txt

The flags tell the story: -j 3 means three concurrent in-flight proofs from the client side, -c 5 means five proofs total. The assistant chose three concurrent clients based on the assumption that with synthesis_concurrency=2 (two parallel syntheses), the pipeline would need at least three in-flight requests to stay fully saturated—one being synthesized, one waiting in the channel, and one being proven on the GPU.

The results are revealing:

  [1/5] COMPLETED — 67.8s (prove=29065 ms, queue=266 ms)
  [2/5] COMPLETED — 106.3s (prove=38358 ms, queue=13385 ms)
  [3/5] COMPLETED — 141.2s (prove=35082 ms, queue=58496 ms)
  [4/5] COMPLETED — 155.5s (prove=31204 ms, queue=26171 ms)
  [5/5] COMPLETED — 86.1s (prove=26069 ms, queue=...)

At first glance, the numbers are concerning. The first proof completes in 67.8 seconds—a reasonable time that includes daemon startup overhead. But the second proof takes 106.3 seconds, the third 141.2 seconds, and the fourth 155.5 seconds. The queue times are ballooning: from 266 ms for the first proof to 58.5 seconds for the third. Something is clearly wrong.

The Reasoning Behind the Configuration

The assistant's choice of -j 3 was not arbitrary. It followed from a careful analysis of the pipeline's dynamics. The waterfall instrumentation from the previous benchmark (message 1854) had shown that with sequential synthesis, the GPU processed proofs in a strict one-at-a-time cadence: synth(N) → GPU(N) → synth(N+1) → GPU(N+1). The GPU idle gap of 12 seconds per proof was the direct consequence of synthesis being the bottleneck.

The theoretical model for parallel synthesis predicted a much better timeline:

Synth A:  [====P1=====][====P3=====][====P5=====]
Synth B:  [====P2=====][====P4=====]
GPU:            [==P1==][==P2==][==P3==][==P4==][==P5==]

With two concurrent syntheses, each taking ~39 seconds, a new proof would arrive every ~20 seconds—less than the GPU's 27-second processing time. The GPU would never idle. The assistant calculated that -j 3 would provide enough pipeline depth: one proof being synthesized, one in the channel buffer, and one on the GPU.

But this model contained a critical flaw: it assumed that synthesis time and GPU time would remain unchanged when run concurrently. This assumption would prove to be incorrect.

The Hidden Assumption: CPU Cores as a Shared Resource

The assistant's reasoning implicitly assumed that CPU synthesis and GPU proving operate on independent resources. In reality, they share the same 96-core AMD Threadripper PRO 7995WX. The GPU proving phase is not purely GPU-bound—it includes a significant CPU step called b_g2_msm (multi-threaded Pippenger multi-scalar multiplication on the G2 curve), which runs on the full rayon thread pool and takes approximately 25 seconds per proof.

When two syntheses run concurrently, each spawns rayon-parallel work across all available cores. Simultaneously, the GPU prover's b_g2_msm step also demands CPU time. The result is a three-way CPU contention: two syntheses fighting each other and both fighting the GPU's CPU-bound work. This contention inflates both synthesis time (from 39 seconds to 50+ seconds) and GPU time (from 27 seconds to 33+ seconds).

The benchmark results reflect this. The first proof (67.8s) benefits from a cold start where only one synthesis runs initially. But as the pipeline fills up—proof 2 starts synthesis while proof 1 is on the GPU—the contention kicks in. Proof 2's synthesis takes 38.3 seconds of prove time (GPU + CPU overhead), but proof 3's synthesis is delayed by 58.5 seconds of queue time because both synthesis slots are occupied by proofs 2 and 3 contending for CPU.

What the Message Reveals About the Thinking Process

The assistant's choice to run with -j 3 reveals a pattern of thinking that is both sophisticated and incomplete. On one hand, the assistant correctly identified the GPU idle gap, designed a targeted fix (parallel synthesis), and reasoned about pipeline depth requirements. The waterfall instrumentation—a custom timeline logging system emitting structured TIMELINE,offset,event,job_id,detail lines—was a brilliant diagnostic tool that made the bottleneck visible.

On the other hand, the assistant underestimated the impact of CPU contention. The mental model treated CPU synthesis and GPU proving as independent pipeline stages, when in fact they share the CPU as a contested resource. This is a classic systems thinking blind spot: optimizing one part of a pipeline without considering how the parts interact when parallelized.

The assistant also made a subtle error in benchmark design. Running only 5 proofs (-c 5) with a warm-up penalty (first proof includes SRS loading and daemon initialization) produces noisy data. The subsequent messages (1872–1882) show the assistant realizing this and running longer benchmarks with 8 proofs, as well as testing different -j values (2, 3, 4) to find the sweet spot.

Input Knowledge Required

To fully understand this message, one needs:

  1. The waterfall analysis (message 1854): The previous benchmark revealed the 12-second GPU idle gap and the sequential synthesis bottleneck. Without this context, the parallel synthesis fix seems unmotivated.
  2. The parallel synthesis implementation (messages 1862–1865): The assistant refactored the engine's synthesis task loop to use a tokio::sync::Semaphore, added the synthesis_concurrency config parameter, and changed the process_batch call from a blocking await to a spawned task.
  3. The hardware configuration: 96-core AMD Threadripper, 754 GiB RAM, RTX 5070 Ti GPU. The large core count makes CPU contention a significant factor.
  4. The proof pipeline architecture: The GPU prover's b_g2_msm step runs on CPU during the GPU phase, creating a shared resource dependency between synthesis and proving.
  5. The benchmark tool: cuzk-bench batch with -j (client concurrency) and -c (proof count) flags. The client concurrency controls how many proofs are in-flight from the bench tool's perspective.

Output Knowledge Created

This message produces several important outputs:

  1. Empirical evidence that parallel synthesis works but has side effects: The GPU is no longer idle between proofs (the waterfall in message 1872 confirms this), but CPU contention inflates both synthesis and GPU times.
  2. A clear signal that -j 3 is too aggressive: The ballooning queue times and increasing proof completion times indicate that three concurrent clients over-saturate the CPU.
  3. Data for the optimization trade-off analysis: The assistant now has concrete numbers to compare sequential vs. parallel synthesis across different concurrency levels.
  4. A diagnostic pattern: The combination of queue time and prove time in the benchmark output provides a fingerprint for identifying pipeline bottlenecks. Queue time growth indicates that the synthesis pipeline cannot keep up with demand.

Mistakes and Incorrect Assumptions

The primary mistake in this message is the choice of -j 3. The assistant assumed that the pipeline needed three in-flight proofs to stay saturated, but this ignored the CPU contention between synthesis and GPU proving. The correct approach—which the assistant discovers in subsequent messages—is to use -j 2 with synthesis_concurrency=2, which yields 42.2 seconds per proof (a 7% improvement over the 45.3-second baseline) with less contention.

A secondary issue is the short benchmark length. Five proofs with a warm-up penalty produce noisy data. The assistant corrects this in message 1873 by running 8-proof benchmarks.

The assistant also initially misattributed the slowdown to the synthesis_lookahead parameter (channel buffer size), testing both lookahead=2 and lookahead=1 before realizing the root cause was CPU contention.

Conclusion

Message 1871 captures a pivotal moment in the optimization journey: the first real-world test of a theoretically sound optimization that reveals an unexpected bottleneck. The parallel synthesis feature did exactly what it was designed to do—it eliminated GPU idle gaps—but it simultaneously created a new problem: CPU contention between synthesis and the GPU prover's b_g2_msm step. The net throughput improvement was modest (5–7%) rather than the dramatic gain predicted by the simple pipeline model.

This is the essence of systems optimization: every change reveals new constraints. The waterfall instrumentation made the GPU idle gap visible; parallel synthesis closed that gap but exposed CPU contention as the next bottleneck. The message demonstrates the importance of measuring real system behavior rather than relying on theoretical models, and it sets the stage for the deeper investigation into CPU resource management that follows in subsequent messages.