The Moment the Pipeline Proved Itself: Validating Async Overlap on Real GPU Hardware

In the lifecycle of any systems engineering project, there comes a pivotal moment when theory meets reality — when an architectural design that has been carefully reasoned about, coded, reviewed, and committed must face the unforgiving judgment of actual hardware. Message [msg 651] in this opencode session captures precisely such a moment. The assistant has just completed implementing a two-stage async overlap pipeline for the cuzk proving engine, a component of the Filecoin Curio storage mining stack responsible for generating Groth16 zero-knowledge proofs. After restructuring the engine from monolithic per-GPU workers into a pipelined architecture with a dedicated CPU synthesis task feeding a bounded channel to GPU workers, the assistant now runs the definitive test: three consecutive 32 GiB PoRep C2 proofs on an RTX 5070 Ti, submitted concurrently to validate that the pipeline delivers its promised throughput improvement.

The Message: Running the Batch Pipeline Test

The message itself is concise but dense with significance. The assistant writes:

The batch command has --concurrency which controls how many proofs are submitted at once. For the pipeline test, I want to submit all 3 proofs upfront (-j 3), so the synthesis task can start pre-synthesizing while the GPU is busy:

>

``` # Batch test: 3 proofs, all submitted upfront (concurrency=3) echo "=== BATCH PIPELINE TEST (3 proofs, concurrent submission) ===" time /home/theuser/curio/extern/cuzk/target/release/cuzk-bench \ --addr http://127.0.0.1:9821 \ batch \ --type porep \ --c1 /data/32gbench/c1.json \ --miner-id 1000 \ --count 3 \ --concurrency 3 2>&1 === BATCH PIPELINE TEST (3 proofs, concurrent submission) === 2026-02-17T20:17:43.559586Z INFO cuzk_bench: loading C1 output path=/data/32gbench/c1.json === Batch Benchmark === proof type: porep count: 3 concurrency: 3

>

[1/3] COMPLETED — 91.1s (prove=35384 ms, queue=235 ms) [2/3] COMPLETED — 153.0s (prove=37148 ms, queue=55738 ms) [3/3] COMPLETED — 212.7s (prove=36319 ms, queue=115885 ms)

>

=== Batch Summary === total time: 2... ```

The output is truncated in the conversation data, but the critical numbers are visible: three proofs completing in 91.1s, 153.0s, and 212.7s respectively. The next message ([msg 652]) provides the full analysis, but the raw data is already telling a compelling story.

Why This Message Was Written: The Validation Imperative

To understand why this message exists, one must understand the architectural journey that preceded it. The cuzk proving engine, in its Phase 1 incarnation, used a straightforward monolithic worker model: each GPU worker would pull a job from the scheduler, run CPU-bound circuit synthesis, then run GPU-bound proving, then mark the job complete. This is simple, correct, and easy to reason about — but it leaves the GPU idle during the ~55 seconds of CPU synthesis, wasting the most expensive resource in the system.

The Phase 2 redesign, implemented across messages [msg 605] through [msg 635], introduced a fundamental architectural shift. The engine was split into two stages connected by a bounded tokio::sync::mpsc channel. A dedicated synthesis task runs on CPU threads, pre-synthesizing proofs and pushing them into the channel. Per-GPU workers pull synthesized jobs from the channel and run only the GPU proving phase. The bounded channel provides backpressure: if the GPU workers fall behind and the channel fills up (controlled by the synthesis_lookahead config parameter, defaulting to 1), the synthesis task blocks, preventing unbounded memory growth from pre-synthesized proofs piling up.

This architecture makes a clear performance prediction: under continuous load, the system should approach a steady state where the GPU is never idle, limited only by the slower of the two stages. Since synthesis takes ~55s and GPU proving takes ~35s, the pipeline should be synthesis-bound, yielding a steady-state throughput of approximately one proof every ~55-60 seconds, compared to ~90 seconds in the monolithic model.

But a prediction is not a proof. The assistant had already validated a single proof ([msg 649]) to confirm the basic plumbing worked — the daemon started, the pipeline mode engaged, a single PoRep completed in 90s with the expected timing breakdown (synth=55s, gpu=35s). However, a single proof cannot demonstrate overlap. The pipeline's benefit only manifests when there are multiple proofs in flight simultaneously, allowing the synthesis of proof N+1 to overlap with the GPU proving of proof N. This message is the deliberate, designed experiment to test that exact claim.## How Decisions Were Made: The Concurrency Choice

A subtle but critical decision is visible in the message: the assistant chooses --concurrency 3 (submitting all three proofs at once) rather than --concurrency 1 (submitting them one at a time). This is not an arbitrary choice — it is a deliberate design decision that makes the pipeline test meaningful.

With --concurrency 1, the bench tool would submit the first proof, wait for it to complete, submit the second, wait, submit the third. In that mode, the pipeline would behave identically to the monolithic worker: each proof would be synthesized and proved sequentially, because the scheduler would only ever see one job at a time. The synthesis task would synthesize proof 1, the GPU worker would prove it, and only then would proof 2 be submitted to the scheduler. The overlap would never materialize.

With --concurrency 3, all three proofs are submitted to the scheduler immediately. The synthesis task can begin synthesizing proof 2 while the GPU is still working on proof 1. This is the condition required for the pipeline to demonstrate its value. The assistant's reasoning — "so the synthesis task can start pre-synthesizing while the GPU is busy" — shows an understanding that the test harness must be configured to exercise the specific concurrency pattern the architecture was designed for.

This decision also reveals an assumption: that the bench tool's --concurrency parameter maps directly to how many jobs are in-flight in the scheduler at once. The assistant had just learned the batch command's interface ([msg 650]) and correctly inferred that concurrency controls the submission pattern. This assumption proved correct, as the results show the expected queue times growing (235ms, 55s, 115s) — indicating that proofs 2 and 3 spent significant time waiting in the scheduler queue while the GPU was busy, exactly as the pipeline design intended.

The Results: Reading the Numbers

The output from the batch test tells a clear story. The first proof completes in 91.1s with only 235ms of queue time — it was submitted, synthesized, and proved immediately with no contention. This is the "cold pipeline" case, essentially identical to the monolithic baseline.

The second proof completes at 153.0s, which is 61.9s after the first. This is dramatically faster than the 90s sequential baseline. The queue time of 55.7s reveals what happened: the GPU was busy proving proof 1 while the synthesis task was synthesizing proof 2. Proof 2 spent 55.7s waiting for the GPU to become available, and then the GPU proved it in 37.1s. The total wall-clock time from submission to completion for proof 2 was 153.0s, but the incremental time between proofs 1 and 2 was only 61.9s — very close to the synthesis time of ~55s.

The third proof completes at 212.7s, 59.7s after the second. Again, the queue time of 115.9s reflects the cumulative wait: proof 3 was synthesized while proofs 1 and 2 were being proved, and then waited for the GPU to become available. The incremental time of 59.7s confirms the steady state.

These numbers validate the core architectural thesis. The steady-state throughput is approximately 60s per proof, compared to 90s in the monolithic model — a 1.27x speedup. The assistant's analysis in the subsequent message ([msg 652]) quantifies this precisely: "Without pipeline: would be 3 × 90s = 270s. With pipeline: 212.7s = 1.27x speedup."

Assumptions and Their Validation

The message rests on several assumptions that deserve examination. First, the assistant assumes that the GPU proving time is consistent across proofs. The prove times of 35,384ms, 37,148ms, and 36,319ms confirm this — they vary by less than 5%, indicating stable GPU performance with no thermal throttling or memory contention.

Second, the assistant assumes that the synthesis time is the binding constraint. The steady-state incremental times of ~60s versus synthesis time of ~55s confirm this, with the ~5s difference attributable to scheduling overhead and channel management.

Third, the assistant assumes that the bounded channel provides adequate backpressure. With synthesis_lookahead = 1 (the default), the channel can hold at most one synthesized job. When the GPU is busy and the channel is full, the synthesis task blocks. This prevents the memory blow-up that would occur if synthesis ran unboundedly ahead of the GPU. The fact that the daemon did not OOM during three consecutive 32 GiB proofs (each requiring ~200 GiB peak memory) validates this assumption.

Fourth, the assistant assumes that the cuzk-bench batch command with --concurrency 3 correctly simulates a continuous workload. This is a reasonable assumption for a validation test, but it differs from the real-world pattern where proofs arrive at unpredictable intervals from upstream Curio tasks. The real benefit would manifest when the scheduler always has queued jobs — a condition the bench tool creates artificially by submitting all proofs at once.

Input Knowledge Required

To fully understand this message, one needs knowledge of several domains. The Groth16 proof generation pipeline for Filecoin's Proof-of-Replication (PoRep) involves CPU-bound circuit synthesis (building the rank-1 constraint system from the sector data) followed by GPU-bound multi-scalar multiplication (MSM) and number-theoretic transform (NTT) operations. The C1 JSON file at /data/32gbench/c1.json contains the intermediate circuit values from the first phase of proving, which the C2 phase consumes. The SRS (Structured Reference String) is a ~45 GiB parameter file that must be loaded into GPU memory before proving can begin — the daemon's preload step took 15.3s in the startup logs ([msg 644]).

The concept of pipeline overlap — decoupling CPU-bound and GPU-bound work to keep both resources busy — is a classic systems optimization, but its application to zero-knowledge proof generation is relatively novel. The Filecoin network requires storage miners to generate proofs within a deadline window, so reducing per-proof latency directly impacts the miner's ability to seal sectors and earn rewards.

Output Knowledge Created

This message produces validated empirical evidence that the async overlap pipeline works correctly on real hardware. The 1.27x speedup on three proofs is modest but meaningful — and importantly, it is a lower bound. With longer runs (dozens or hundreds of proofs), the steady-state throughput would approach the theoretical limit of ~55s per proof (synthesis-bound), yielding a 1.64x speedup over the 90s baseline. The assistant notes this in the follow-up message: "The steady-state is ~60s/proof, which is very close to the synthesis time of 55s."

The message also creates confidence in the bounded channel design. The queue times grow linearly (235ms, 55.7s, 115.9s), exactly as expected for a system where the bottleneck (GPU) is slower than the feeder (synthesis) and the channel capacity is 1. This linear growth in queue time with proof index is a hallmark of correct pipeline behavior — it confirms that proofs are being synthesized ahead of time and waiting their turn for the GPU, rather than being synthesized on-demand.

The Thinking Process: What the Message Reveals

The assistant's reasoning is visible in the brief commentary that precedes the command: "For the pipeline test, I want to submit all 3 proofs upfront (-j 3), so the synthesis task can start pre-synthesizing while the GPU is busy." This sentence encapsulates the entire architectural insight. The assistant understands that the pipeline's benefit depends on having multiple jobs in-flight simultaneously, and chooses the test parameters accordingly.

The message also reveals a methodical approach to validation. The assistant had already run a single proof test ([msg 649]) to verify basic functionality. Only after that succeeded did the assistant proceed to the multi-proof batch test. This two-step validation — first confirm the system works, then confirm the optimization works — is a hallmark of disciplined engineering.

The truncated output (the batch summary is cut off) is a minor frustration, but the visible data is sufficient to draw conclusions. The assistant in the next message ([msg 652]) performs a detailed breakdown, calculating incremental times and comparing to the sequential baseline. The fact that the assistant immediately follows up with a log analysis (grep -E "synthesis complete|GPU worker picked|proof completed") shows a commitment to verifying not just the aggregate numbers but the internal dynamics — confirming that synthesis of proof N+1 truly began while GPU was proving proof N.

Conclusion

Message [msg 651] is the validation point for a significant architectural transformation. It represents the moment when a carefully designed pipeline — with its dedicated synthesis task, bounded channel, backpressure semantics, and per-GPU workers — met real hardware and real proofs, and the numbers matched the theory. The 1.27x speedup is not just a performance improvement; it is evidence that the architectural reasoning was sound, that the implementation was correct, and that the cuzk project is on track for its next phase of optimization.