The Async Overlap Turning Point: Architecting Throughput for Continuous Proof Streams

In the ongoing development of the cuzk pipelined SNARK proving engine for Filecoin, a single message from the assistant marks a decisive architectural shift. After resolving a critical performance regression in the previous chunk—where per-partition synthesis had degraded single-proof latency from ~93 seconds to ~611 seconds—the assistant now turns its attention to the original goal of Phase 2: throughput on a continuous stream of proofs. Message 602 is the moment where the assistant lays out the design for true async overlap and begins reading the existing code to implement it. This message is small in surface area but enormous in strategic weight: it represents the transition from fixing a regression to building the core throughput architecture that justifies the entire Phase 2 pipeline effort.

The Message in Full

The assistant writes:

Now for the async overlap implementation. This is the key Phase 2 throughput win: when there's a continuous stream of proofs, we want to overlap synthesis of proof N+1 with GPU proving of proof N.

The design: - Engine has a bounded channel (synth_queue) between synthesis and GPU workers - A synthesis task pulls proof requests, runs synthesis on a blocking thread, pushes SynthesizedProof into the channel - GPU workers pull from the channel and run gpu_prove() - Backpressure: channel capacity = synthesis_lookahead (from config)

Let me now implement this in the engine. The key changes: 1. When pipeline_enabled, the worker loop changes to a two-stage pipeline 2. A dedicated synthesis task pulls from the scheduler and synthesizes 3. GPU workers pull synthesized proofs from the channel

The message then issues a read tool call on /home/theuser/curio/extern/cuzk/cuzk-core/src/engine.rs, reading lines 245–252 to inspect the existing worker spawning code that will be refactored.

The Reasoning and Motivation

To understand why this message was written, one must trace the arc of the preceding work. The assistant had just completed the batch-all-partitions synthesis mode (synthesize_porep_c2_batch) in the previous chunk, which restored single-proof latency to match the monolithic Phase 1 baseline (~91.2 seconds vs ~93 seconds). That fix was a necessary detour: the initial per-partition pipeline design, while architecturally clean, serialized 10 partitions sequentially, producing a 6.6× slowdown. The batch mode fixed this by running all partitions in a single rayon parallel synthesis call and a single GPU call.

But the batch mode is not the endgame. The entire purpose of Phase 2's pipeline architecture is not to match monolithic performance on a single proof—it is to exceed it on a continuous stream of proofs. The monolithic approach blocks the GPU while the CPU synthesizes the next proof, and vice versa. The pipelined approach, by contrast, can keep both the CPU and GPU busy simultaneously: while the GPU is proving proof N, the CPU can already be synthesizing proof N+1. This overlap is the "key Phase 2 throughput win" that the assistant refers to.

The motivation is thus deeply strategic. The assistant is not fixing a bug or adding a feature; it is executing on a deliberate architectural vision that was established much earlier in the conversation (see [msg 558] and surrounding context). The async overlap design is the culmination of the Phase 2 pipeline concept: split synthesis and GPU proving, then run them concurrently on separate hardware resources.

How Decisions Were Made

The message reveals a clear design decision process. The assistant enumerates four design elements:

A bounded channel (synth_queue) as the communication primitive between the synthesis stage and the GPU stage. This is a classic producer-consumer pattern, well-suited to Rust's async ecosystem where tokio::sync::mpsc channels are idiomatic.
A dedicated synthesis task that pulls proof requests from the scheduler (the existing priority queue) and runs synthesis on a blocking thread. This is necessary because synthesis is CPU-bound and potentially long-running—it cannot be run directly on an async task without blocking the runtime.
GPU workers that pull synthesized proofs from the channel and run gpu_prove(). The existing worker architecture already spawns one task per GPU; the change is to make these workers consume from the channel instead of directly pulling from the scheduler.
Backpressure via a configurable synthesis_lookahead parameter that limits channel capacity. This prevents the synthesis task from running too far ahead of the GPU, which would waste memory by holding many synthesized proofs in the channel. The decision to use a bounded channel rather than an unbounded one is notable. It reflects an awareness of the memory footprint of synthesized proofs—each SynthesizedProof contains circuit evaluations that can be substantial (the earlier analysis in segment 0 identified ~200 GiB peak memory for the monolithic pipeline). Bounding the channel ensures that the system cannot accumulate an unbounded backlog of synthesized proofs, providing a natural memory safety mechanism. The assistant also makes a key architectural decision: the worker loop changes to a "two-stage pipeline" when pipeline_enabled is true. This means the engine will have two modes—a direct mode (for backward compatibility or debugging) and the pipelined mode. The decision to keep both modes rather than replacing the worker architecture entirely shows engineering prudence: the batch-mode pipeline already works for single proofs, and the async overlap is an optimization on top of that foundation.

Assumptions Made

Several assumptions underpin this message:

Assumption 1: Synthesis time is comparable to GPU proving time. The overlap design only yields throughput gains if synthesis and GPU proving take roughly similar amounts of time. If one stage is dramatically faster than the other, the pipeline will be bottlenecked by the slower stage and the overlap benefit diminishes. The assistant's earlier measurements showed synthesis at ~55.7 seconds and GPU at ~35.2 seconds for PoRep C2—close enough that overlapping them could yield meaningful throughput improvements (potentially up to ~35% reduction in inter-proof latency).

Assumption 2: The channel-based producer-consumer pattern introduces negligible overhead. The assistant assumes that the cost of pushing SynthesizedProof objects through a channel and pulling them on the GPU side is small compared to the compute time of either stage. This is reasonable given that the data transfer is in-process (memory) rather than cross-process.

Assumption 3: A single synthesis task is sufficient. The design calls for one dedicated synthesis task, implying that a single CPU core (or a rayon thread pool) can keep up with all available GPUs. If the system has multiple GPUs, a single synthesis task might become a bottleneck. The assistant does not address multi-GPU synthesis scaling in this message, though the earlier multi-GPU worker pool (implemented in segment 6) suggests this may be addressed later.

Assumption 4: The scheduler interface is compatible with the two-stage pipeline. The assistant assumes that the existing scheduler (which queues proof requests with priority ordering) can be consumed by the synthesis task in the same way that GPU workers previously consumed it. This is likely true, but it implies that the scheduler's priority semantics are preserved across the pipeline—a high-priority proof submitted after a low-priority one should still be synthesized first.

Input Knowledge Required

To understand this message fully, a reader needs substantial context:

The Phase 2 pipeline architecture: That synthesis (CPU-bound circuit building) and GPU proving have been split into separate phases, with SynthesizedProof as the intermediate data structure.
The SRS manager: That shared reference strings (SRS) are preloaded and managed separately, avoiding reload overhead (the 20.5% speedup measured earlier).
The worker architecture: That the engine spawns one tokio task per GPU, each pulling from a shared priority scheduler.
The batch-mode pipeline: That the just-completed synthesize_porep_c2_batch function handles all 10 partitions in one call, and that this is the synthesis function the async overlap will invoke.
The config system: That pipeline_enabled and synthesis_lookahead are configuration parameters already defined in the engine's config struct.
The bellperson fork: That the assistant created a minimal fork of bellperson to expose the synthesis/GPU split APIs, and that this fork is what enables the pipeline design. Without this context, the message reads as a straightforward design sketch. With it, the message reveals itself as the capstone of a multi-segment architectural effort spanning hundreds of messages.

Output Knowledge Created

This message produces two things:

A design specification for the async overlap architecture, documented in the conversation. The four-point design (bounded channel, synthesis task, GPU consumers, backpressure) becomes the blueprint for the implementation that follows in subsequent messages.
A code reading of the existing worker spawning code (lines 245–252 of engine.rs). This reading is the first step of implementation—the assistant is orienting itself to the code that needs to be refactored. The specific lines read show the worker loop structure: each worker clones the scheduler, tracker, and shutdown receiver, then presumably enters a loop pulling proof requests. The message does not yet produce code changes—those will come in the following messages. But it establishes the conceptual framework that will guide those changes.

The Thinking Process Visible in the Reasoning

The assistant's reasoning is remarkably structured. It opens with a strategic framing ("Now for the async overlap implementation. This is the key Phase 2 throughput win"), immediately establishing priority and context. It then states the core insight ("when there's a continuous stream of proofs, we want to overlap synthesis of proof N+1 with GPU proving of proof N") in plain language, making the design motivation accessible.

The design points are enumerated with bullet-point clarity, each one building on the previous: channel → synthesis producer → GPU consumers → backpressure. The assistant then translates this design into concrete implementation steps ("The key changes: 1. When pipeline_enabled... 2. A dedicated synthesis task... 3. GPU workers pull..."), showing a clear mental model of how the existing code will be transformed.

The decision to read the engine.rs file immediately after stating the design is telling: the assistant is not just thinking abstractly—it is grounding the design in the actual code that needs to change. The read targets lines 245–252, which contain the worker spawning loop. This is precisely the code that will need to be restructured to support the two-stage pipeline. The assistant is verifying its mental model of the existing architecture before making changes.

Notably, the assistant does not second-guess itself. There is no hedging language, no "we could also consider" alternatives. The design is presented as settled. This confidence comes from the extensive groundwork laid in previous messages: the bellperson fork, the SRS manager, the batch-mode synthesis, and the PoSt/SnapDeals synthesis functions are all already in place. The async overlap is the final piece of the Phase 2 puzzle, and the assistant knows exactly what it needs to build.

Conclusion

Message 602 is a hinge point in the cuzk proving engine's development. It marks the transition from reactive problem-solving (fixing the per-partition performance regression) to proactive architecture-building (implementing the throughput-maximizing async overlap). The message is concise—barely 150 words of design text plus a file read—but it encapsulates the strategic vision that drives the entire Phase 2 effort. The assistant's clear reasoning, grounded design decisions, and confident execution planning make this message a model of technical communication in an AI-assisted coding session. The async overlap architecture it describes will ultimately enable the cuzk daemon to achieve its primary goal: continuous, memory-efficient proof generation that outperforms the monolithic baseline on throughput, completing the transformation from a batch-oriented proving system to a streaming pipeline optimized for the heterogeneous cloud rental markets that motivated the entire project.