The Pivot to Async Overlap: Architecting a Two-Stage Proving Pipeline

Introduction

In the course of building a pipelined SNARK proving engine for Filecoin's proof-of-replication (PoRep) and proof-of-spacetime (PoSt) circuits, a single message marked a critical inflection point. After successfully validating a batch-mode pipeline that matched monolithic proving performance at 91.2 seconds per proof, the assistant turned its attention to the next architectural challenge: true asynchronous overlap between CPU-bound synthesis and GPU-bound proving. Message [msg 603] captures this moment of deliberate reasoning—a pause to assess complexity before diving into implementation.

The message is brief but dense. It contains no code, no tool calls, and no commands. It is pure architectural reasoning: a recognition that the next step is not a simple refactor but a "significant architectural change." This article examines why that message was written, what decisions it frames, the assumptions it carries, and the thinking process it reveals.

Context: Where We Are in the Pipeline

To understand message [msg 603], we must understand what preceded it. The assistant had just completed a major milestone in Phase 2 of the cuzk proving engine project. Earlier in the same segment ([msg 600]), an end-to-end GPU test of the batch-mode PoRep C2 pipeline had succeeded spectacularly: a valid 1920-byte Groth16 proof produced in 91.2 seconds, matching the monolithic Phase 1 baseline of ~93 seconds and representing a 6.7× improvement over the earlier per-partition sequential approach that took 611 seconds.

This success was the culmination of a multi-step effort. The initial pipeline implementation ([msg 564]) had introduced a per-partition approach where each of the 10 partitions in a 32 GiB PoRep was synthesized and proved sequentially. While architecturally elegant—it allowed synthesis of one partition to overlap with GPU proving of another—it proved disastrous for single-proof latency. The assistant had diagnosed this correctly: per-partition pipelining is designed for throughput on a stream of proofs, not for single-proof latency. The fix was to add a batch-all-partitions mode (synthesize_porep_c2_batch) that synthesizes all 10 partitions in a single rayon parallel call and proves them in one GPU call.

With that fix validated, the assistant updated the todo list ([msg 601]) and then, in [msg 602], announced the next goal: "Now for the async overlap implementation. This is the key Phase 2 throughput win: when there's a continuous stream of proofs, we want to overlap synthesis of proof N+1 with GPU proving of proof N."

The Message Itself: A Moment of Deliberate Reasoning

Message [msg 603] begins with a striking admission: "Looking at this more carefully, the true async overlap is a significant architectural change that intersects with the current single-worker-per-GPU design."

This sentence reveals that the assistant had been holding a simplified mental model of what the async overlap would entail. The phrase "more carefully" signals that upon closer examination—likely after reading the engine's worker architecture in engine.rs—the assistant realized the change was deeper than initially assumed. The word "intersects" is key: the async overlap is not a bolt-on addition but a transformation that touches the core worker loop, job tracking, completion signaling, and memory management.

The message then lays out the architectural contrast in stark terms:

Current architecture: Each GPU worker does: pull request → (synthesize + GPU prove) → complete

Desired: Synthesis task does pull request → synthesize → push to channel. GPU worker does pull synthesized → GPU prove → complete.

This is the heart of the message. The assistant is decomposing a monolithic worker loop into a two-stage pipeline with a bounded channel between stages. The current architecture is simple: one worker task per GPU, each worker pulling a proof request from a shared priority scheduler, running the full synthesis+prove cycle, and completing the job. The desired architecture separates concerns: a dedicated synthesis task handles the CPU-bound circuit construction, while GPU workers focus exclusively on the GPU-bound Groth16 proving. The bounded channel (synth_queue) provides both communication and backpressure.

The Complication: What Makes This Hard

The assistant identifies three specific complications:

Job tracking across stages: The current architecture has a single JobTracker that tracks each proof from submission to completion. In the two-stage design, a job's state must be maintained across two independent tasks—the synthesis task holds it during CPU work, then the GPU worker takes ownership for GPU proving. The completion channel, timing instrumentation, and error handling must all span this boundary.
Completion channels: The gRPC API includes an AwaitProof RPC that allows clients to wait for a proof result. This is implemented via per-job completion channels. In the two-stage design, the completion channel must be wired through both stages—the GPU worker must be the one to signal completion, not the synthesis task.
Memory pressure: The synthesis result for PoRep 32G is approximately 136 GiB (all 10 partitions' intermediate state: a/b/c evaluations, density trackers, and witness assignments). The bounded channel can only hold synthesis_lookahead of these in flight at once. This is a critical design constraint: if synthesis_lookahead is 2, the system could hold 272 GiB of intermediate state in memory, on top of the SRS parameters and other allocations. The assistant's reasoning about memory is particularly sharp. The 136 GiB figure comes from earlier analysis in the session: each partition produces ~13.6 GiB of intermediate state (a/b/c vectors at 32 bytes each plus auxiliary assignments), and 10 partitions in batch mode produce ~136 GiB total. This is not a number the assistant invented on the spot—it was established through careful measurement and analysis in earlier phases of the project.

The Key Insight: Single-GPU as the Common Case

The message concludes with a crucial architectural insight: "for a single GPU (most common case), the pipeline overlap happens naturally if we have a separate synthesis thread that stays one proof ahead. For multi-GPU, each GPU pulls from the same synth_queue."

This insight simplifies the design considerably. In the single-GPU case (which is expected to be the most common deployment—a single GPU rented in a cloud environment), the overlap is straightforward: the synthesis task works ahead, keeping one proof synthesized and ready in the channel while the GPU works on the current proof. When the GPU finishes, it immediately pulls the next synthesized proof from the channel. The synthesis task, meanwhile, has already started on the next proof. This creates a natural pipeline where the critical path is the longer of synthesis time and GPU time, rather than their sum.

For multi-GPU, the design generalizes naturally: a single synthesis task feeds multiple GPU workers from the same channel. This is a producer-consumer pattern where the synthesis task is the sole producer and each GPU worker is a consumer. The bounded channel provides load-leveling: if GPU workers are faster than synthesis, they'll stall waiting for new work; if synthesis is faster, the channel fills up and provides backpressure.

Assumptions Embedded in the Message

Several assumptions underlie the reasoning in [msg 603]:

Assumption 1: The synthesis task is the bottleneck. The assistant implicitly assumes that synthesis takes longer than GPU proving for PoRep C2 (55.7s vs 35.2s, from the batch-mode test). This means the pipeline will be synthesis-bound, and the GPU will periodically wait for new work. If the ratio were reversed, the design would need different tuning.

Assumption 2: A single synthesis task is sufficient. The assistant assumes one synthesis task can feed all GPU workers. For a system with many GPUs (say, 8+), a single synthesis task might not keep up. The design doesn't yet account for multiple synthesis tasks or partition-level parallelism across GPUs.

Assumption 3: The bounded channel is the right abstraction. The assistant chooses a channel-based design with synthesis_lookahead capacity. This assumes that the channel provides sufficient backpressure and that the synthesis_lookahead config value (from Config.pipeline.synthesis_lookahead) is a reasonable control knob.

Assumption 4: Job tracking can be cleanly split across stages. The assistant acknowledges this as a complication but assumes it can be solved with careful wiring of completion channels and timing instrumentation.

What the Message Does Not Say

Notably, the message does not address several important questions:

Error handling: If synthesis fails, how does the error propagate to the GPU worker and then to the completion channel? If a GPU worker crashes, how does the synthesis task learn about it?
Shutdown coordination: The engine has a graceful shutdown mechanism via a watch channel. How does the two-stage pipeline shut down cleanly—does the synthesis task stop accepting new work first, then drain the channel?
Fairness and priority: The scheduler supports priority ordering. In the two-stage design, the synthesis task pulls from the scheduler, but the GPU workers pull from the channel in FIFO order. Does this preserve the priority semantics? If a high-priority job arrives after a low-priority job has been synthesized, can it jump the queue in the GPU stage?
SRS loading: The SRS manager loads parameters on demand per circuit type. In the two-stage design, the synthesis task needs the SRS for circuit construction, but the GPU worker also needs it for proving. How is SRS loading coordinated between the two stages? These unanswered questions are not mistakes—they reflect the assistant's methodology of reasoning step by step. The message is a planning step, not a specification. The assistant is identifying the core architectural pattern before diving into implementation details.

Input Knowledge Required

To fully understand message [msg 603], a reader needs knowledge of:

The cuzk engine architecture: The current worker-per-GPU design with a shared priority scheduler, job tracker, and completion channels. This was established in Phase 0 and Phase 1 of the project.
The batch-mode pipeline results: The 91.2s total time, with 55.7s synthesis and 35.2s GPU breakdown. These numbers inform the assumption that synthesis is the bottleneck.
The memory footprint of intermediate state: The 136 GiB figure for PoRep 32G batch synthesis, which constrains the channel capacity and lookahead.
The bellperson fork: The split synthesis/GPU API (synthesize_circuits_batch and prove_from_assignments) that makes the two-stage design possible. Without this fork, the assistant would not be able to separate synthesis from GPU proving.
The SynthesizedProof type: Defined in pipeline.rs, this is the data structure that carries intermediate state from synthesis to GPU proving. It includes circuit ID, provers, input/aux assignments, r/s scalars, and timing information.
The synthesis_lookahead config value: Defined in Config.pipeline.synthesis_lookahead, this controls how many synthesized proofs can be in flight at once.

Output Knowledge Created

Message [msg 603] creates several pieces of output knowledge:

The two-stage architecture pattern: A clear articulation of the desired architecture (synthesis task → bounded channel → GPU workers) versus the current architecture (monolithic worker loop).
The complication taxonomy: Three specific challenges (job tracking, completion channels, memory pressure) that must be solved.
The single-GPU insight: The recognition that the common case (single GPU) simplifies the design because a single synthesis thread staying one proof ahead creates natural overlap.
The multi-GPU generalization: The insight that multiple GPU workers can pull from the same bounded channel, making the design scale.
The memory constraint: The explicit statement that synthesis result size (~136 GiB for PoRep 32G) limits the number of in-flight synthesized proofs.

The Thinking Process: A Window into Architectural Reasoning

The message reveals a distinctive thinking process. The assistant begins with a meta-cognitive observation ("Looking at this more carefully...") that signals a shift from a simplified model to a more nuanced understanding. This is followed by a structured decomposition: current architecture → desired architecture → complications → key insight.

The use of bullet points and bold formatting is not decorative—it reflects the assistant's internal organization of the problem space. The "Current architecture" and "Desired" sections are presented as code-like flow diagrams, showing the transformation from sequential to pipelined execution.

The phrase "Let me implement this cleanly" is telling. It suggests the assistant has moved from analysis to synthesis—having identified the complications, it now believes it understands the problem well enough to proceed. The "key insight" paragraph serves as the foundation for the implementation: the single-GPU case is the design center, and multi-GPU is a natural generalization.

The message ends with a [read] command to examine engine.rs, indicating the assistant is about to begin implementation. This is the transition from planning to coding—the message is the last moment of pure reasoning before the hands-on work begins.

Significance in the Larger Project

Message [msg 603] sits at a pivotal point in the Phase 2 implementation. The assistant has just proven that the batch-mode pipeline works correctly and matches monolithic performance. The next step—async overlap—is the feature that justifies the entire pipeline architecture. Without async overlap, the pipeline is merely a code reorganization that exposes synthesis/GPU timing. With async overlap, it becomes a throughput multiplier: instead of taking ~91 seconds per proof (synthesis + GPU), the system can approach ~55 seconds per proof (synthesis-bound) in steady state, because the GPU is kept busy while the CPU works ahead.

The message also demonstrates a disciplined engineering approach: validate the basic pipeline first, then add optimization. The assistant did not attempt to implement async overlap before proving that the batch-mode pipeline produced correct proofs. This sequential validation—correctness first, then performance—is a hallmark of careful systems engineering.

Conclusion

Message [msg 603] is a brief but dense piece of architectural reasoning that marks the transition from a working batch-mode pipeline to the more ambitious async overlap design. It identifies the core architectural pattern, acknowledges the complications, and distills the key insight that makes the design tractable. For a reader unfamiliar with the project, it offers a window into how a complex systems engineer thinks about decomposing a monolithic worker loop into a pipelined producer-consumer architecture, balancing throughput, memory pressure, and design simplicity. The message is not about code—it is about structure, about the shape of the solution before any code is written.