The SynthesizedJob Bridge: Architecting Async Overlap in a GPU Proving Pipeline

Introduction

In the span of a single, deceptively brief message, a critical architectural transformation took place. Message <msg id=619> contains just two sentences of reasoning followed by an edit to engine.rs — yet that edit represents the culmination of an entire design phase for the cuzk proving engine. The assistant wrote: "Now I need to add the SynthesizedJob type that carries both the synthesized proof and the job metadata through the channel, and restructure the engine's start() method. Let me rewrite the engine significantly." This seemingly simple statement encapsulates a profound shift: from a sequential, per-worker proving model to a true two-stage asynchronous pipeline where CPU-bound synthesis and GPU-bound proving execute concurrently, connected by a bounded channel.

To understand why this message matters, one must appreciate the problem it solves. The cuzk engine generates Groth16 proofs for Filecoin's Proof-of-Replication (PoRep) and related proof types. Each proof requires two computationally expensive phases: synthesis (CPU-bound, using ~142 cores and ~200 GiB of RAM) and GPU proving (GPU-bound, using one or more NVIDIA GPUs). In the Phase 1 architecture, each GPU worker performed these phases sequentially — synthesize a proof, then prove it, then move to the next. This meant the GPU sat idle during synthesis, and the CPU sat idle during GPU proving. The goal of Phase 2 was to overlap these phases so that while the GPU is proving proof N, the CPU is already synthesizing proof N+1.

The Context: A Multi-Step Implementation

Message <msg id=619> did not appear in a vacuum. It was the second of two edits applied to engine.rs within a single round, following an extensive preparatory sequence. In the preceding messages, the assistant had:

1. Verified the workspace state and confirmed all 15 unit tests passed (<msg id=610>).
2. Committed the batch-mode pipeline rewrite for all four proof types (<msg id=612>).
3. Read and analyzed the full codebase — engine.rs, pipeline.rs, config.rs, types.rs, scheduler.rs, srs_manager.rs, and lib.rs (<msg id=615>–<msg id=616>).
4. Articulated a detailed design for the async overlap architecture (<msg id=617>).
5. Applied a first edit to engine.rs that began the restructuring (<msg id=618>). The first edit (<msg id=618>) had already begun modifying engine.rs, but the assistant realized mid-implementation that a critical piece was missing: a channel message type that could carry both the synthesized proof data and the associated job metadata through the tokio::sync::mpsc channel. This is the insight that drives message <msg id=619>.

Why This Message Was Written: The Missing Type

The core architectural insight is that the bounded channel connecting the synthesis task to the GPU workers must carry composite payloads. A SynthesizedProof alone is insufficient — the GPU worker needs to know which job it's completing, what kind of proof it's generating, what job ID to report back, and how to route the completed proof to the correct response channel. The assistant recognized this gap after applying the first edit and realized that a SynthesizedJob wrapper type was necessary.

This type serves as the bridge between the two pipeline stages. It must be:

• Self-contained: carrying all information the GPU worker needs to execute the prove phase and report completion.
• Channel-compatible: implementing the traits required by tokio::sync::mpsc (which requires Send and ownership transfer).
• Backpressure-aware: the bounded channel capacity (synthesis_lookahead) limits how many synthesized proofs can be queued, preventing OOM from runaway synthesis. The assistant's decision to create this type reflects a fundamental systems design principle: define the data contract before implementing the control flow. By naming SynthesizedJob explicitly, the assistant established the interface between the two pipeline stages, making the rest of the restructuring coherent.

How Decisions Were Made

The assistant's decision-making process is visible across the preceding messages, culminating in <msg id=619>. Several key decisions were made:

1. Single synthesis task vs. per-GPU synthesis tasks. In <msg id=617>, the assistant initially considered per-GPU synthesis tasks but then refined to a single synthesis task feeding a shared channel. The reasoning is sound: PoRep synthesis uses ~142 CPU cores and ~200 GiB RAM — running multiple syntheses in parallel would not improve throughput (they'd compete for the same CPU cores and memory bandwidth) and would dramatically increase peak memory. A single synthesis task is the correct design.

2. Bounded channel with backpressure. The assistant chose tokio::sync::mpsc with a configurable capacity (synthesis_lookahead). This is a deliberate trade-off: a larger lookahead allows more synthesis-to-GPU overlap but increases peak memory (each synthesized proof in the channel holds ~20 GiB of intermediate state). The default of 1 provides one proof of overlap while keeping memory bounded.

3. Shared channel for all GPUs. Rather than per-GPU channels, the assistant opted for a single shared channel. This simplifies the synthesis task (it doesn't need to route to specific GPUs) and allows dynamic load balancing — if one GPU finishes faster, it can grab the next job from the shared queue.

4. Spawn_blocking for synthesis. The assistant planned to use tokio::task::spawn_blocking for the CPU-bound synthesis work, which is the correct pattern for running long-running CPU-intensive work in a tokio async runtime without starving the async executor.

Assumptions Made

The implementation in <msg id=619> rests on several assumptions, some explicit and some implicit:

• Synthesis is strictly serializable. The assistant assumes that only one synthesis can run at a time, which is true given the CPU and memory requirements of PoRep C2 synthesis. However, for smaller proof types (WinningPoSt, WindowPoSt, SnapDeals), multiple syntheses might theoretically run concurrently — the assistant implicitly assumes this isn't beneficial.
• The SrsManager is thread-safe and clone-friendly. The synthesized proof carries an Arc<SuprasealParameters> reference. The assistant assumes this reference can be shared across the channel without ownership issues or GPU-side contention.
• GPU workers are interchangeable. The shared channel model assumes any GPU worker can prove any synthesized proof. This holds because the SRS parameters are loaded into GPU memory per-worker, and the synthesized proof data is CPU-side (it gets transferred to GPU during the prove phase).
• The bounded channel provides sufficient backpressure. The assistant assumes that a channel capacity of 1 (default) prevents OOM while still providing meaningful overlap. This is validated later in the session with E2E GPU testing.
• CUDA_VISIBLE_DEVICES isolation is sufficient. The assistant assumes that setting CUDA_VISIBLE_DEVICES per GPU worker correctly isolates GPU access, which is the standard approach but requires careful handling in the async context.

Input Knowledge Required

To understand and implement message <msg id=619>, one needs substantial domain knowledge:

1. Groth16 proving pipeline internals: Understanding the two-phase nature of Groth16 (synthesis → proving), the data dependencies between phases, and the memory characteristics of each phase.
2. Tokio async runtime: Knowledge of tokio::sync::mpsc bounded channels, spawn_blocking for CPU-intensive work, and async task lifecycle management including graceful shutdown via watch channels.
3. Filecoin proof types: Understanding the differences between PoRep C2 (10 partitions, batch synthesis), WinningPoSt, WindowPoSt, and SnapDeals, and how each maps to circuit construction.
4. CUDA GPU programming model: Understanding that GPU proving requires exclusive access to a GPU device, that CUDA contexts are per-process, and that CUDA_VISIBLE_DEVICES is the standard isolation mechanism.
5. Systems design patterns: Familiarity with producer-consumer pipelines, bounded buffers for backpressure, and the trade-offs between shared vs. per-consumer queues.
6. The existing codebase: Knowledge of the SynthesizedProof type, the ProofRequest type, the scheduler interface, and the SRS manager API — all of which were read and analyzed in the preceding messages.

Output Knowledge Created

Message <msg id=619> produced several forms of knowledge:

1. The SynthesizedJob type definition: A concrete data structure that couples synthesized proof data with job metadata, enabling the channel-based pipeline.
2. The restructured start() method: The engine's main entry point was rewritten to spawn a synthesis task and GPU worker tasks, connected by the bounded channel.
3. A validated architectural pattern: The async overlap design was subsequently tested with 3 consecutive PoRep C2 proofs, achieving a 1.27x throughput improvement (from ~90s/proof to ~60s/proof in steady state).
4. Documentation of the design rationale: The assistant's reasoning, captured in the conversation, serves as architectural documentation for future maintainers.
5. A reusable pattern for other proof types: While PoRep C2 was the primary target, the same async overlap architecture extends to WinningPoSt, WindowPoSt, and SnapDeals.

The Thinking Process

The assistant's thinking process, visible across messages <msg id=617>–<msg id=619>, reveals a structured engineering approach:

1. Understand the current state (<msg id=615>–<msg id=616>): Read all relevant source files to build a complete mental model of the existing architecture.
2. Design the target architecture (<msg id=617>): Articulate the two-stage pipeline with explicit diagrams showing the before and after states.
3. Consider alternatives (<msg id=617>): Evaluate per-GPU synthesis tasks vs. shared synthesis task, and explicitly reject the per-GPU approach due to resource contention.
4. Implement incrementally (<msg id=618>–<msg id=619>): Apply edits in sequence, discovering missing pieces (like SynthesizedJob) during implementation rather than trying to design everything upfront.
5. Name the concept: By giving the channel payload a name (SynthesizedJob), the assistant makes the design explicit and discussable. The recognition that SynthesizedJob was needed — captured in <msg id=619> — is a classic example of emergent design: the need for a composite type became apparent only when the assistant tried to wire the channel into the existing start() method. The existing code had separate paths for job completion (reporting results back to the requester) and proof generation, and the channel needed to unify these.

Mistakes and Incorrect Assumptions

While the implementation proved successful (validated by subsequent GPU testing), several potential issues deserve scrutiny:

• The single synthesis task assumption may not hold for all proof types. For smaller proofs like WinningPoSt, synthesis is much faster and uses fewer resources. A single serial synthesis task might become a bottleneck when multiple small proofs are queued behind a large PoRep C2 proof. The architecture could benefit from priority-aware synthesis scheduling.
• The shared channel creates a fairness problem. If a GPU worker grabs a PoRep C2 job (which takes ~60s to prove), other proof types waiting in the channel are delayed. A priority queue or per-proof-type channels might provide better latency for urgent proofs like WindowPoSt.
• Backpressure granularity is per-job, not per-byte. The bounded channel limits the count of queued synthesized proofs, but different proof types have vastly different memory footprints. A PoRep C2 synthesized proof (~20 GiB) and a WinningPoSt synthesized proof (~100 MiB) consume the same channel slot. Memory-aware backpressure would be more precise.
• The SynthesizedJob type assumes GPU workers are homogeneous. If the system has heterogeneous GPUs (e.g., an RTX 4090 and an RTX 3060), the shared channel model might assign a large job to the slower GPU, creating a "tail latency" problem. Per-GPU channels with work-stealing would be more sophisticated. None of these issues invalidate the design — they represent trade-offs that were consciously or unconsciously accepted. The 1.27x throughput improvement validated the core idea, and these refinements could be addressed in future phases.

Conclusion

Message <msg id=619> is a masterclass in concise, purposeful engineering communication. In two sentences and one edit, the assistant identified a missing abstraction (SynthesizedJob), explained why it was necessary, and implemented it. The message sits at the intersection of design and implementation — it is neither pure architecture (that was done in <msg id=617>) nor pure coding (that was spread across multiple edits). It is the moment when the design meets reality and the missing piece becomes visible.

The SynthesizedJob type is more than a struct definition; it is the contractual bridge between two pipeline stages, the embodiment of the async overlap pattern, and the key that unlocked a 27% throughput improvement. In the broader narrative of the cuzk proving engine, this message represents the pivot from sequential to concurrent execution — a shift that transforms the engine from a batch processor into a true streaming pipeline.