The Architecture of Overlap: Designing a Two-Stage Proving Pipeline for Filecoin PoRep

Introduction

In the evolution of a high-performance SNARK proving daemon for Filecoin, few decisions carry as much architectural weight as the moment when the assistant stops and thinks carefully about how to restructure the core execution model. Message 618 of this opencode session captures exactly such a moment: the assistant, having just committed a batch-mode pipeline rewrite for all four Filecoin proof types (PoRep C2, WinningPoSt, WindowPoSt, SnapDeals), pauses to design the next critical piece — true asynchronous overlap between CPU-bound circuit synthesis and GPU-bound proof generation.

This message is not merely an edit to a Rust source file. It is a window into the reasoning process behind a fundamental architectural shift: moving from a sequential per-worker model where each GPU worker independently synthesized and proved proofs one at a time, to a two-stage pipeline where a dedicated synthesis task continuously feeds synthesized jobs into a bounded channel that GPU workers consume from. The message reveals how the assistant navigates constraints around CPU core utilization, GPU isolation, memory pressure, and multi-GPU topology to arrive at a clean, composable design.

The Immediate Context

To understand message 618, we must first understand where the project stands. The cuzk proving daemon has been built in phases. Phase 0 established the scaffold with observability and batch benchmarking. Phase 1 implemented all four Filecoin proof types with multi-GPU worker pools and priority scheduling. Phase 2 began with a fork of the bellperson library to expose separate synthesis and GPU proving APIs, followed by a batch-mode pipeline rewrite that synthesized all ten PoRep partitions in a single rayon-parallel call.

At the moment of message 618, the assistant has just committed that batch-mode pipeline rewrite (message 612, commit 698c32b3). The git log shows three Phase 2 commits: the bellperson fork, the pipelined synthesis/GPU prover for PoRep C2, and now the batch pipeline for all proof types. All 15 unit tests pass. The code compiles cleanly. But the architecture is still fundamentally sequential: each GPU worker pulls a request, loads SRS parameters, synthesizes the circuit, runs the GPU proof, and completes the job — all in serial.

The next step, clearly articulated in the assistant's own todo list, is to "implement true async overlap in engine.rs (synthesis task + bounded channel + GPU workers)." This is the message we are analyzing.

The Reasoning Process: A Detailed Walkthrough

The message begins with a confident declaration: "Now I have a thorough understanding of all the code." This is not empty posturing — it follows a sequence of reads that span engine.rs, pipeline.rs, config.rs, types.rs, scheduler.rs, srs_manager.rs, and lib.rs (messages 615-617). The assistant has built a complete mental model of the codebase before attempting the redesign.

The first articulation of the change is precise: "replace the single per-GPU worker loop with a two-stage pipeline: a shared synthesis task that feeds synthesized proofs into per-GPU bounded channels, and GPU workers that consume from those channels." Already we see a design choice — the channels are "per-GPU," suggesting one synthesis task per GPU.

But then the assistant catches itself and thinks deeper. The paragraph beginning "Actually, let me think more carefully about the architecture" is the heart of the message. Here, the assistant enumerates the fundamental constraints:

1. Synthesis is CPU-bound — it uses approximately 142 cores and roughly 200 GiB of RAM. This means only one synthesis operation can run at a time for PoRep proofs, because the CPU and memory resources are exhausted by a single instance.
2. GPU proving is GPU-bound — it uses one GPU per worker. The GPU is the bottleneck, not the CPU.
3. Multiple GPUs exist — each GPU worker needs its own CUDA_VISIBLE_DEVICES environment variable for isolation. These constraints lead to a critical design question: for the multi-GPU case, should there be one synthesis task per GPU (each feeding its own channel), or a single shared synthesis task feeding all GPU workers? The assistant reasons through this carefully. If synthesis uses all CPU cores for PoRep, having multiple synthesis tasks would not provide parallelism — they would serialize on CPU resources anyway. The memory constraint (~200 GiB per synthesis) also argues against concurrent synthesis tasks, as that would multiply memory pressure. Therefore, a single synthesis task feeding a shared channel that all GPU workers compete on is the better design. But there's a subtlety: CUDA_VISIBLE_DEVICES must be set per GPU worker. The assistant notes that the SRS (SuprasealParameters) is shared via Arc, so any GPU worker can use any SRS regardless of which physical GPU it runs on. The CUDA_VISIBLE_DEVICES environment variable only controls which GPU is addressed by CUDA API calls. This means a shared channel design works: the synthesis task produces jobs, any GPU worker can pick them up, and the worker sets its own CUDA_VISIBLE_DEVICES before running the GPU proof.

The Design Decision

The final architecture is:

• One synthesis task: pulls proof requests from the scheduler, runs synthesis on a blocking thread (via spawn_blocking), and pushes (SynthesizedProof, job_metadata) into a bounded asynchronous channel.
• N GPU workers: each pinned to a GPU via CUDA_VISIBLE_DEVICES, pull from the shared channel and run gpu_prove.
• Channel capacity: controlled by config.pipeline.synthesis_lookahead (default: 1), providing backpressure to prevent OOM. The bounded channel is the key mechanism for backpressure. If the GPU workers fall behind, the channel fills up, and the synthesis task blocks on push, preventing it from consuming more memory with additional synthesized proofs. This is a textbook producer-consumer pattern adapted to the specific constraints of SNARK proving. The assistant also considers the shutdown signal — the synthesis task must respect graceful shutdown, which is handled via a watch channel already present in the engine.

Assumptions Embedded in the Design

Several assumptions underpin this architecture, and the message makes them explicit:

1. Synthesis is the exclusive CPU consumer. The assistant assumes that a single PoRep synthesis saturates all available CPU cores (~142 cores). This is derived from earlier analysis in the session (segment 0's deep-dive into SUPRASEAL_C2) which characterized the computational hotpaths.
2. Memory is the binding constraint. The ~200 GiB peak memory for PoRep C2 synthesis means that having more than one synthesis in flight risks OOM. The bounded channel enforces this limit.
3. SRS is shareable across GPUs. The assistant notes that SuprasealParameters is wrapped in Arc, so any GPU worker can use any SRS instance. This is correct because the parameters are read-only after loading.
4. GPU workers are interchangeable. Any GPU worker can prove any synthesized proof, regardless of which proof kind or which partition set was synthesized. This is true because the synthesized proof contains all the circuit state needed for the GPU phase.
5. The scheduler is the single source of truth. The synthesis task pulls from the scheduler, which handles priority ordering and FIFO semantics. The GPU workers don't interact with the scheduler directly — they only consume from the channel.

Input Knowledge Required

To understand this message, a reader needs familiarity with several domains:

• Filecoin PoRep (Proof-of-Replication): the proving system that storage miners must periodically execute to prove they are storing data. PoRep C2 is the most computationally intensive proof kind, involving ten partitions.
• Groth16: the SNARK proving system used by Filecoin, consisting of a CPU-bound circuit synthesis phase and a GPU-bound proving phase involving multi-scalar multiplication (MSM) and number-theoretic transform (NTT).
• CUDA_VISIBLE_DEVICES: an environment variable that restricts CUDA API visibility to specific GPU indices, used for multi-GPU isolation.
• Rayon: Rust's data parallelism library, used for parallel circuit synthesis across partitions.
• Tokio channels: specifically tokio::sync::mpsc for bounded asynchronous communication between tasks.
• The cuzk project's prior architecture: the sequential per-worker model that existed before this change, where each GPU worker handled the full synthesis-to-proof lifecycle.

Output Knowledge Created

This message produces a concrete architectural blueprint that is immediately actionable. The assistant applies an edit to engine.rs that restructures the worker loop. The output knowledge includes:

• The decision to use a single shared synthesis task rather than per-GPU synthesis tasks.
• The decision to use a bounded tokio::sync::mpsc channel with capacity controlled by synthesis_lookahead.
• The decision to have GPU workers pull from the channel rather than the scheduler directly.
• The understanding that backpressure via bounded channels prevents OOM.
• The recognition that SRS sharing via Arc enables any GPU to prove any synthesized job. This knowledge is encoded not just in the edit itself, but in the reasoning that precedes it. The message serves as design documentation for why the architecture takes this specific form.

Potential Issues and Unaddressed Questions

While the reasoning is sound, several questions remain unaddressed in this message:

1. What about non-PoRep proof kinds? WinningPoSt, WindowPoSt, and SnapDeals have different resource profiles. WinningPoSt, for example, involves many small proofs (one per sector in a partition). The assistant's architecture assumes a single synthesis task, but for WinningPoSt, multiple small syntheses could potentially run concurrently without exhausting CPU or memory. The design doesn't address whether the single synthesis task is a bottleneck for these lighter proof kinds.
2. How does the synthesis task handle mixed proof kinds? If a PoRep C2 (heavy, 10 partitions) is followed by a WinningPoSt (light, many small proofs), does the single synthesis task serialize them? The bounded channel provides backpressure, but the scheduling policy within the synthesis task is not discussed.
3. What about GPU worker affinity? The assistant mentions CUDA_VISIBLE_DEVICES for GPU isolation, but doesn't discuss whether certain synthesized proofs should prefer certain GPUs (e.g., because SRS is already loaded in GPU memory). The Arc<SRS> sharing handles CPU-side SRS, but GPU-side SRS loading (into GPU memory) is a separate concern.
4. Error handling in the synthesis task. If synthesis fails for a job, how does the error propagate to the GPU workers and back to the requester? The message doesn't specify the error handling protocol for the channel. These are not necessarily mistakes — they may be addressed in subsequent messages or left for future phases. But they represent gaps in the reasoning presented in this single message.

Conclusion

Message 618 is a masterclass in architectural reasoning under real-world constraints. The assistant takes a seemingly straightforward task — "implement async overlap" — and unpacks it into a careful analysis of CPU utilization, memory pressure, GPU isolation, and backpressure mechanics. The result is a clean producer-consumer architecture that respects the physical limits of the hardware while enabling throughput improvement.

The message also demonstrates the value of thinking out loud in design documents. By articulating the constraints, enumerating the options, and reasoning through the trade-offs, the assistant produces not just code but understanding. A future reader of this message — or a teammate reviewing the pull request — can follow the logic and evaluate whether the assumptions hold for their deployment scenario.

In the broader context of the cuzk project, this message marks the transition from a prototype that works to a system that performs. The sequential baseline took ~90 seconds per PoRep C2 proof. With async overlap, the assistant will later measure 212.7 seconds for three consecutive proofs — a 1.27x throughput improvement, dropping from ~90s/proof to ~60s/proof in steady state. That improvement is a direct consequence of the architectural decisions made in this message.