The Architecture of Overlap: Designing a Two-Stage Proving Pipeline for Filecoin Proofs

In the world of high-performance cryptographic proving, the difference between a system that works and one that performs often comes down to a single question: what is the CPU doing while the GPU is busy? Message 617 of this opencode session captures the moment when an AI assistant confronts that question head-on, designing and beginning to implement a true async overlap architecture for the cuzk proving engine — a custom Groth16 proof generator for Filecoin's Proof-of-Replication (PoRep) and related proof types. This message is a masterclass in systems-level reasoning, architectural decision-making, and the kind of iterative refinement that separates a naive pipeline from a production-grade one.

The Context: Why This Message Exists

To understand message 617, we must first understand what came before it. The cuzk project had been evolving through multiple phases. Phase 0 established the scaffold — a basic proving daemon with observability and batch benchmarking. Phase 1 added support for all four Filecoin proof types (PoRep C2, WinningPoSt, WindowPoSt, SnapDeals) and introduced multi-GPU worker pools with priority scheduling. Phase 2 began with a critical architectural insight: the monolithic proving pipeline could be split into two phases — CPU-bound circuit synthesis and GPU-bound proof generation — and these phases could be overlapped.

The assistant had just committed the batch-mode pipeline rewrite (message 612), which consolidated all 10 PoRep partitions into a single rayon-parallel synthesis call followed by a single GPU pass. This matched monolithic performance (~91s vs ~93s) but was still fundamentally sequential: each proof was synthesized, then proved, then the next proof began. The GPU sat idle during synthesis; the CPU sat idle during GPU proving. Message 617 represents the pivot from "parallel within a single proof" to "overlap across consecutive proofs."

The user's prompt at message 606 — "Continue if you have next steps, or stop and ask for clarification if you are unsure how to proceed" — gave the assistant autonomy to drive the work forward. After verifying compilation, running tests, and committing the batch pipeline, the assistant was ready for the core architectural challenge.

The Architecture of Overlap

The message opens with a crystal-clear articulation of both the current and target architectures, rendered as ASCII diagrams:

**Current architecture** (sequential per-worker):
Worker: pull_request → SRS_load → synthesize → GPU_prove → complete → repeat

**Target architecture** (two-stage pipeline):
Synthesis task:  pull_request → SRS_load → synthesize → push to channel
                 pull_request → SRS_load → synthesize → push to channel (overlapping with GPU)
GPU worker:      pull from channel → GPU_prove → complete
                 pull from channel → GPU_prove → complete

This is more than a diagram — it is a thesis statement. The assistant is declaring that the fundamental unit of parallelism should shift from within a proof (parallel partitions) to across proofs (overlapping synthesis and proving). The key insight is that synthesis and GPU proving use entirely different hardware resources — CPU cores and RAM for synthesis, GPU compute and VRAM for proving — so they can safely run concurrently without resource contention.

Key Design Decisions

The message lists six explicit design decisions, each with clear rationale:

1. A synthesis task pulls from the scheduler, runs synthesis on spawn_blocking, and pushes synthesized jobs into a bounded async channel. This separates the concerns of job acquisition (scheduler), CPU work (synthesis), and inter-stage communication (channel). Using spawn_blocking is significant — it's a tokio mechanism that offloads CPU-heavy work to a dedicated thread pool, preventing the async runtime from being starved.

2. Each GPU worker pulls from the channel and runs gpu_prove. This keeps the GPU worker loop simple: it doesn't need to know about the scheduler or synthesis; it just consumes ready-to-prove jobs from the channel.

3. Channel capacity = config.pipeline.synthesis_lookahead (default: 1). This is a crucial design parameter. The bounded channel provides natural backpressure: if the GPU workers are busy and the channel is full, the synthesis task blocks, preventing runaway memory consumption. The default of 1 means at most one synthesized proof is queued ahead, which for a 200 GiB PoRep C2 proof is already substantial memory pressure.

4. The synthesis task respects the shutdown signal. Graceful shutdown is a production requirement, not an afterthought.

5. SRS is loaded before synthesis (still via SrsManager). The Structured Reference String (Groth16 parameters) must be loaded into GPU memory before either synthesis or proving can proceed. The assistant correctly keeps this as a prerequisite step in the synthesis task.

6. The pipeline is per-GPU — each GPU gets its own synthesis task + channel + GPU worker pair, so CUDA_VISIBLE_DEVICES is set correctly. This ensures GPU isolation: each worker sees only its assigned GPU.

The Thinking Process: What the Message Reveals

Message 617 is notable for what it doesn't say — or rather, what it says and then immediately refines. The assistant states "The pipeline is per-GPU — each GPU gets its own synthesis task + channel + GPU worker pair." But in the very next message (618), the assistant reconsiders:

"Actually, let me think more carefully about the architecture... Given that synthesis uses ~all CPU cores for PoRep, having multiple synthesis tasks would serialize anyway. Better to have a single synthesis task feeding a shared channel that all GPU workers compete on."

This is a critical refinement. The per-GPU pipeline design would create N synthesis tasks for N GPUs, but since PoRep C2 synthesis saturates all available CPU cores (the system has ~142 cores), multiple synthesis tasks would simply serialize on CPU resources. The single-synthesis-task design is more efficient: one synthesis task feeds a shared bounded channel, and N GPU workers compete for jobs. The CUDA_VISIBLE_DEVICES isolation is still maintained per GPU worker.

This refinement demonstrates a deep understanding of the actual resource profile of the system — not just the abstract architecture, but the concrete constraints of the hardware. It's the kind of insight that separates a design that looks good on paper from one that actually performs.

Assumptions Embedded in the Design

The message makes several assumptions worth examining:

Synthesis and GPU proving can truly overlap without resource contention. This assumes the CPU and GPU are independent resources, which is generally true for discrete GPUs but could be false for integrated GPUs or systems with shared memory bandwidth.
The bounded channel provides sufficient backpressure. With a default capacity of 1 and a single PoRep C2 proof consuming ~200 GiB of RAM, the system is already operating at the edge of memory. If synthesis completes faster than GPU proving (unlikely given the ~60s synthesis vs ~30s GPU times observed later), the channel could fill and block the synthesis task.
SRS loading can be done before synthesis in the synthesis task. This assumes the SRS is not needed during GPU proving, which is correct — the SRS is loaded once and cached via Arc<SuprasealParameters<Bls12>>.
The scheduler interface is compatible with a pull-based synthesis task. The scheduler was designed for push-based dispatch to workers; the synthesis task pulling from it requires the scheduler to expose a pop or dequeue method.

Input Knowledge Required

To fully understand message 617, one needs familiarity with several domains:

Groth16 proofs and Filecoin PoRep: Understanding that proof generation involves circuit synthesis (CPU-bound, memory-intensive) and GPU proving (GPU-bound, compute-intensive) is foundational.
Tokio async primitives: spawn_blocking, tokio::sync::mpsc, and the concept of bounded channels for backpressure are central to the design.
CUDA GPU isolation: CUDA_VISIBLE_DEVICES is the standard mechanism for assigning specific GPUs to specific processes in multi-GPU systems.
The existing cuzk codebase: The scheduler, SRS manager, pipeline module, and engine architecture were all read by the assistant in messages 615-616 before this design was produced.
The bellperson fork: The split between synthesize_circuits_batch and gpu_prove APIs was established in an earlier commit and is the foundation for the two-stage pipeline.

Output Knowledge Created

Message 617 produces several concrete artifacts:

A documented architectural design for the async overlap pipeline, including diagrams and decision rationale.
A set of six explicit design decisions that guide the implementation.
A config parameter (synthesis_lookahead) that controls the channel capacity and thus the memory-pressure/throughput tradeoff.
The beginning of implementation: the message ends with the assistant reading the pipeline.rs file to start coding.
A refinement that will appear in the next message: the shift from per-GPU synthesis tasks to a single shared synthesis task, which is the kind of architectural evolution that only emerges from deep engagement with the code.

The Broader Significance

Message 617 represents a pivotal moment in the cuzk project. The batch-mode pipeline committed in message 612 matched monolithic performance but didn't improve throughput. The async overlap architecture designed here is the mechanism that will deliver the 1.27x speedup validated later in the session (message 620+). It transforms the proving engine from a sequential pipeline (synthesize → prove → synthesize → prove) into a concurrent one (synthesize → [channel] → prove, with synthesis of N+1 overlapping prove of N).

The design is also notable for its restraint. The assistant doesn't propose exotic synchronization primitives, distributed queues, or complex load-balancing schemes. It uses a simple bounded channel — one of the most basic async communication patterns — and lets the natural backpressure of the system regulate itself. This is the hallmark of good systems design: finding the simplest mechanism that satisfies the constraints.

In the end, message 617 is about recognizing that the bottleneck in a proving pipeline isn't always the slowest stage — sometimes it's the idle time between stages. By overlapping synthesis and proving, the assistant effectively reclaims that idle time, turning a sequential pipeline into a concurrent one with minimal additional complexity. The bounded channel is the fulcrum on which this transformation rests, and the design decisions laid out in this message are the blueprint for making it work.