The Critical Wiring: How a Single Read Operation Anchored Phase 2 of the cuzk Pipelined Proving Engine

Introduction

In the middle of a sprawling, multi-session implementation effort to build a pipelined SNARK proving daemon for Filecoin, there comes a moment that separates architectural design from operational reality. That moment arrives in message <msg id=509> of the conversation, where the assistant declares: "Now the critical part: update the GPU worker loop to use the pipelined prover for PoRep C2 when pipeline mode is enabled." This single sentence, followed by a read operation on the engine source file, represents the inflection point where weeks of analysis, design documentation, fork creation, and module scaffolding converge into a concrete, executable change to the system's central nervous system.

The message itself is deceptively simple — a brief statement of intent and a file read. But to understand its significance, one must appreciate the architecture being built, the long chain of dependencies that had to be resolved before this moment, and the careful reasoning about concurrency, memory pressure, and GPU utilization that informed the design.

Context: The cuzk Pipeline Architecture

The cuzk project is a pipelined SNARK proving daemon for Filecoin's proof-of-replication (PoRep) and proof-of-spacetime (PoSt) protocols. Filecoin storage miners must continuously generate Groth16 proofs to demonstrate they are storing data correctly. These proofs are computationally expensive — a single 32 GiB PoRep C2 proof requires approximately 200 GiB of peak memory, involves synthesizing circuits with ~106 million constraints per partition across 10 partitions, and performs massive number-theoretic transforms (NTTs) and multi-scalar multiplications (MSMs) on the GPU.

Phase 0 and Phase 1 of cuzk established a working monolithic prover: a gRPC daemon that accepted proof jobs, dispatched them to GPU workers, and returned results. The prover used the existing filecoin-proofs-api library, which called into the monolithic seal_commit_phase2() function. This function performed all 10 partitions' circuit synthesis on the CPU (consuming ~136 GiB of intermediate memory), then shipped the entire batch to the GPU for the cryptographic proving phase. The approach worked, but it had severe limitations: the ~200 GiB memory footprint required expensive machines, SRS (Structured Reference String) parameters were loaded lazily and cached in a private process-global cache with no eviction policy, and the monolithic design prevented overlapping computation — the GPU sat idle while the CPU synthesized all 10 partitions.

Phase 2, the subject of this message, aims to replace this monolithic architecture with a per-partition pipelined model. Instead of synthesizing all 10 partitions before touching the GPU, the pipeline synthesizes one partition at a time (~13.6 GiB intermediate state), ships it to the GPU, and while the GPU proves partition N, the CPU synthesizes partition N+1. This reduces peak memory from ~136 GiB to ~27 GiB (two partitions in flight), enabling the system to run on 128 GiB machines. It also introduces an explicit SRS manager that loads parameters directly via SuprasealParameters::new(path), bypassing the private GROTH_PARAM_MEMORY_CACHE and providing control over parameter residency, preloading, and eviction.

The Message: A Pivot Point

The subject message occurs after the assistant has already completed several critical prerequisites:

Created the bellperson fork (<msg id=430> onward) that exposes the previously private synthesize_circuits_batch() function and adds a new prove_from_assignments() function, enabling the synthesis/GPU split.
Implemented the SRS manager (srs_manager.rs) that maps CircuitId values to exact .params filenames on disk and supports preload/evict operations with memory budget tracking.
Implemented the pipeline module (pipeline.rs) containing the SynthesizedProof type and the split synthesize_porep_c2_partition() / gpu_prove() functions.
Added pipeline configuration to the config system (config.rs), with a pipeline.enabled toggle.
Updated the Engine struct to hold an Arc<Mutex<SrsManager>> and wired it through the constructor and start() method, including SRS preloading logic. What remains — and what this message addresses — is the most consequential change: modifying the GPU worker loop itself. The worker loop is the heart of the proving daemon. It's the async task that polls the scheduler for new jobs, dispatches them to the GPU, and reports results. In Phase 1, this loop called prover::prove_porep_c2() monolithically. In Phase 2, it must conditionally call pipeline::prove_porep_c2_pipelined() when pipeline mode is enabled and the proof kind is PoRepSealCommit.

The Reasoning: Why This Is "The Critical Part"

The assistant's framing — "Now the critical part" — is not rhetorical. Several factors make this the highest-risk change in the entire Phase 2 implementation:

Concurrency model complexity. The worker loop runs in an async Tokio task. The pipelined prover, however, involves synchronous, CPU-intensive work (circuit synthesis using rayon parallelism across ~142 cores) followed by GPU work (NTT/MSM via CUDA). Bridging these two worlds requires careful handling: the synthesis must run on a blocking thread (not the async runtime), the GPU must be isolated via CUDA_VISIBLE_DEVICES, and the SRS manager (wrapped in Arc<Mutex<>>) must be accessible without deadlock. The assistant's reasoning in <msg id=504> shows awareness of these issues: "This keeps the same worker model (one worker per GPU) but changes what the worker does internally."

Per-partition vs. cross-proof overlap. The assistant explicitly distinguishes between two forms of pipelining. Per-partition pipelining (within a single proof) synthesizes partition N+1 while partition N is on the GPU. Cross-proof overlap (synthesizing the next job while the GPU finishes the current one) would require a separate synthesis thread pool feeding a bounded channel. The assistant correctly identifies that per-partition pipelining already provides significant benefit and defers cross-proof overlap to a future enhancement. This is a pragmatic engineering decision — implementing the simpler form first validates the architecture and delivers immediate memory savings, while the more complex form can be layered on later without redesigning the core.

Backward compatibility. The pipeline must be optional. When pipeline.enabled = false, the engine must fall back to the Phase 1 monolithic path exactly as before. This means the worker loop needs a conditional branch that is transparent to the rest of the system — job submission, scheduling, result reporting, and observability must all work identically regardless of which proving path is taken. The assistant's design achieves this by keeping the same WorkerState model and only changing what the worker's blocking task does internally.

SRS manager integration. The SRS manager is shared across all workers via Arc<Mutex<SrsManager>>. In the preload phase (message <msg id=508>), the engine iterates over all configured circuit IDs and calls srs_manager.ensure_loaded(). But the worker loop must also handle the case where the required SRS is not yet loaded — perhaps because a new circuit type appears dynamically. The assistant's design relies on the SRS manager's ensure_loaded() being idempotent and thread-safe, which is a reasonable assumption given the Mutex wrapping.

Assumptions Embedded in This Message

The assistant makes several assumptions, most of which are well-founded but worth examining:

The pipeline_enabled flag and srs_manager can be passed into the worker spawn closure. This assumes the worker spawn infrastructure (currently using tokio::spawn with a closure that captures self fields) can be refactored to accept these new parameters. The assistant's subsequent edits in <msg id=510> and <msg id=511> confirm this is feasible, but it requires changing the worker's function signature and the spawn site.

The pipelined prover's output format matches the monolithic prover's. The engine's result reporting code expects a ProofResult with specific fields (proof bytes, timing breakdown, etc.). The assistant assumes that pipeline::prove_porep_c2_pipelined() returns a compatible structure. This is a safe assumption because the pipeline module was designed to produce the same SynthesizedProof type that wraps the same Groth16 proof bytes, but it has not yet been validated end-to-end with real GPU execution.

The SRS manager's ensure_loaded() is safe to call from within the worker's blocking task. Since the SRS manager is behind Arc<Mutex<>>, calling it from the worker thread will acquire the lock. If another worker (or the preload loop) holds the lock for an extended period (e.g., while loading a 45 GiB parameter file from disk), the worker will block. The assistant does not address this potential contention issue in this message, though the design implicitly assumes that preloading happens before workers start, and that runtime ensure_loaded() calls are rare.

The pipeline.enabled configuration is static (set at startup, not changed dynamically). The engine reads the config once during construction. If an operator wants to toggle pipeline mode, they must restart the daemon. This is a reasonable simplification for Phase 2, but it means the worker loop does not need to handle hot-switching between monolithic and pipelined modes.

Input Knowledge Required

To understand this message fully, one must be familiar with:

The cuzk project architecture — the workspace structure, the engine/scheduler/worker model, the gRPC API, and the distinction between Phase 0/1 (monolithic) and Phase 2 (pipelined).
The bellperson fork — specifically that synthesize_circuits_batch() and prove_from_assignments() are now public, enabling the synthesis/GPU split.
The SRS manager design — that SrsManager wraps SuprasealParameters, maps CircuitId to disk paths, and supports ensure_loaded() / preload() / evict() operations.
The PoRep circuit structure — that a 32 GiB PoRep has 10 partitions, each with ~106M constraints and ~13.6 GiB of intermediate state, and that per-partition pipelining reduces peak memory by an order of magnitude.
The engine's worker loop — the async structure, the WorkerState tracking, the CUDA_VISIBLE_DEVICES isolation, and the blocking task pattern used to run synchronous proving code without blocking the async runtime.
The Filecoin proof types — that PoRep C2 is the most memory-intensive proof, while PoSt and SnapDeals are smaller and remain on the monolithic path in Phase 2.

Output Knowledge Created

This message, combined with the subsequent edits it enables, creates several forms of knowledge:

A validated architecture for pipelined proving. The worker loop change is the final integration point that proves the pipeline architecture works end-to-end (pending GPU validation). It demonstrates that the bellperson fork's exposed APIs, the SRS manager's direct parameter loading, and the per-partition synthesis/prove split can be composed into a working system.

A template for future pipeline extensions. The conditional branch in the worker loop (if pipeline_enabled && proof_kind == PoRepSealCommit { use pipelined prover } else { use monolithic prover }) establishes a pattern that can be extended to other proof types (PoSt, SnapDeals) in later phases, and to cross-proof overlap (separate synthesis thread pool) in Phase 3.

A concrete memory reduction pathway. The per-partition pipelining approach, now wired into the engine, provides a measurable path from ~200 GiB peak memory to ~27 GiB, enabling the proving daemon to run on 128 GiB machines that are dramatically cheaper in cloud rental markets.

A benchmarkable baseline. With the pipeline wired in, the assistant can now run end-to-end tests against the golden test data in /data/32gbench/, comparing pipeline mode against monolithic mode on proof correctness, wall-clock time, memory usage, and GPU utilization.

The Thinking Process Visible in the Message

The assistant's reasoning, visible across the surrounding messages, reveals a disciplined engineering approach:

Top-down decomposition. The assistant does not start by editing the worker loop. It first creates the SRS manager, then the pipeline module, then the config system, then the engine constructor and preload logic. Only after all dependencies are in place does it tackle the worker loop. This is classic dependency-driven development — each layer builds on the previous one, and the worker loop change is the capstone.

Explicit consideration of alternatives. In <msg id=504>, the assistant explicitly considers two forms of pipelining (per-partition vs. cross-proof overlap) and makes a conscious choice to implement the simpler one first. The reasoning is pragmatic: "the per-partition pipelining approach already provides significant benefit." This is not an oversight but a deliberate scope management decision.

Awareness of concurrency hazards. The assistant's design keeps the same worker model (one blocking task per GPU) and only changes what the blocking task does internally. This avoids introducing new concurrency primitives (channels, thread pools) that would increase the risk of deadlocks, race conditions, or async-blocking violations. The Arc<Mutex<SrsManager>> pattern is a standard Rust idiom for shared mutable state, and its use here is appropriate.

Documentation-driven development. The assistant frequently reads the existing code before editing it (as in this message, where it reads engine.rs lines 215-225). This habit ensures that edits are precise and that the assistant understands the current state of the code before making changes. It also serves as a form of self-documentation — the read output in the conversation provides a record of what the code looked like at each step.

Potential Issues and Risks

While the assistant's approach is sound, several risks deserve attention:

The GPU worker's blocking task may hold the SRS manager lock for extended periods. If ensure_loaded() is called within the worker's blocking task (e.g., for a circuit type not preloaded), the 45 GiB parameter file must be read from disk while holding the lock. This would block all other workers and the preload loop. A better design might use a read-write lock or a lock-free structure, but the assistant's approach works for the common case where all needed SRS is preloaded at startup.

The per-partition pipeline does not yet overlap across proofs. The current design pipelines partitions within a single proof but serializes proofs — the GPU finishes proof N entirely before proof N+1 starts synthesis. True cross-proof overlap would require a synthesis thread pool feeding a bounded channel to the GPU worker, which is explicitly deferred. This means the throughput benefit is limited to memory reduction and within-proof GPU utilization; the GPU still idle-waits between proofs.

The pipeline has only been tested without GPU (--no-default-features). The 15 unit tests pass, but they test the pipeline module's type structure and stub functions, not actual GPU proving. The real validation — end-to-end proof correctness and performance benchmarking — requires a GPU build (--features cuda-supraseal) and the golden test data. Until that validation is complete, the pipeline remains unproven in production conditions.

Conclusion

Message <msg id=509> is a small but pivotal moment in a large engineering effort. A single sentence — "Now the critical part: update the GPU worker loop" — followed by a file read, marks the transition from building components to integrating them into a working system. The message encapsulates the essence of systems engineering: the most important work is not the code you write but the architecture you design, the dependencies you resolve, and the moment you choose to connect them.

The assistant's disciplined approach — reading before editing, considering alternatives explicitly, deferring complexity to future phases, and maintaining backward compatibility — transforms what could be a risky, destabilizing change into a controlled, well-understood integration. The worker loop edit that follows this message is the final stitch that binds the SRS manager, the pipeline module, the configuration system, and the engine infrastructure into a coherent whole. It is the moment the pipelined proving engine becomes real.