The Integration Point: Wiring Phase 2 Pipelining into the cuzk Engine

Message Overview

The subject message ([msg 505]) is deceptively brief — just two sentences and an edit command:

[assistant] Now let me update the engine. The key change is in the worker loop — when pipeline mode is enabled and the proof kind is PoRepSealCommit, use the pipelined prover. The SRS manager is shared across workers via Arc<Mutex<SrsManager>>: [edit] /home/theuser/curio/extern/cuzk/cuzk-core/src/engine.rs Edit applied successfully.

This short message belies its significance. It represents the critical integration moment where an entire Phase 2 architecture — spanning three new modules, a configuration system, and a forked bellperson API — gets wired into the cuzk proving engine's central worker loop. Without this edit, all the preceding work creating pipeline.rs, srs_manager.rs, and the pipeline configuration would remain disconnected infrastructure, never actually invoked during proof generation.

The Context: From Monolithic to Pipelined Proving

To understand why this message was written, one must trace the arc of the cuzk project. The cuzk proving daemon began as a multi-GPU worker pool (Phase 0/1) that wrapped the existing monolithic filecoin-proofs-api calls. When a PoRep C2 proof job arrived, the engine dispatched it to a GPU worker, which called seal_commit_phase2() — a single function that performed circuit synthesis, loaded the 32 GiB SRS parameters, ran the GPU proof, and returned serialized proof bytes. This monolithic approach had severe drawbacks: it required ~200 GiB of peak memory (because all partitions were synthesized before any GPU work began), it loaded and discarded the SRS parameters for every proof (wasting I/O and memory bandwidth), and it kept the GPU idle while CPU synthesis completed.

The Phase 2 redesign, executed across the messages in this chunk ([msg 430] through [msg 530]), aimed to replace this monolithic prover with a per-partition pipelined architecture. The core insight, developed through extensive analysis in earlier segments ([msg 0] through [msg 429]), was that the Groth16 proving process could be split at a natural boundary: circuit synthesis (CPU-bound, memory-intensive) and GPU proving (GPU-bound, parameter-dependent). By synthesizing one partition at a time and immediately handing it to the GPU, peak intermediate memory drops from ~136 GiB to ~13.6 GiB — making the pipeline viable on 128 GiB machines that previously could not run the monolithic prover.

The Architecture Being Wired In

By the time the assistant reaches message 505, it has already constructed the major pieces:

1. srs_manager.rs: A module that provides explicit control over SRS (Structured Reference String) parameter loading. It maps CircuitId values to exact .params filenames on disk, supports preload and evict operations with memory budget tracking, and bypasses the private GROTH_PARAM_MEMORY_CACHE that the monolithic prover used internally. The SRS manager is the key to eliminating redundant parameter loading — parameters can be loaded once and kept resident across multiple proofs.
2. pipeline.rs: The core pipeline module containing the SynthesizedProof type (the intermediate artifact produced by synthesis and consumed by GPU proving) and the two split functions: synthesize_porep_c2_partition() and gpu_prove(). These functions leverage the bellperson fork's exposed synthesize_circuits_batch() and prove_from_assignments() APIs — APIs that were previously internal to the monolithic prover.
3. Pipeline configuration: A pipeline section added to config.rs with an enabled boolean flag, allowing operators to toggle between the Phase 1 monolithic path and the Phase 2 pipelined path without recompilation.
4. Dependency wiring: New dependencies added to cuzk-core/Cargo.toml including filecoin-proofs, storage-proofs-core, bellperson, blstrs, rayon, and ff — all necessary for the pipeline module to compile and link against the bellperson fork's exposed APIs. The assistant had already verified that all 15 unit tests pass (12 pre-existing plus 3 new pipeline tests) and that the cuzk code compiles with zero warnings ([msg 497], [msg 498]). The infrastructure was sound; it just needed to be connected.

The Engine Edit: Design Decisions in the Worker Loop

The engine (engine.rs) is the central coordinator of the cuzk proving daemon. It owns the scheduler, the GPU workers, and — critically after this edit — the SRS manager. The worker loop is where GPU-bound threads poll for jobs, execute proof generation, and return results.

The edit described in message 505 makes two fundamental changes to this worker loop:

1. Conditional Routing: Pipeline vs. Monolithic

The worker loop now checks two conditions before dispatching a proof job:

• Is pipeline.enabled set to true in the configuration?
• Is the proof kind PoRepSealCommit (i.e., PoRep C2)? If both conditions are true, the worker calls pipeline::prove_porep_c2_pipelined() instead of the Phase 1 prover::prove_porep_c2(). This conditional routing is critical because the pipeline is specifically optimized for PoRep C2 — the proof type with the largest memory footprint and the most to gain from per-partition streaming. PoSt (Proof of Spacetime) and SnapDeals proof types continue to use the monolithic path, as their memory requirements are smaller and the pipeline complexity would not yield proportional benefits.

2. Shared SRS Manager via Arc<Mutex<SrsManager>>

The SRS manager is shared across all GPU workers using Arc<Mutex<SrsManager>>. This design choice reflects several assumptions and trade-offs:

• Thread safety: Mutex ensures that only one worker loads or evicts parameters at a time, preventing race conditions on the underlying parameter memory.
• Shared residency: Because the SRS manager is shared, parameters loaded by one worker are available to all workers. If Worker A loads the 32 GiB PoRep parameters for a proof, Worker B can reuse them without reloading — a significant savings given that parameter loading from disk can take tens of seconds.
• Simplicity over performance: An Arc<Mutex<...>> is a coarse-grained approach. A more sophisticated design might use a read-write lock or a lock-free structure, but the Mutex is sufficient for the current workload where parameter loading is infrequent relative to proof generation. The assistant's reasoning in the preceding message ([msg 504]) reveals awareness of a further optimization — true cross-proof overlap where synthesis of the next job begins while the GPU finishes the current one — but explicitly defers this to future work. The per-partition pipelining already provides the primary memory benefit.

Assumptions Embedded in This Message

Several assumptions underpin the engine edit:

Assumption 1: PoRep C2 is the only proof type that needs pipelining. The conditional check proof kind is PoRepSealCommit hard-codes this assumption. While true for the current use case (Filecoin storage mining, where PoRep C2 dominates memory), it means that if future proof types grow in complexity, the pipeline will need extension.

Assumption 2: The SRS manager's Mutex will not become a contention bottleneck. This assumes that parameter loading is rare relative to proof generation. If the system processes proofs at high throughput with frequent parameter changes (e.g., alternating between different circuit types), the Mutex could become a contention point.

Assumption 3: The bellperson fork's exposed APIs are correct and stable. The pipeline module depends on synthesize_circuits_batch() and prove_from_assignments() being available from the forked bellperson crate. If the fork diverges from upstream or introduces bugs, the pipeline breaks.

Assumption 4: Per-partition pipelining is sufficient without cross-proof overlap. The current implementation pipelines within a single multi-partition proof but does not overlap synthesis of one proof with GPU work of another. The assistant acknowledges this as a future enhancement, implying an assumption that the current design provides enough benefit to justify the implementation complexity.

What This Message Creates: Output Knowledge

Message 505 produces a concrete artifact: an edited engine.rs that integrates the Phase 2 pipeline. But it also creates several forms of output knowledge:

1. A verified integration point: The edit completes the chain from configuration → engine → pipeline module → SRS manager → bellperson fork → GPU. All prior work is now connected and, as later messages confirm, functional.
2. A pattern for future pipeline extensions: The conditional routing pattern (if pipeline.enabled && PoRepSealCommit) establishes a template for adding pipeline support for other proof types. A developer adding WindowPoSt pipelining would follow the same pattern: add a new pipeline function, add a condition in the worker loop, and extend the SRS manager if needed.
3. A validated architectural decision: The Arc<Mutex<SrsManager>> pattern is now committed. Future developers can see this as the canonical way to share SRS state across workers, for better or worse.
4. A measurable baseline for benchmarking: With the pipeline wired in, the system can now be tested end-to-end with GPU hardware. The next step — integration testing against golden test data — will produce the first performance comparisons between Phase 1 monolithic and Phase 2 pipelined proving.

The Thinking Process: What Message 504 Reveals

The assistant's reasoning in message 504 — the immediate predecessor to the subject message — provides invaluable insight into the thinking behind the engine edit:

"The engine refactoring is the most complex part. The key design: when pipeline mode is enabled for PoRep C2, instead of calling prover::prove_porep_c2() monolithically on a blocking thread, the worker calls pipeline::prove_porep_c2_pipelined() which uses the SRS manager's params directly. This keeps the same worker model (one worker per GPU) but changes what the worker does internally."

This reveals a deliberate design constraint: minimize architectural disruption. The Phase 1 worker model — one thread per GPU, polling a scheduler — is preserved. The change is purely internal to the worker's execution path. This is a wise engineering choice: it limits the scope of the refactor, reduces the risk of regressions, and makes it easy to toggle between old and new paths via configuration.

The assistant continues:

"For true overlapping (synthesis of next proof while GPU processes current one), we'd need a separate synthesis thread pool feeding a bounded channel to the GPU worker. However, the per-partition pipelining approach already provides significant benefit: within a single proof, partition N+1 can be synthesized while partition N is on the GPU."

This is a critical clarification. The Phase 2 pipeline as implemented in message 505 does not provide cross-proof overlap. It provides within-proof per-partition streaming. The distinction matters: cross-proof overlap would require a fundamentally different architecture (synthesis thread pool + bounded channel), while per-partition pipelining fits within the existing worker model. The assistant explicitly prioritizes the simpler approach that still delivers the primary benefit (memory reduction).

Input Knowledge Required

To fully understand message 505, one needs:

1. The cuzk architecture: The engine owns workers, the scheduler dispatches jobs, and workers execute proof generation on specific GPUs via CUDA_VISIBLE_DEVICES isolation.
2. The Phase 1 monolithic path: prover::prove_porep_c2() calls seal_commit_phase2() which does synthesis + GPU proving in one shot, loading parameters internally.
3. The Phase 2 pipeline modules: srs_manager.rs provides SrsManager with ensure_loaded() and preload() methods; pipeline.rs provides SynthesizedProof, synthesize_porep_c2_partition(), and gpu_prove().
4. The bellperson fork: A minimal fork of bellperson that exposes synthesize_circuits_batch() and prove_from_assignments() as public APIs, enabling the split.
5. The configuration system: Config now has a pipeline subsection with an enabled boolean.
6. The PoRep proof structure: PoRep C2 proofs have multiple partitions (10 for 32 GiB sectors), each of which can be synthesized and proved independently.

Mistakes and Incorrect Assumptions

While the message itself is sound, several potential issues deserve examination:

The Arc<Mutex<SrsManager>> pattern could mask performance problems. If the Mutex is held during parameter loading (which can take tens of seconds for 32 GiB files), all other workers are blocked from loading or evicting parameters. A more granular design — perhaps separating the loading operation from the parameter storage — could reduce contention. However, since parameter loading is typically a one-time operation at daemon startup (parameters are preloaded and kept resident), this may not be a practical problem.

The conditional routing is proof-kind-specific, not generic. The hard-coded PoRepSealCommit check means that adding pipeline support for a new proof type requires modifying the engine. A more extensible design might use a trait or plugin system, but that would add complexity that the current use case does not justify.

The assumption that per-partition pipelining reduces memory to ~13.6 GiB depends on partition count. For 32 GiB sectors with 10 partitions, each partition's synthesis requires ~13.6 GiB. But for different sector sizes or proof configurations, the memory savings would scale differently. The pipeline is currently hard-coded for SectorShape32GiB, as the assistant explicitly noted in earlier messages ([msg 489]).

Conclusion

Message 505 is the keystone of the Phase 2 pipeline implementation. In two sentences and an edit command, it connects the new pipeline architecture to the engine's worker loop, completing the chain from configuration to GPU proof. The brevity of the message reflects the assistant's confidence: the hard work was in designing and implementing the pipeline modules, the SRS manager, and the configuration system. The engine edit is the final wiring — the moment when infrastructure becomes functionality.

The message also exemplifies a disciplined engineering approach: minimize architectural disruption, preserve the existing worker model, use conditional routing for backward compatibility, and defer non-critical optimizations (cross-proof overlap) to future work. The result is a pipeline that reduces peak memory by an order of magnitude while keeping the system toggleable between old and new paths — exactly the kind of incremental improvement that makes complex systems evolvable.