The Pipelined Partition Prover: How Parallel Synthesis and Per-Partition GPU Proving Reshaped the cuzk SNARK Engine

Introduction

In the high-stakes world of Filecoin proof generation, memory is the silent dictator. For a single 32 GiB PoRep (Proof-of-Replication) Groth16 zk-SNARK proof, the cuzk proving engine previously required approximately 228 GiB of RAM — a footprint that pushed cloud instance costs skyward and made multi-GPU deployments economically marginal. The Phase 6 slotted partition pipeline was designed to break this memory wall, but the journey from design to validated implementation revealed something far more interesting: the initial design's assumptions about GPU cost structure were wrong, and the path to success required a fundamental rethinking of how synthesis and GPU proving interact.

This article tells the full story of Segment 19 — the implementation and validation of the Phase 6 pipelined partition prover. It covers the initial slot-grouping approach, the surprising discovery of a GPU fixed-cost bottleneck, the redesign toward true parallel synthesis with per-partition GPU proving, the benchmark results that validated a 3.2× memory reduction (71 GiB vs 228 GiB) with only 16% latency overhead (72.0s vs 62.3s), and the subsequent daemon-level end-to-end testing that revealed a critical architectural limitation. More importantly, it reveals the engineering methodology that turned a design that didn't work into one that did.

The Design: Slotted Partition Pipeline

The Phase 6 design, documented in c2-optimization-proposal-6.md and read into the conversation at <msg id=1658>, proposed replacing the monolithic batch-all-then-prove model with a fine-grained slotted pipeline. The core insight was that with the Pre-Compiled Constraint Evaluator (PCE) from Phase 5, per-circuit synthesis time (3.55s) and per-circuit GPU proving time (3.4s) were nearly matched — the ideal condition for pipelining.

The design specified grouping partitions into slots of configurable size (slot_size), with a two-stage pipeline connected by a bounded channel:

                     bounded channel (cap=1)
                     ┌──────────────┐
[synth thread] ────→ │ SynthSlot    │ ────→ [GPU thread]
  partition k..k+S   │ (S circuits) │         gpu_prove()
                     └──────────────┘         → partial_proofs[k..k+S]

The channel capacity of 1 meant the synthesis thread could be at most one slot ahead of the GPU thread, bounding working memory to 2 × slot_size × 13.6 GiB. The design predicted dramatic improvements: for slot_size=2, 42.3s total time with 54 GiB peak memory; for slot_size=1, 38.9s with 27 GiB.

These predictions were derived from precise formulas grounded in the measured per-circuit timings. The design document was thorough, quantitative, and appeared to account for all variables. It even identified a risk: per-circuit synthesis time might increase at slot_size=1 because rayon would have fewer circuits to parallelize across. The mitigation was simple: "Benchmark; fallback to slot_size=2."

The Initial Implementation: Systematic and Coordinated

The implementation of the slotted pipeline, spanning messages <msg id=1666> through <msg id=1680>, was a model of systematic engineering. The assistant worked through a todo list with methodical precision, reading the existing codebase before writing any code, and verifying each edit's success before proceeding.

Refactoring C1 Deserialization

The first change, at <msg id=1666>, was refactoring C1 JSON deserialization. The existing synthesize_porep_c2_partition() function deserialized the 51 MB C1 JSON on every call. In the slotted pipeline, where partitions would be synthesized one slot at a time, this would mean 10 redundant parses — a significant overhead.

The solution was a ParsedC1Output struct that stored the deserialized vanilla proofs, replica_id, seed, comm_r, comm_d, and porep_config. The slotted pipeline function would deserialize once, then call a lightweight inner function per slot that reused the parsed data. This avoided redundant 51 MB parses while minimizing changes to the existing code path.

The ProofAssembler

Each GPU call produces slot_size × 192 bytes of proof data. These must be concatenated in partition order to form the final proof. The ProofAssembler struct, added at <msg id=1667>, provided a clean interface for collecting per-slot proof bytes, supporting out-of-order completion for future multi-GPU partition parallelism.

The Core Pipeline Function

The heart of the implementation was prove_porep_c2_slotted(), written using std::thread::scope with a bounded sync_channel(1). This function:

1. Deserialized the C1 JSON once into ParsedC1Output
2. Spawned a synthesis thread and a GPU thread within a std::thread::scope
3. Connected them with a sync_channel(1)
4. Iterated over partitions in groups of slot_size
5. On the synthesis side: built and synthesized each slot, sent to channel
6. On the GPU side: received slots, called gpu_prove(), fed results to ProofAssembler
7. When all slots completed, assembled and returned the final proof bytes The choice of std::thread::scope over tokio tasks or rayon threads was deliberate. The slotted pipeline needed precisely two threads with a synchronous channel between them. Scoped threads provide automatic joining when the scope exits, which is ideal for a self-contained pipeline that must complete before returning.

Configuration and Wiring

The slot_size parameter was added to PipelineConfig at <msg id=1668>, with semantics: slot_size = 0 means batch-all (backward compatibility), slot_size > 0 enables the slotted pipeline for PoRep C2 and SnapDeals.

The slotted pipeline was wired into the engine's process_batch method in a series of edits spanning <msg id=1669> through <msg id=1680>. This was the most intricate integration work, requiring careful threading of the slot_size parameter through the engine's call chain.

The SlottedBench Subcommand

To validate the implementation, a SlottedBench subcommand was added to the cuzk-bench binary at <msg id=1682> through <msg id=1684>. This subcommand accepted a --c1 input file path and a --slot-sizes list, ran the slotted pipeline for each slot size, and reported detailed timing breakdowns, RSS measurements, and GPU utilization statistics.

A key addition was GPU utilization tracking, added at <msg id=1685>. The benchmark recorded the GPU's active time percentage during each slot, providing insight into how effectively the overlap mechanism was working.

The First Benchmark: A Surprising Discovery

With compilation verified through a layered build sequence at <msg id=1716>–<msg id=1724>, the assistant launched the benchmark at <msg id=1725>:

cd /home/theuser/curio/extern/cuzk && CUDA_VISIBLE_DEVICES=0 \
  FIL_PROOFS_PARAMETER_CACHE=/data/zk/params \
  ./target/release/cuzk-bench slotted-bench \
  --c1 /data/32gbench/c1.json \
  --slot-sizes 10,5,2,1

The benchmark tested four configurations on an AMD Ryzen Threadripper PRO 7995WX (96 cores) with an RTX 5070 Ti GPU. The extracted results at <msg id=1727> told a story that didn't match the design predictions:

| slot_size | Total Time | Synthesis | GPU | GPU Active | Peak RSS | |-----------|-----------|-----------|-----|------------|----------| | 10 (baseline) | 63.4s | 36.7s | 26.5s | 42% | 227.9 GiB | | 5 | 98.3s | 65.0s | 49.6s | 50% | 141.2 GiB | | 2 | 177.8s | 149.8s | 123.1s | 69% | 98.0 GiB | | 1 | ~300s | ~290s | ~31s | ~10% | ~80 GiB |

These numbers were dramatically worse than the design predictions. slot_size=2 was supposed to take 42.3 seconds but took 177.8 seconds. slot_size=5 was supposed to take 52.8 seconds but took 98.3 seconds. Only slot_size=10 (the batch-all baseline) matched expectations at 63.4 seconds.

The b_g2_msm Fixed Cost

The analysis at <msg id=1729> revealed the root cause. The GPU's b_g2_msm operation — a multi-scalar multiplication in G2 — had a ~22-23 second fixed cost that did NOT scale with circuit count for batch sizes of 2 or more:

| slot_size | GPU per slot | b_g2_msm per slot | ntt_msm_h per slot | |-----------|-------------|-------------------|-------------------| | 10 (batch) | 26.5s | 23.6s | 23.7s | | 5 | 24.4s | 22.6s | 11.6s | | 2 | 24.5s | 22.8s | 4.7s | | 1 | 3.1s | 0.4s | 2.3s |

With num_circuits=1, b_g2_msm dropped to ~0.4s because the multi-threaded MSM implementation processes a single circuit efficiently. But with num_circuits>=2, the MSM had to aggregate across circuits, incurring a large fixed setup cost that dominated the total GPU time.

This finding upended the design document's core assumption of linear GPU scaling. The design was based on 10-circuit and 20-circuit measurements showing 3.4s per circuit, but those measurements were taken at batch sizes where the fixed b_g2_msm cost was amortized across many circuits. At smaller batch sizes, the fixed cost dominated.

The design document's risk assessment had identified "GPU fixed overhead higher than measured" as a "Low" likelihood risk. The benchmark proved this assessment wrong — the fixed overhead was not just higher than measured, it was catastrophically higher for small batch sizes.

The Redesign: True Parallel Synthesis with Per-Partition GPU Proving

The user's response at <msg id=1730> was direct and insightful: "The pipelines were meant to overlap. There should be essentially two independent sets of 'work slots' — 'gpu assigned work' and 'synth work slots'. Essentially the idea is that one synth slot = partition, gpu just chews on those as they come."

This reframed the problem. Instead of grouping partitions into slots and paying the b_g2_msm penalty for each group, the pipeline should:

1. Synthesize all partitions in parallel — all 10 partitions start synthesizing concurrently
2. Prove each partition individually on the GPU — num_circuits=1 gives the fast b_g2_msm path
3. Use channel backpressure to bound memory — a bounded channel limits how many synthesized partitions can be in-flight The assistant explored the GPU proving interface through subagent tasks at <msg id=1734> and <msg id=1735>, confirming that prove_from_assignments with num_circuits=1 was indeed efficient and that the b_g2_msm fixed cost only manifested with num_circuits>=2.

The New Architecture

The redesigned pipeline, implemented at <msg id=1744> through <msg id=1754>, replaced the single synth thread with parallel synthesis:

Synth Workers (10 threads, all start immediately)
  partition 0 ─── synthesize ───┐
  partition 1 ─── synthesize ───┤
  partition 2 ─── synthesize ───┼──> sync_channel(max_concurrent)
  ...                           │     (backpressure bounds RAM)
  partition 9 ─── synthesize ───┘
                                     GPU Consumer (1 thread)
                                       receives partitions as they arrive
                                       gpu_prove() ~3.8s each (num_circuits=1)
                                       assembles proofs by partition index

Key changes from the original design:

1. No slot grouping: Each partition is its own unit of work. The slot_size parameter was replaced with max_concurrent_slots, which controls how many synthesized partitions can be queued in the channel.
2. Parallel synthesis: All 10 partitions are synthesized concurrently via std::thread::scope. The bounded channel provides natural backpressure — if the channel is full, synthesis threads block until the GPU consumes a partition.
3. Per-partition GPU proving: Each GPU call processes exactly 1 circuit, triggering the fast b_g2_msm path (~0.4s multi-threaded instead of ~23s).
4. Out-of-order proof assembly: The ProofAssembler was redesigned to index by partition number rather than insertion order, supporting the arbitrary completion order that arises from parallel synthesis. The implementation required renaming the new function to prove_porep_c2_partitioned to avoid collision with the existing prove_porep_c2_pipelined function, a naming conflict discovered during the build at <msg id=1759>.

The Second Benchmark: Validation

The redesigned pipeline was benchmarked at <msg id=1769> with configurations max_concurrent=1,2,3,10:

| max_concurrent | Wall Time | Synth Sum | GPU Sum | GPU % | Overlap | Peak RAM | |---------------|-----------|-----------|---------|-------|---------|----------| | 1 | 72.0s | 351.4s | 38.6s | 54% | 5.42x | 71.3 GiB | | 2 | 72.7s | 353.1s | 38.0s | 52% | 5.38x | 86.8 GiB | | 3 | 71.9s | 347.8s | 38.2s | 53% | 5.37x | 86.8 GiB | | 10 (batch) | 62.3s | 36.2s | 25.5s | 41% | 0.99x | 228.5 GiB |

These results validated the redesign:

• 3.2× memory reduction: 71.3 GiB vs 228.5 GiB at max_concurrent=1
• 5.4× overlap: The sum of synthesis and GPU time (351.4s + 38.6s = 390s) completed in 72s wall time, a 5.42× overlap factor
• Only 16% latency overhead: 72.0s vs 62.3s batch-all baseline
• GPU per partition ~3.8s: Confirming the fast b_g2_msm path with num_circuits=1 The overlap factor of 5.4× was a direct measure of how well the pipeline was working. In the batch-all model, synthesis and GPU were purely sequential (0.99× overlap). In the pipelined model, 390 seconds of work completed in 72 seconds — the GPU was consuming partitions as fast as they arrived.

Why max_concurrent=1 Was Sufficient

A surprising finding was that max_concurrent=1 performed identically to max_concurrent=3. The reason is that synthesis is the bottleneck: each partition takes ~35s to synthesize (when 10 run concurrently, contention adds ~6s vs the single-partition baseline of 29s), while the GPU takes only ~3.8s per partition. With 10 partitions, the total synthesis time is ~36s (all parallel), and the GPU needs ~38s to consume them all. The channel capacity of 1 means at most one partition is queued ahead of the GPU, but since synthesis is ~9× slower than GPU per partition, the GPU never starves.

The max_concurrent parameter primarily controls peak RAM (each queued partition holds ~13.6 GiB of synthesized data), not throughput. At max_concurrent=1, peak RAM was 71.3 GiB; at max_concurrent=3, it rose to 86.8 GiB with no throughput benefit.

The Commit and Integration Verification

The working state was committed at <msg id=1773> with the message:

feat(cuzk): Phase 6 — pipelined partition proving with parallel synthesis

Redesign the slotted pipeline to truly pipeline partition synthesis with
GPU proving. All 10 partitions are synthesized in parallel (bounded by
channel capacity), and the GPU consumes them one at a time as they
arrive.

The commit touched 6 files with 1076 insertions, spanning the pipeline implementation, engine integration, configuration, bench subcommand, and dependency updates.

The Daemon E2E Testing: A Critical Architectural Insight

Following the commit, the user directed the assistant at <msg id=1776> to verify daemon integration and run full end-to-end tests across multiple concurrencies. The assistant dispatched subagent tasks at <msg id=1778> to explore the daemon code, confirming that the daemon's Engine::new() receives the full Config including slot_size, the process_batch method routes PoRep C2 through the partitioned pipeline when slot_size > 0, and the batch bench subcommand with -j N can drive concurrent proofs through the daemon via gRPC.

The e2e benchmark at <msg id=1798> tested six configurations — slot_size=0 (standard batch-all pipeline), slot_size=1,2,3,5 (partitioned), and slot_size=10 (batch-all via fallback) — each with 3 proofs at concurrency 2:

| slot_size | Throughput | s/proof | GPU prove avg | Peak RSS | |-----------|-----------|---------|---------------|----------| | 0 (standard) | 1.257 proofs/min | 47.7s | 27.0s | 370.8 GiB | | 1 (partitioned) | 0.833 proofs/min | 72.1s | 37.7s | 264.9 GiB | | 2 (partitioned) | 0.842 proofs/min | 71.3s | 38.0s | 264.8 GiB | | 3 (partitioned) | 0.822 proofs/min | 73.0s | 39.1s | 264.8 GiB | | 5 (partitioned) | 0.841 proofs/min | 71.3s | 38.9s | 261.0 GiB | | 10 (batch fallback) | 0.838 proofs/min | 71.6s | 37.4s | 264.8 GiB |

Two Critical Discoveries

1. The standard path (slot_size=0) was dramatically faster — 47.7s/proof vs ~72s for all partitioned modes. The reason was architectural: slot_size=0 uses the engine's two-stage pipeline (synthesis task → GPU worker via synth_tx channel), which enables inter-proof overlap. While proof N is on the GPU, proof N+1's synthesis starts. The partitioned path (slot_size > 0) runs entirely inside process_batch via spawn_blocking, blocking the synthesis task and preventing any overlap between proofs.

The timing diagram for the standard path showed beautiful interleaving:

Proof 1: synth=35.8s → GPU=27.1s  (total=63.2s, first proof, no overlap)
Proof 2: synth=40.6s → GPU=26.5s  (synth overlapped with proof 1 GPU)
Proof 3: synth=37.7s → GPU=27.5s  (synth overlapped with proof 2 GPU)

Steady-state throughput was approximately max(synth, GPU) ≈ 37-40s/proof — the GPU was fully utilized after the first proof.

2. Peak RSS of ~265 GiB for partitioned modes was far higher than expected — the in-process bench showed 71 GiB. This was because the daemon's -j 2 concurrency caused two proofs' worth of data to be live simultaneously. Each proof's partitioned pipeline held ~71 GiB of synthesized data, and with two proofs in flight (one being synthesized while the other was being GPU-proved), the total reached ~265 GiB. This was a consequence of the partitioned path blocking the synthesis task — the next proof couldn't start its synthesis until the current proof completed its GPU work, but the scheduler had already queued the next proof's data.

The Architectural Root Cause

The partitioned path's architecture had a fundamental limitation: it ran the entire pipeline — synthesis AND GPU proving — inside a single spawn_blocking call on the synthesis task. This meant:

1. The synthesis task was blocked for the full ~72s duration of each proof
2. The next proof in the scheduler queue couldn't start synthesis until the current proof's GPU work completed
3. No inter-proof overlap was possible In contrast, the standard path split synthesis and GPU proving into two separate stages connected by a channel. The synthesis task could start on proof N+1 as soon as proof N's synthesis completed, while proof N's GPU work was still in progress. The fix would require a significant architectural change: the partitioned path would need to send individual partitions (or groups of partitions) through the engine-level GPU channel, allowing the synthesis task to start on the next proof while the current proof's partitions are being GPU-proved. This is a natural direction for future work — it would combine the memory reduction of the partitioned pipeline with the inter-proof overlap of the standard pipeline.

Engineering Methodology: What Made This Work

The success of the Phase 6 pipelined partition prover was not the result of getting the design right on the first try. It was the result of a disciplined engineering methodology that treated the first design as a hypothesis to be tested, not a blueprint to be executed.

Design as Hypothesis

The design document made specific, quantitative predictions: 42.3s for slot_size=2, 38.9s for slot_size=1. These predictions were falsifiable — they could be proven wrong by measurement. When the benchmark showed 177.8s for slot_size=2, the design was falsified, and the assistant's analysis at <msg id=1729> identified exactly why: the b_g2_msm fixed cost.

This is the scientific method applied to engineering. A design is not a commitment; it's a hypothesis. The benchmark is the experiment. When the experiment contradicts the hypothesis, you update the hypothesis, not the data.

Root-Cause Analysis

When the benchmark results didn't match predictions, the assistant didn't just report "benchmark failed." It decomposed the GPU time into components (b_g2_msm, ntt_msm_h, etc.), identified the anomalous component, traced it to a mechanism (fixed cost per GPU invocation), and quantified the impact. This root-cause analysis was essential for designing the correct fix.

Reading Before Building

Before writing any code, the assistant read the existing codebase thoroughly. Messages <msg id=1663> through <msg id=1665> show the assistant reading pipeline.rs, engine.rs, config.rs, and the bench main.rs to understand the existing architecture. This reading phase identified the C1 deserialization inefficiency, the exact structure of the functions that would need to be modified, and the naming conflicts that would arise.

Layered Build Verification

Before running the benchmark, the assistant built all three targets — bench, core, and daemon — in a deliberate sequence that tested the changes across the entire dependency chain. This layered approach caught type errors, missing imports, and feature-gate inconsistencies before they could cause runtime failures.

Empirical Validation

The benchmark was designed as a systematic sweep across configurations, starting with the baseline and progressively testing finer granularities. The results were analyzed with root-cause reasoning, and the conclusions were reported honestly — including the failure of the original design's predictions.

Honest Reporting

When the design predictions proved incorrect, the assistant reported the discrepancy honestly, without spin or deflection. The analysis at <msg id=1729> explicitly stated: "The design doc's prediction of linear GPU scaling was based on 10-circuit measurements and doesn't hold for 2-5 circuit sub-batches." This intellectual honesty is essential for learning from empirical data and for maintaining trust in the engineering process.

Implications for the cuzk Project

The pipelined partition prover has several important implications for the cuzk project:

Memory-Constrained Deployments

The 3.2× memory reduction (71 GiB vs 228 GiB) makes PoRep proof generation feasible on machines with 64-128 GiB of RAM — machines that were previously incapable of running the proof pipeline at all. For Filecoin storage miners, this means lower hardware costs and more flexible deployment options.

The Throughput Tradeoff

The 16% latency overhead (72.0s vs 62.3s) in single-proof mode is a tradeoff that may be acceptable in memory-constrained deployments. However, the daemon e2e testing revealed that the partitioned path loses the inter-proof overlap that makes the standard path so efficient in steady state. The 47.7s/proof throughput of the standard path vs 72s/proof for the partitioned path is a 1.5× throughput difference that would be hard to ignore in production.

The Path Forward: Combining Memory Reduction with Inter-Proof Overlap

The natural next step is to redesign the partitioned path to send individual partitions through the engine-level GPU channel, rather than running the entire pipeline inside a spawn_blocking call. This would:

1. Preserve the memory reduction of per-partition GPU proving
2. Enable inter-proof overlap by freeing the synthesis task to start on the next proof
3. Allow the GPU to process partitions from multiple proofs concurrently This is a significant architectural change but one that could deliver the best of both worlds: the memory efficiency of the partitioned pipeline with the throughput of the standard pipeline.

The b_g2_msm Discovery

The discovery of the b_g2_msm fixed cost is itself a valuable finding. It suggests an optimization opportunity: if b_g2_msm could be run once and shared across multiple GPU calls, the per-partition GPU time could be reduced further. This is a potential direction for future work.

Conclusion

The Phase 6 pipelined partition prover represents a fundamental architectural change to the cuzk SNARK proving engine. By replacing the monolithic batch-all-then-prove model with a true producer-consumer pipeline where all 10 partitions synthesize concurrently and the GPU proves them one at a time, the implementation achieved a 3.2× memory reduction (71 GiB vs 228 GiB) with only 16% latency overhead (72.0s vs 62.3s).

The journey from design to validated implementation was not linear. The initial slot-grouping design was falsified by benchmark data that revealed an unexpected GPU fixed-cost structure. But the discipline of measurement, analysis, and iteration turned a design that didn't work into one that did. The key insight — that num_circuits=1 triggers a fast b_g2_msm path while num_circuits>=2 incurs a ~23s fixed cost — was discovered only because the benchmark was run and the results were analyzed with root-cause reasoning.

The daemon-level e2e testing then revealed a second-order architectural limitation: the partitioned path's spawn_blocking implementation prevented inter-proof overlap, capping throughput at 0.83 proofs/min vs 1.26 proofs/min for the standard path. This finding points the way to the next phase of optimization: integrating per-partition GPU proving with the engine-level GPU channel to combine memory efficiency with inter-proof overlap.

This is the essence of engineering as a scientific discipline. Designs are hypotheses. Benchmarks are experiments. Data, not intuition, determines the truth. And the willingness to be wrong — to have your predictions falsified and to learn from the failure — is what separates effective engineering from wishful thinking.

The pipelined partition prover is now committed, integrated, and ready for production deployment in memory-constrained environments. It makes PoRep proof generation accessible on a wider range of hardware, lays the foundation for multi-GPU scaling, and stands as a testament to the power of systematic, measurement-driven engineering.

References

[1] "The Slotted Pipeline: How a 4.2× Memory Reduction Was Engineered in the cuzk SNARK Proving Engine" — Chunk article covering the design document and initial implementation approach for Phase 6.