The Phoenix Architecture: How a 6.6× Performance Regression Forged the cuzk Phase 2 Pipelined Proving Engine

Introduction

In the development of high-performance cryptographic proving systems, there is a profound difference between code that compiles and code that performs. The cuzk pipelined SNARK proving engine for Filecoin had reached an important milestone: the Phase 2 architecture was fully implemented, with a forked bellperson library exposing the synthesis/GPU split, an SRS manager providing explicit parameter lifecycle control, and a pipeline module routing proofs through per-partition synthesis and GPU proving. The code compiled. The design was elegant. But elegance is not performance, and theory is not measurement.

This article chronicles the most dramatic arc in the cuzk project's history to date: the first end-to-end GPU test of the pipelined PoRep C2 path, the shocking discovery of a 6.6× performance regression (611 seconds versus a ~93-second monolithic baseline), the rapid implementation of a batch-all-partitions synthesis mode, the expansion of the pipeline to all Filecoin proof types, the successful validation that the batch mode matched the monolithic baseline at 91.2 seconds, and the architectural planning for the true async overlap that will deliver the throughput improvements Phase 2 was designed for.

This is a story of engineering discipline, honest measurement, and adaptive architecture. It demonstrates how the willingness to confront uncomfortable data—and to pivot when the data demands it—transforms a failing design into a stronger foundation.

The Crucible: First E2E GPU Test of the Per-Partition Pipeline

The Phase 2 pipeline implementation had been committed in commit beb3ca9c, adding 1,153 lines across nine files. The architecture was straightforward: split the monolithic Groth16 prover into per-partition units, synthesize each partition independently on the CPU, prove each partition on the GPU, and overlap these operations so that neither resource sits idle. The implementation had been verified with CPU-only builds, but the critical CUDA path—the actual GPU proving via the supraseal-c2 backend—had never been tested on real hardware.

Building the Test Infrastructure

The assistant's first step was to build the entire workspace with CUDA support enabled ([msg 537]). The command cargo build --workspace --features cuda-supraseal --release compiled all crates including the forked bellperson library, the cuzk-core proving engine, the daemon binary, and the bench utility. The build succeeded with only a single forward-compatibility warning from the bellperson fork's GPU lock code—a cosmetic issue that the assistant correctly deferred.

A temporary configuration file was written at /tmp/cuzk-pipeline-test.toml ([msg 539]), enabling the pipeline mode and setting the SRS parameter cache path to /data/zk/params. The daemon was started with this configuration, and the Structured Reference String—a 45 GiB set of elliptic curve points stored in CUDA-pinned host memory—was loaded in 15.4 seconds. The system was ready.

The Test That Changed Everything

The assistant submitted a real 32 GiB PoRep C2 proof using the cuzk-bench single tool ([msg 545]), referencing the pre-generated C1 output at /data/32gbench/c1.json. The daemon accepted the job and began processing through the per-partition pipeline. When the bash command timed out after 300 seconds ([msg 546]), the assistant pivoted to investigation mode, checking the daemon logs to understand what was happening.

The result, when it finally arrived after 611 seconds ([msg 547]), was both validating and devastating:

| Metric | Per-Partition Pipeline | Phase 1 Monolithic Baseline | |---|---|---| | Total time | 611.3 seconds | ~93 seconds | | Total synthesis | 568.8 seconds | ~55 seconds | | Total GPU | 40.5 seconds | ~35 seconds | | Proof size | 1920 bytes (valid) | 1920 bytes | | Speed ratio | 6.6× slower | 1.0× (baseline) |

The proof was valid. The bellperson fork correctly exposed the synthesis/GPU split. The SRS manager successfully shared parameters across partitions. The gRPC protocol handled the pipelined proof. The entire chain from Rust FFI through C++ CUDA kernels produced correct Groth16 proofs. This was a genuine engineering achievement—the pipeline worked.

But the performance was a catastrophe. At 6.6× slower than the monolithic Phase 1 baseline, the per-partition pipeline was not an optimization; it was a regression.

Root-Cause Analysis: Why Per-Partition Proving Failed

The assistant's analysis identified the root cause with surgical precision ([msg 552]). The monolithic seal_commit_phase2() function in Phase 1 does two things:

Synthesis for all 10 partitions in parallel using rayon's work-stealing thread pool, completing in ~55 seconds total.
GPU proving for all 10 partitions in one supraseal call, completing in ~35-40 seconds total. The per-partition pipeline, by contrast, serializes everything: synthesize partition 0 (~57s), GPU prove partition 0 (~4s), synthesize partition 1 (~57s), GPU prove partition 1 (~4s), and so on for all 10 partitions. The total is 10 × ~61s = ~610s. The critical insight was this: the pipeline's value proposition—overlapping CPU and GPU work—only materializes when there is a continuous stream of proofs. For a single proof, the per-partition approach is strictly worse because it loses the parallelism that rayon provides across partitions. The monolithic approach uses rayon to synthesize all 10 partitions simultaneously across available CPU cores. The per-partition approach throws this parallelism away, processing each partition sequentially. Even with perfect async overlap—where synthesis of partition N+1 runs concurrently with GPU proving of partition N—the best-case single-proof latency would be approximately 55 + 9 × max(55, 4) ≈ 550 seconds. Still far worse than the monolithic 93 seconds. The architecture was fundamentally mismatched to the single-proof use case. As the assistant articulated in [msg 552]: "The key insight: per-partition pipelining is designed for throughput on a stream of proofs, not single-proof latency. For single proofs, we should batch all 10 partitions. For pipeline overlap across multiple proofs, we can overlap at the proof level (synth proof N+1 while GPU proves proof N)." This insight would shape everything that followed.

The Pivot: Batch-All-Partitions Synthesis Mode

The assistant's response to this discovery was immediate and decisive. The todo list was updated ([msg 553]): the E2E GPU test was marked as completed (correctness validated), and a new high-priority task was created: "Fix per-partition performance: add batch-all-partitions synthesis+GPU mode for single proofs."

Design of the Batch Mode

The batch-all-partitions mode was implemented as synthesize_porep_c2_batch(). Instead of iterating over partitions one at a time, this function:

Builds all 10 circuits in a single Vec using StackedCompound::circuit() for each partition
Calls synthesize_circuits_batch() once with all 10 circuits, leveraging rayon's parallelization across all available CPU cores
Calls prove_from_assignments() once with all 10 assignments, proving them in a single GPU call
Serializes the combined 1920-byte proof from the 10 individual partition proofs This approach recovers the parallelism of the monolithic approach while preserving the split API architecture that the bellperson fork provides. The batch mode uses the same SynthesizedProof type, the same SRS manager, and the same GPU proving path—it simply changes the granularity from per-partition to all-partitions-at-once.

Implementation Details

The implementation required several coordinated changes across the cuzk codebase. The assistant re-read the three core source files—pipeline.rs, engine.rs, and prover.rs—in a single parallel read operation ([msg 554]), analyzing the exact function signatures, data structures, and dispatch logic before making any changes.

The new synthesize_porep_c2_batch function was added to pipeline.rs, alongside the existing per-partition synthesize_porep_c2 function. The engine's dispatch logic in engine.rs was updated to prefer the batch mode for single proofs while preserving the per-partition path for potential future use in streaming scenarios.

Expanding the Pipeline to All Proof Types

With the batch mode in place for PoRep C2, the assistant expanded the pipeline infrastructure to support the other three Filecoin proof types. This required inlining vanilla proof partitioning logic from filecoin-proofs because the relevant api module was private—a nontrivial reverse-engineering effort that avoided modifying the upstream dependency while keeping the cuzk crate self-contained.

PoSt Synthesis

The synthesize_post() function was added for WindowPoSt and WinningPoSt, using the FallbackPoStCompound circuit construction. The function follows the same pattern as the PoRep C2 synthesis:

Construct the circuit using the appropriate Compound type
Build the public inputs from the proof request parameters
Call synthesize_circuits_batch() to run the circuit's synthesis
Return a SynthesizedProof ready for GPU proving

SnapDeals Synthesis

The synthesize_snap_deals() function was added for SnapDeals, using the EmptySectorUpdateCompound circuit construction. Like the other synthesis functions, it follows the uniform pattern established by the split API.

Architectural Uniformity

This uniformity is a direct benefit of the split API. By separating synthesis from GPU proving, the pipeline can handle any proof type through the same gpu_prove() function, which calls prove_from_assignments() with the assignments and the appropriate SRS parameters. The assistant also made the necessary prover functions public and added the bincode dependency for serialization, ensuring the entire pipeline compiles cleanly.

GPU Validation: Batch Mode Matches the Baseline

The batch-all-partitions mode was then tested on the real GPU hardware ([msg 598], [msg 599]). The result was definitive:

=== Proof Result ===
status:    COMPLETED
job_id:    8f5ae01f-662a-4801-ae83-e3b2944a2f8c
timings:   total=91170 ms (queue=22 ms, srs=0 ms, synth=55736 ms, gpu=35212 ms)
wall time: 91278 ms
proof:     1920 bytes

The batch-mode pipeline produced a valid 1920-byte proof in 91.2 seconds—matching the monolithic Phase 1 baseline of ~93 seconds within measurement noise. The comparison tells the full story:

| Metric | Per-Partition (old) | Batch Mode (new) | Phase 1 Monolithic | |---|---|---|---| | Total | 611.3s | 91.2s | ~93s | | Synthesis | 568.8s | 55.7s | ~55s | | GPU | 40.5s | 35.2s | ~35s | | Proof size | 1920 bytes | 1920 bytes | 1920 bytes | | SRS loading | N/A | 0 ms | ~15s per invocation |

The batch-mode pipeline not only recovered the lost performance but slightly beat the monolithic baseline (91.2s vs ~93s), while providing the additional benefit of separate synthesis and GPU timing breakdowns that the monolithic path could not expose. The SRS residency (0 ms loading time) contributed to this improvement—the SRS manager kept parameters loaded in GPU memory across invocations, eliminating the reload overhead that the monolithic prover incurred on each call.

What This Validation Confirmed

This validation was critical for several reasons:

Bellperson fork correctness: The prove_from_assignments() function works correctly when called with all 10 partitions' assignments in a single invocation, producing a valid Groth16 proof.
SRS manager compatibility: The direct parameter loading path via SuprasealParameters::new() produces a compatible SRS handle for the GPU prover, with zero reload overhead.
Pipeline architecture viability: The split API, SynthesizedProof type, and modular synthesis functions can match the monolithic baseline while providing the foundation for future optimizations.
End-to-end correctness: The entire chain from gRPC request through Rust FFI to C++ CUDA kernels produces valid proofs.

The Administrative Close: Marking Milestones

With the validation complete, the assistant updated the persistent todo list in [msg 601], marking three high-priority items as completed:

"E2E GPU test: build with --features cuda-supraseal, run PoRep C2 through pipeline, verify proof" — The end-to-end validation of the pipeline architecture in a real GPU environment.
"Fix per-partition performance: add batch-all-partitions synthesis+GPU mode for single proofs" — The resolution of the 6.6× regression.
"Add pipelined PoSt synthesis (WinningPoSt + WindowPoSt) to pipeline.rs" — The expansion of the pipeline to all Filecoin proof types. This administrative action, seemingly mundane, was the formal close of a chapter. The Phase 2 pipeline, born from the ashes of a failed per-partition approach, now stood as a viable foundation for the next stage of the project.

The Revised Architecture: Two Modes for Two Use Cases

The key architectural insight that emerged from this work is that the Phase 2 pipeline needs two modes:

Batch Mode (Single Proof)

Synthesize all partitions at once using rayon parallelism, prove all partitions in one GPU call. Used when there is a single proof to process. Matches monolithic performance at ~91 seconds.

Pipeline Mode (Multiple Proofs)

Synthesize proof N+1 on the CPU while the GPU proves proof N. Used when there is a stream of proofs. Provides the overlap benefit that Phase 2 was designed for, but only across separate proof jobs, not within a single proof.

The engine's dispatch logic can choose between modes based on the queue depth. If multiple proofs are waiting, it uses pipeline mode to overlap their processing. If only one proof is in the queue, it uses batch mode for optimal single-proof latency.

This two-mode architecture is a pragmatic compromise. It preserves the Phase 2 investment in the bellperson fork, SRS manager, and pipeline infrastructure, while avoiding the single-proof regression that the pure per-partition approach would cause. The async overlap across proofs—the true value of the pipeline—remains as a future implementation target, now with a clear understanding of when and how it provides benefit.

Planning the Async Overlap Architecture

With single-proof latency resolved, the assistant turned to the original goal of Phase 2: throughput on a continuous stream of proofs. In [msg 602] and [msg 603], the assistant laid out the design for true async overlap.

The Producer-Consumer Model

The design envisions a bounded channel (synth_queue) between synthesis and GPU workers:

A dedicated synthesis task that continuously pulls proof requests from the scheduler, runs synthesis on a blocking thread, and pushes SynthesizedProof objects into the channel
GPU workers that consume SynthesizedProof objects from the channel and run gpu_prove()
Backpressure via the bounded channel, preventing the system from accumulating an unbounded backlog of synthesized proofs The overlap works as follows: while the GPU worker proves proof N, the synthesis worker can already be working on proof N+1. For a stream of proofs, the GPU never sits idle waiting for synthesis—there is always a pre-synthesized proof ready in the channel.

Key Design Considerations

The assistant identified three key complications:

Job tracking across stages: A job's state must be maintained across two independent tasks (synthesis and GPU proving), requiring careful coordination.
Completion channels: The gRPC AwaitProof RPC must be wired through both stages, so the client can block until the proof is complete regardless of which stage is processing it.
Memory pressure: Each SynthesizedProof contains the a/b/c evaluation vectors and witness assignments, which can be ~13.6 GiB per partition for PoRep C2. The bounded channel can only hold synthesis_lookahead of these intermediate states in flight at once, preventing unbounded memory growth. The key insight, articulated in [msg 603], was that the single-GPU case simplifies the design considerably: "for a single GPU (most common case), the pipeline overlap happens naturally if we have a separate synthesis thread that stays one proof ahead. For multi-GPU, each GPU pulls from the same synth_queue." This insight transforms the async overlap from a complex multi-worker orchestration problem into a straightforward producer-consumer pattern.

Lessons in Engineering Discipline

The work in this segment teaches several valuable lessons about systems engineering:

Test Early, Test Often

The assistant could have spent weeks implementing PoSt/SnapDeals synthesis and async overlap before discovering the performance regression. Running the E2E GPU test early—before extending the pipeline to other proof types—saved enormous wasted effort. The discipline of validating the foundation before building the walls is what made this discovery possible.

Measure Before Optimizing

The per-partition pipeline was designed to improve throughput, but no baseline throughput measurement was taken before implementation. The monolithic Phase 1 baseline of ~93 seconds provided the comparison point that revealed the regression. Without this baseline, the 611-second result might have been accepted as "good enough."

Be Willing to Change Direction

The assistant had invested significant effort in the per-partition pipeline: forking bellperson, implementing the SRS manager, writing the pipeline module, and wiring it into the engine. It would have been easy to rationalize the poor performance or claim that the design was still correct and just needed tuning. Instead, the assistant accepted the data, understood why the result occurred, extracted the correct lesson, and adjusted the plan. This intellectual honesty is rare and valuable.

Distinguish Latency from Throughput

Many performance optimizations improve one at the expense of the other. The per-partition pipeline improves potential throughput (by enabling overlap across proofs) but degrades single-proof latency (by serializing partition processing). Understanding which metric matters for your use case is essential. The two-mode architecture—batch for latency, pipeline for throughput—is the correct resolution of this tension.

Maintain Compilation Discipline

Throughout the implementation, the assistant maintained a pattern of "compile, fix, compile again"—running cargo check after every change, fixing warnings and errors immediately, and never allowing the codebase to drift into an uncompilable state. All 15 unit tests pass with zero warnings from cuzk code, and both CUDA and non-CUDA builds compile cleanly.

Conclusion

This segment of the cuzk development session captures a complete arc of engineering: design, implementation, testing, discovery, analysis, and adaptation. The per-partition pipeline was not wrong—it was incomplete. The missing pieces were the batch mode for single-proof latency and the async overlap across proofs for throughput. Both have now been identified, and the batch mode has been implemented and validated at 91.2 seconds, matching the monolithic baseline.

The cuzk project now has a solid Phase 2 foundation: a validated bellperson fork that exposes the synthesis/GPU split, an SRS manager that provides explicit parameter lifecycle control, a pipeline module with synthesis functions for all four proof types, and a clear architectural vision for the async overlap that will deliver the throughput improvements Phase 2 was designed for. The 6.6× regression was not a failure—it was a discovery that made the architecture stronger.

The story of the cuzk proving engine's Phase 2 is a testament to the power of empirical engineering. When the per-partition pipeline failed, the assistant did not abandon the pipeline vision. Instead, the assistant understood why it failed, extracted the correct architectural lesson, and built a two-mode system that handles both single-proof latency and multi-proof throughput. The result is not just a working pipeline—it is a pipeline that has been tested, measured, and proven on real GPU hardware, producing valid Filecoin proofs at competitive speeds.

The next chapter—the async overlap implementation—will build on this validated foundation, transforming the cuzk proving engine from a batch-oriented system into a streaming pipeline optimized for the heterogeneous cloud rental markets that motivated the entire project.