The Proving Pipeline Showdown: How End-to-End Benchmarking Revealed the True Value of Partitioned Proof Generation in cuzk

Introduction

In the sprawling investigation of the SUPRASEAL_C2 Groth16 proof generation pipeline for Filecoin's Proof-of-Replication (PoRep) system, a pivotal moment arrived when the conversation shifted from static code analysis to dynamic performance measurement. The earlier phases of the session had produced comprehensive documentation of the pipeline architecture, identified nine structural bottlenecks, and proposed three ambitious optimization proposals. But all of that work rested on a critical unanswered question: how does the system actually perform under load?

This article synthesizes the work captured in segment 20, chunk 0 of the opencode session — a phase that combined a deep-dive subagent exploration of the daemon's wiring with rigorous end-to-end benchmarking of the cuzk proving daemon. The results were surprising and consequential. The standard pipeline path, which the team had assumed was the legacy path, turned out to dramatically outperform the newer partitioned path in throughput. And the partitioned path, originally designed for throughput improvements, revealed its true value proposition: dramatic memory reduction at the cost of some performance. These findings reshaped the entire optimization strategy.

Context: The Investigation So Far

The root session had already accomplished a remarkable amount. It had mapped the full call chain from Curio's Go task layer through Rust FFI into C++/CUDA kernels, accounting for the ~200 GiB peak memory footprint of the SUPRASEAL_C2 proof generation pipeline. It had produced a comprehensive background reference document identifying nine structural bottlenecks and three composable optimization proposals: Sequential Partition Synthesis (streaming partitions sequentially to reduce peak memory), Persistent Prover Daemon (eliminating SRS loading overhead across proofs), and Cross-Sector Batching (improving throughput by batching multiple sectors' circuits).

But these proposals were built on assumptions about how the cuzk-daemon actually worked — assumptions that needed to be verified against the real source code and real performance data. The user's request that launched this phase was precise: "I need a thorough exploration of how the cuzk-daemon is wired up end-to-end." The user specified five files to read in full, with four specific verification goals: confirming that PCE (Pre-Compiled Circuit Evaluator) is auto-extracted on the first proof, that the partitioned pipeline is actually called when slot_size > 0, that slot_size flows from daemon config through to the engine, and that the standard batch path still works for slot_size=0 or multi-sector batches.

The Subagent Exploration: Mapping the Daemon Wiring

To answer these questions, a subagent session was launched with the focused task of exploring the daemon wiring. What followed was a masterclass in systematic codebase analysis, spanning eight messages of file discovery, parallel reads, continuation reads for truncated files, and ultimately a comprehensive synthesis.

The assistant's approach was methodical. It began with glob commands to discover all relevant source files ([msg 1]), confirming the existence of main.rs, Cargo.toml, service.rs, lib.rs, engine.rs, pipeline.rs, config.rs, types.rs, batch_collector.rs, prover.rs, srs_manager.rs, and others. Then, in a single parallel burst ([msg 2]), it read nine key files simultaneously — the daemon entry point, the daemon's Cargo.toml, the gRPC server files, the engine, the batch collector, the config types, and the core library entry point [9].

This parallel read strategy was deliberate. The assistant recognized that none of these reads depended on each other, so they could all be dispatched together, minimizing wall-clock time. As one analysis noted, "the assistant is optimizing for wall-clock time, recognizing that file I/O is independent and can be parallelized without conflict" [9].

But the reads encountered a practical obstacle: engine.rs and pipeline.rs were too large to return in a single invocation. The assistant handled this with targeted continuation reads ([msg 3]), reading the rest of engine.rs starting from line 985 and the beginning of pipeline.rs [8]. This was followed by another continuation read ([msg 4]) targeting the partitioned pipeline section at line 1370 of pipeline.rs, alongside a glob for the workspace-level Cargo.toml [6].

The assistant also traced the feature flag configuration, reading the workspace Cargo.toml ([msg 5]) and then cuzk-core/Cargo.toml ([msg 7]) to understand how the cuda-supraseal feature — which gates all CUDA-specific code paths — flows from the library's default features through the workspace dependency declaration to the daemon [4][5].

The culmination of this exploration was a comprehensive 1,500-word analysis ([msg 8]) that traced the entire wiring from daemon configuration to GPU kernel execution. The analysis confirmed all four verification points: PCE auto-extraction is triggered on the first proof in both the partitioned and standard paths; the partitioned pipeline is called only when all three conditions are met (PoRepSealCommit, single request, and slot_size > 0); slot_size flows from Config.pipeline.slot_size through Engine::new() to engine.start() and into every process_batch() call; and the standard batch path remains functional for slot_size=0, multi-sector batches, and non-PoRep proof types [7].

The End-to-End Benchmark: Standard vs. Partitioned Paths

With the wiring confirmed, the investigation pivoted to performance measurement. The assistant ran comprehensive end-to-end benchmarks using the daemon's gRPC interface, comparing the standard pipeline path (slot_size=0) against the partitioned path (slot_size>0). The results were striking.

The standard pipeline path completed a proof in approximately 47.7 seconds. The partitioned path required approximately 72 seconds — a 51% increase in proof time.

Why such a dramatic difference? The answer lies in the engine's two-stage architecture. In the standard path, the engine runs a synthesis task that feeds into a GPU channel. The key insight is that synthesis of proof N+1 can overlap with GPU proving of proof N. While the GPU is working on one proof's MSM and NTT operations, the CPU can be synthesizing the next proof's circuit. This inter-proof overlap is the engine's native optimization.

The partitioned path, by contrast, runs the entire proof (synthesis + GPU) inside a single spawn_blocking call. This means the synthesis task is blocked for the entire duration of the proof — it cannot start working on the next proof until the current one is completely finished. The inter-proof overlap is destroyed.

This was a crucial discovery. The partitioned path, which had been designed as a Phase 6 enhancement to the pipeline, was actually regressing throughput compared to the standard path. The very mechanism that made the standard path efficient — the producer-consumer pipeline between synthesis and GPU — was being bypassed.

GPU Utilization and Throughput Analysis

The benchmarks went deeper. The assistant tested varying concurrency levels (-j 1, 2, 3, 5) to understand how the system scales with multiple synthesis tasks. The results revealed a clear bottleneck profile.

GPU utilization saturated at approximately 57% with -j >= 2, achieving a throughput of 1.3 proofs per minute. The bottleneck was synthesis time (~38 seconds) consistently exceeding GPU time (~26 seconds), leaving a ~12 second GPU idle gap between proofs. Even with synthesis_lookahead=2 — which allows the synthesis task to queue more pre-synthesized jobs ahead of the GPU — the gap could not be fully closed because synthesis remained the longer phase.

This finding has profound implications. The GPU, which is typically the most expensive resource in a proving system, was spending nearly half its time idle. The system was CPU-bound, not GPU-bound. Any optimization that reduces synthesis time would directly translate to improved GPU utilization and higher throughput.

The analysis also revealed that further GPU utilization gains would require either reducing synthesis time (already near-minimum with PCE enabled) or running multiple synthesis tasks in parallel — a significant engine architecture change that would require rethinking the single-synthesis-task model.

The Shifted Value Proposition: Memory vs. Throughput

The most consequential finding of this benchmarking phase was a fundamental shift in the value proposition of the partitioned path.

The partitioned path was originally conceived as a throughput improvement — by splitting a sector's 10 PoRep partitions into separate jobs, the system could process them in parallel and potentially finish faster. But the benchmarks showed the opposite: the partitioned path was slower.

However, the partitioned path had another benefit that the benchmarks made clear: memory reduction. The standard path required approximately 228 GiB of peak memory, driven by holding all 10 partition circuits in memory simultaneously plus the ~48 GiB SRS in pinned GPU memory. The partitioned path, by streaming partitions one at a time through the GPU, reduced peak memory to approximately 71 GiB — a 69% reduction.

This reframed the entire optimization strategy. The partitioned path was not a throughput optimization; it was a memory optimization. Its value was not in proving faster, but in proving on hardware that couldn't otherwise fit the 228 GiB footprint. For memory-constrained deployments — cloud instances with limited RAM, shared GPU clusters, or edge nodes — the partitioned path made proof generation possible where it would otherwise be impossible.

The standard path remained optimal for throughput. If you have the memory headroom, use the standard path and get proofs in 47.7 seconds. If you're memory-constrained, use the partitioned path and accept 72 seconds in exchange for fitting into 71 GiB.

Implications for the Optimization Proposals

These findings had direct implications for the three optimization proposals developed earlier in the session.

Sequential Partition Synthesis — the proposal to stream partitions sequentially to reduce peak memory — was validated by the benchmarks. The partitioned path already demonstrated this capability, reducing memory from 228 GiB to 71 GiB. The proposal could build on this foundation, potentially reducing memory further by optimizing the streaming mechanism or by reducing the SRS pinned memory allocation.

Persistent Prover Daemon — the proposal to eliminate SRS loading overhead by keeping the proving process alive across proofs — was shown to be even more valuable than initially estimated. The benchmarks revealed that the ~60 seconds per proof for SRS loading was a significant fraction of total proof time. Eliminating this overhead would directly improve throughput for both the standard and partitioned paths.

Cross-Sector Batching — the proposal to batch multiple sectors' circuits into single GPU invocations — was identified as the most promising path to significant throughput gains. The benchmarks showed that GPU utilization was only 57%, meaning there was substantial headroom for batching. By using the memory freed by the partitioned path to hold multiple sectors' circuits, the system could potentially achieve 2-3× throughput per GPU and 5-6× reduction in cost per proof when combined with the other proposals.

The Micro-Optimization Analysis

A final round of analysis examined the computational hotpaths at the instruction level. The CPU synthesis hotpaths were characterized by SHA-256 bit manipulation operations and Fr (field) arithmetic — the fundamental operations of circuit synthesis. The GPU NTT (Number Theoretic Transform) and MSM (Multi-Scalar Multiplication) operations were found to be memory-bandwidth-bound rather than compute-bound, meaning that further GPU optimization would require improving memory access patterns rather than raw compute throughput.

The analysis also examined the feasibility of recomputing a/b/c vectors on-the-fly to avoid materializing them in memory. This was identified as a potential optimization for the partitioned path, where the memory savings from not storing these intermediate vectors could further reduce the peak footprint.

Conclusion

Segment 20 of this investigation represents a critical inflection point. The subagent exploration of the daemon wiring confirmed that the architecture was sound — the configuration flowed correctly, the conditional logic for path selection was correct, and the PCE extraction was properly automated. But the end-to-end benchmarks revealed a more nuanced truth than the optimization proposals had assumed.

The standard path was faster. The partitioned path was more memory-efficient. Neither was universally superior; each had its place depending on the deployment constraints. The optimization strategy had to shift from "make the partitioned path faster" to "use the right path for the right deployment."

This is the kind of insight that only comes from measurement. Static code analysis can tell you what the code can do, but only benchmarking can tell you what it actually does under real conditions. The cuzk proving daemon, with its dual-path architecture, turned out to be more sophisticated than a simple optimization story. It was a system that offered a trade-off — memory for throughput — and the right choice depended on the hardware and the business requirements.

The investigation concluded with a clear roadmap: for throughput-optimized deployments, use the standard path with synthesis_lookahead tuning; for memory-constrained deployments, use the partitioned path and consider Sequential Partition Synthesis refinements; and for maximum cost efficiency, pursue Cross-Sector Batching to exploit the GPU headroom revealed by the benchmarks. The architecture was sound. The path forward was clear. And it was all built on the foundation of reading the code, running the benchmarks, and letting the data speak.