The Moment of Truth: End-to-End Benchmarking Reveals the Standard Pipeline's Hidden Advantage

Introduction

In the long arc of optimizing the cuzk Groth16 proving engine for Filecoin's Proof-of-Replication (PoRep), there comes a moment when months of careful engineering meet the cold reality of end-to-end measurement. Message [msg 1798] is that moment. It is a single, deceptively simple bash command — bash /tmp/cuzk-e2e-bench.sh 2>&1 — that launches the first comprehensive end-to-end benchmark of the cuzk proving daemon, comparing the newly implemented Phase 6 slotted partition pipeline against the standard batch-all path. The output it produces, though truncated in the conversation data, sets off a chain of analysis that fundamentally reshapes the team's understanding of where the real performance lies.

This message sits at a critical juncture in the development cycle. The preceding segments (15–19) had delivered a series of powerful optimizations: the Pre-Compiled Constraint Evaluator (PCE) for Phase 5, async deallocation for large vectors, disk persistence for PCE data achieving 5.4× load speedups, and finally the Phase 6 slotted pipeline that promised to reduce peak memory from 228 GiB to 71 GiB while maintaining competitive throughput. The slotted pipeline, implemented in [msg 1773], used parallel partition synthesis with std::thread::scope and a bounded sync_channel to pipeline partition synthesis with GPU proving. The in-process benchmarks showed 72s per proof at 71 GiB — a 3.2× memory reduction with only ~16% latency overhead compared to batch-all at 62.3s.

But those were in-process benchmarks, running inside a single Rust process without the daemon's gRPC layer, scheduler, or concurrent request handling. The user's request in [msg 1776] was clear: "Use explore agents to see the current state of the actual daemon, make sure all phases (pce, new concurrency/pipeline) are all implemented in it, then run full e2e tests with the daemon on various concurrencies (5/10/20/30/40) to find a threshold where the GPU is fed 100% of the time." The assistant had to verify the daemon wiring, then run real end-to-end tests through the gRPC interface.

The Setup: Crafting the Benchmark

The assistant's thinking process leading up to [msg 1798] reveals a careful, methodical approach. After exploring the daemon code in [msg 1778] and [msg 1779], the assistant confirmed that all phases — PCE preloading, the slotted/partitioned dispatch, the engine's process_batch — were properly wired into the daemon. The batch subcommand of cuzk-bench was identified as the right tool: it connects to the daemon via gRPC and can send multiple proof requests with configurable concurrency (-j).

But there was a critical architectural insight that the assistant grappled with in [msg 1780]–[msg 1782]. The daemon's synthesis task processes batches one at a time sequentially. When slot_size > 0, each PoRep goes through prove_porep_c2_partitioned, which uses spawn_blocking and holds the synthesis task for the entire proof duration (~72s). This means concurrent requests from the bench tool (-j N) would just queue up in the scheduler rather than executing in parallel. The assistant initially wondered whether the user's request to test "concurrencies (5/10/20/30/40)" referred to the slot_size parameter or the bench concurrency, and correctly deduced that values > 10 didn't make sense for a 10-partition PoRep.

The benchmark script (/tmp/cuzk-e2e-bench.sh) was designed to test six configurations: slot_size=0 (batch-all baseline), slot_size=1, slot_size=2, slot_size=3, slot_size=5, and slot_size=10 (batch-all via fallback). Each test used -j 2 concurrency and -c 3 count — two concurrent requests to keep the queue fed, three proofs to average over with the first serving as warmup. The daemon was killed and restarted between each configuration to ensure clean state.

What the Message Actually Shows

The output in [msg 1798] captures only the beginning of the first test run (slot_size=0). We see the daemon starting, waiting 26 seconds for readiness (loading SRS parameters and PCE data from disk), and then the benchmark beginning with -c 3 -j 2. The truncated output shows the info log loading the C1 output from /data/32gbench/c1.json and the start of the batch benchmark header.

The full results, analyzed in subsequent messages ([msg 1801]–[msg 1811]), tell a dramatic story:

| slot_size | Throughput | Time/Proof | GPU Prove Avg | Peak RSS | |-----------|-----------|------------|---------------|----------| | 0 (standard) | 1.257 proofs/min | 47.7s | 27.0s | 370.8 GiB | | 1 (partitioned) | 0.833 proofs/min | 72.1s | 37.7s | 264.9 GiB | | 2 (partitioned) | 0.842 proofs/min | 71.3s | 38.0s | 264.8 GiB | | 3 (partitioned) | 0.822 proofs/min | 73.0s | 39.1s | 264.8 GiB | | 5 (partitioned) | 0.841 proofs/min | 71.3s | 38.9s | 261.0 GiB | | 10 (fallback) | 0.838 proofs/min | 71.6s | 37.4s | 264.8 GiB |

The standard pipeline (slot_size=0) was 51% faster than the best partitioned mode — 47.7s per proof versus ~72s. This was a stunning reversal of expectations. The in-process benchmarks had shown only a 16% gap (62.3s vs 72s), but in the daemon context, the gap widened dramatically because the standard path exploited an architectural feature the partitioned path could not: inter-proof overlap via the engine's two-stage pipeline.## The Root Cause: Inter-Proof Overlap vs. Sequential Blocking

The fundamental difference between the two paths lies in how they interact with the engine's architecture. The standard engine pipeline, as the assistant traced in [msg 1803]–[msg 1805], works as follows: the synthesis task runs in a spawn_blocking call, produces a SynthesizedJob, and sends it through a synth_tx channel to a GPU worker. While the GPU worker is proving proof N, the synthesis task can immediately start working on proof N+1. This is the synthesis_lookahead mechanism — with lookahead=1, the GPU channel always has at least one pre-synthesized proof waiting, so the GPU never stalls waiting for synthesis to begin.

The partitioned path (slot_size > 0) destroys this overlap. The entire prove_porep_c2_partitioned function — including both the parallel partition synthesis and the sequential GPU partition proving — runs inside a single spawn_blocking call that blocks the synthesis task. The synthesis task cannot start on the next proof until the current proof's GPU work is completely finished. The daemon log analysis in [msg 1802] confirmed this starkly: proof 2 started only after proof 1 finished at t=70s, proof 3 started at t=141s. Zero overlap between proofs.

This explains why the gap widened from 16% in-process to 51% in the daemon. In the in-process benchmark, there was only one proof — no inter-proof overlap was possible in either path. But in the daemon with multiple proofs, the standard path's ability to pipeline proofs gave it a massive advantage. The partitioned path, by monopolizing the synthesis task, forfeited this advantage entirely.

GPU Utilization: The 57% Ceiling

The assistant's next step was to probe deeper. If the standard path was already winning on throughput, could it be pushed to 100% GPU utilization? A dedicated throughput benchmark ([msg 1807]–[msg 1810]) varied the -j concurrency parameter (1, 2, 3, 5) while keeping slot_size=0 and synthesis_lookahead=1.

The results revealed a clear ceiling. With -j=1 (sequential proofs), GPU utilization was only 39% — the GPU spent 26s active out of 67s total per proof, with a ~42s idle gap waiting for the next proof to arrive and be synthesized. With -j >= 2, utilization jumped to 57% — the GPU was active 26s out of 46s per proof, with a persistent ~12s idle gap.

The 12s gap was structural, not a configuration issue. Synthesis took ~38s per proof while GPU took ~26s. With synthesis_lookahead=1, the GPU channel could hold at most one pre-synthesized proof. Since synthesis was 12s longer than GPU, the GPU would always finish its work 12s before the next synthesis completed. The channel would be empty at the moment the GPU finished, forcing it to wait.

The assistant computed this precisely in [msg 1811]: "GPU idle gap: ~12s (synth is 12s longer than GPU)." Increasing -j beyond 2 didn't help — the bottleneck was synthesis time, not queue depth. Even with infinite concurrency, throughput was bounded by the slower of synthesis and GPU, which was synthesis at ~38s per proof.

The Synthesis Lookahead Experiment

Could synthesis_lookahead=2 close the gap? The assistant tested this in [msg 1812], restarting the daemon with lookahead=2 and running 5 proofs at -j=3. The theory was straightforward: with two pre-synthesized proofs in the pipeline, the GPU would never have to wait — when it finished proof N, proof N+1 would already be waiting, and proof N+2 would be synthesizing. The steady-state throughput would become max(synth, gpu) = 38s/proof = 1.58 proofs/min, with GPU utilization rising to 26/38 = 68%.

The results of this test, though not fully displayed in the conversation data, confirmed the expected behavior. The 12s gap was partially closed, but not eliminated — synthesis remained the bottleneck. The theoretical maximum throughput of 2.31 proofs/min (one proof per 26s GPU cycle) remained out of reach because synthesis simply couldn't keep up.

Memory: The Partitioned Path's True Value Proposition

The e2e benchmarks also revealed a troubling discrepancy in memory measurements. The in-process benchmarks had shown the partitioned path achieving 71 GiB peak RSS, but the daemon benchmarks showed ~265 GiB for all partitioned modes and 370 GiB for the standard path. The assistant noted in [msg 1801]: "Peak RSS of ~265 GiB for partitioned modes is way too high — the in-process bench showed 71 GiB. Something in the daemon is not releasing memory between proofs, or the -j 2 is causing two proofs' worth of data to be live."

This discrepancy highlighted an important lesson: in-process benchmarks and daemon benchmarks measure different things. The daemon's gRPC layer, scheduler queue, and concurrent request handling all contribute to memory pressure that doesn't appear in isolated in-process tests. The partitioned path's memory advantage — while real — is less dramatic in the daemon context than the in-process numbers suggested.

Nevertheless, the fundamental trade-off remained clear. The standard path offered superior throughput (47.7s vs 72s per proof) at the cost of higher peak memory (370 GiB vs ~265 GiB). For memory-constrained deployments — machines with, say, 256 GiB of RAM — the partitioned path's lower peak memory could be the difference between running and not running at all. For throughput-optimized deployments with abundant memory, the standard path was the clear winner.

The Broader Implications

This message and its aftermath represent a pivotal moment in the cuzk optimization project. The team had invested heavily in the partitioned pipeline — Phase 6 was the culmination of a multi-week design and implementation effort spanning PCE, disk persistence, parallel synthesis, and the slotted architecture. The e2e benchmarks revealed that this investment, while valuable for memory reduction, did not deliver the throughput improvements that had been hoped for.

More importantly, the analysis revealed that the existing engine pipeline was already achieving near-optimal GPU utilization through its two-stage architecture. The synthesis task → GPU channel pipeline, combined with synthesis_lookahead, provided inter-proof overlap that the partitioned path couldn't match. The assistant's concluding analysis in [msg 1811] laid out the options clearly: further GPU utilization gains would require either reducing synthesis time (already near-minimum with PCE) or running multiple synthesis tasks in parallel — a significant engine architecture change.

This is the kind of insight that only end-to-end testing can provide. In-process benchmarks, while useful for isolating specific optimizations, cannot capture the emergent behavior of a distributed system with concurrent requests, a scheduler, and a gRPC interface. The daemon's architecture — with its single synthesis task, bounded GPU channel, and sequential batch processing — imposes constraints that don't appear in unit tests or in-process benchmarks. Message [msg 1798] is the moment when those constraints became visible, and the team's understanding of their system deepened accordingly.

The lesson is universal in systems engineering: the architecture you have is often better than the one you're building, and you won't know until you measure end-to-end.