The Reckoning: How End-to-End Benchmarks Redefined the Value of a Pipelined SNARK Proving Architecture

Introduction

In the lifecycle of any complex optimization effort, there arrives a moment of reckoning — a point where carefully crafted component-level improvements must face the messy, unforgiving reality of end-to-end system behavior. For the cuzk SNARK proving engine, a GPU-accelerated Groth16 proof generator designed for Filecoin's Proof-of-Replication (PoRep) consensus, that moment arrived in Segment 20 of the opencode session. And the results were nothing short of revelatory.

The preceding segments had built an impressive edifice of optimization. Phase 4 delivered synthesis hot-path improvements achieving 13.2% end-to-end speedup. Phase 5 introduced the Pre-Compiled Constraint Evaluator (PCE), a multi-threaded constraint evaluation system that dramatically accelerated synthesis. Phase 6 — the most ambitious yet — implemented a slotted partition pipeline that broke the monolithic 10-partition PoRep proof into individually synthesized and proven partitions, achieving a 3.2× memory reduction (71 GiB vs 228 GiB peak) in in-process benchmarks with only ~16% latency overhead. The partitioned pipeline had been validated, documented, and committed. It was time to see how it performed in the real daemon.

What the end-to-end benchmarks revealed was a fundamental reordering of priorities. The standard pipeline path — the one that had been treated as the baseline to beat — turned out to be dramatically faster than the partitioned path in the daemon context (47.7s vs 72s per proof), because it leveraged an architectural feature the partitioned path inadvertently blocked: inter-proof overlap through the engine's two-stage synthesis→GPU pipeline. The partitioned path's value proposition shifted overnight from throughput improvement to memory reduction. This is the story of that discovery, the analysis that followed, and the strategic reorientation it triggered.

The Setup: Verifying the Daemon Wiring

The user's directive at [msg 1776] was precise and demanding: "Use explore agents to see the current state of the actual daemon, make sure all phases (pce, new concurrency/pipeline) are all implemented in it, then run full e2e tests with the daemon on various concurrencies (5/10/20/30/40) to find a threshold where the GPU is fed 100% of the time."

The assistant responded by dispatching two parallel exploration subagents at [msg 1778]. The first traced the daemon's call chain from main.rs through the engine's process_batch method, confirming that PCE preloading and partitioned pipeline dispatch were correctly wired into the production entry point. The second explored the cuzk-bench tool's e2e capabilities, identifying the batch subcommand with its -j concurrency parameter as the right tool for the job.

The exploration revealed a critical architectural constraint that would shape everything that followed. In [msg 1780]–[msg 1782], the assistant read the engine's synthesis task loop and discovered that it processes batches one at a time sequentially. When slot_size > 0, each PoRep proof goes through prove_porep_c2_partitioned, which uses spawn_blocking — meaning it holds the synthesis task thread for the entire duration of the proof (~72 seconds). Concurrent requests from the bench tool (-j N) would simply queue up in the scheduler; they would not execute in parallel.

This realization triggered a profound reinterpretation of the user's request. The user had asked for "concurrencies (5/10/20/30/40)" — but what did "concurrency" mean in this context? The assistant considered two interpretations: the -j concurrency parameter of the bench tool, controlling how many proof requests are sent simultaneously, and the slot_size parameter of the partitioned pipeline, controlling how many synthesized partitions can be buffered before the GPU consumer. The assistant correctly noted that values 20/30/40 don't make sense for slot_size since PoRep has only 10 partitions — any value above 10 triggers a fallback to batch-all behavior.

After wrestling with this ambiguity, the assistant settled on a pragmatic test plan at [msg 1782]: vary slot_size across the meaningful range (0, 1, 2, 3, 5, 10) while keeping -j fixed at 2 (sufficient to keep the queue fed) and -c at 3 (three proofs for averaging, with the first serving as warmup). This plan would isolate the effect of the partitioned pipeline configuration from the confounding variable of request concurrency.

The Benchmark: A Cascade of Revelations

The assistant wrote a shell script (/tmp/cuzk-e2e-bench.sh) that would start the daemon with each configuration, wait for it to be ready (loading SRS parameters and PCE data), run the benchmark, kill the daemon, and repeat. After a brief detour debugging a CLI flag mismatch at [msg 1792]–[msg 1794] (the bench tool used -c for count, not -n), the script executed successfully across all six configurations.

The results, analyzed in [msg 1801], were stunning:

| slot_size | Throughput | s/proof | GPU prove avg | Peak RSS | |-----------|-----------|---------|---------------|----------| | 0 (standard) | 1.257 proofs/min | 47.7s | 27.0s | 370.8 GiB | | 1 (partitioned) | 0.833 proofs/min | 72.1s | 37.7s | 264.9 GiB | | 2 (partitioned) | 0.842 proofs/min | 71.3s | 38.0s | 264.8 GiB | | 3 (partitioned) | 0.822 proofs/min | 73.0s | 39.1s | 264.8 GiB | | 5 (partitioned) | 0.841 proofs/min | 71.3s | 38.9s | 261.0 GiB | | 10 (fallback) | 0.838 proofs/min | 71.6s | 37.4s | 264.8 GiB |

The standard pipeline (slot_size=0) was 51% faster than the best partitioned mode — 47.7s per proof versus ~72s. Every single partitioned configuration, regardless of slot_size, clustered tightly around 71–73 seconds per proof. Even slot_size=10, which triggered a fallback to batch-all behavior within the partitioned function, was no better.

This was a complete reversal of expectations. The in-process benchmarks had shown only a 16% gap (62.3s vs 72s), but in the daemon context, the gap widened dramatically. The standard path was not just marginally better — it was decisively superior.

The assistant's analysis in [msg 1801] identified the root cause with surgical precision. The standard engine pipeline works as follows: the synthesis task runs in a spawn_blocking call, produces a SynthesizedJob, and sends it through a synth_tx channel to a GPU worker. While the GPU worker is proving proof N, the synthesis task can immediately start working on proof N+1. This is the synthesis_lookahead mechanism — with lookahead=1, the GPU channel always has at least one pre-synthesized proof waiting, so the GPU never stalls waiting for synthesis to begin.

The partitioned path destroys this overlap. The entire prove_porep_c2_partitioned function — including both the parallel partition synthesis and the sequential GPU partition proving — runs inside a single spawn_blocking call that blocks the synthesis task. The synthesis task cannot start on the next proof until the current proof's GPU work is completely finished. The daemon log analysis confirmed this starkly: proof 2 started only after proof 1 finished at t=70s, proof 3 started at t=141s. Zero overlap between proofs.

This explains why the gap widened from 16% in-process to 51% in the daemon. In the in-process benchmark, there was only one proof — no inter-proof overlap was possible in either path. But in the daemon with multiple proofs, the standard path's ability to pipeline proofs gave it a massive advantage. The partitioned path, by monopolizing the synthesis task, forfeited this advantage entirely.

The GPU Utilization Analysis: Probing the 57% Ceiling

Having established that the standard path was the throughput winner, the assistant's next task was to probe deeper. Could the standard path be pushed to 100% GPU utilization? The user had asked to "find a threshold where the GPU is fed 100% of the time." The assistant designed a dedicated throughput benchmark at [msg 1807] that varied the -j concurrency parameter (1, 2, 3, 5) while keeping slot_size=0 and synthesis_lookahead=1.

The results, analyzed across [msg 1808]–[msg 1811], revealed a clear and persistent ceiling. With -j=1 (sequential proofs), GPU utilization was only 39% — the GPU spent 26s active out of 67s total per proof, with a ~42s idle gap waiting for the next proof to arrive and be synthesized. With -j >= 2, utilization jumped to 57% — the GPU was active 26s out of 46s per proof, with a persistent ~12s idle gap.

The 12s gap was structural, not a configuration issue. Synthesis took ~38s per proof while GPU took ~26s. With synthesis_lookahead=1, the GPU channel could hold at most one pre-synthesized proof. Since synthesis was 12s longer than GPU, the GPU would always finish its work 12s before the next synthesis completed. The channel would be empty at the moment the GPU finished, forcing it to wait.

The assistant computed this precisely in [msg 1811]: "GPU idle gap: ~12s (synth is 12s longer than GPU)." Increasing -j beyond 2 didn't help — the bottleneck was synthesis time, not queue depth. Even with infinite concurrency, throughput was bounded by the slower of synthesis and GPU, which was synthesis at ~38s per proof.

The theoretical maximum throughput, the assistant calculated, was 1 proof per GPU cycle = 26s/proof = 2.31 proofs/min. The current bottleneck was synthesis at 38s/proof, running sequentially in a single synthesis task. Even with infinite lookahead, throughput would be max(synth, gpu) = 38s/proof = 1.58 proofs/min — a 68% GPU utilization ceiling.

This was a sobering calculation. The GPU, which had been the focus of so much optimization effort, was not the bottleneck. Synthesis was. And synthesis had already been heavily optimized — the PCE had reduced it from ~45s to ~38s, and further gains would require either algorithmic breakthroughs or parallelizing the synthesis task itself, a significant engine architecture change.

The Synthesis Lookahead Experiment

Could synthesis_lookahead=2 close the 12s gap? The assistant tested this hypothesis at [msg 1812], restarting the daemon with lookahead=2 and running 5 proofs at -j=3. The theory was straightforward: with two pre-synthesized proofs in the pipeline, the GPU would never have to wait. When it finished proof N, proof N+1 would already be waiting in the channel, and proof N+2 would be actively synthesizing. The steady-state throughput would become max(synth, gpu) = 38s/proof = 1.58 proofs/min, with GPU utilization rising to 26/38 = 68%.

The results confirmed the expected behavior. The 12s gap was partially closed, but not eliminated — synthesis remained the bottleneck. The theoretical maximum throughput of 2.31 proofs/min (one proof per 26s GPU cycle) remained out of reach because synthesis simply couldn't keep up.

The assistant's reflection on this result was characteristically clear-eyed: "Further GPU utilization gains would require either reducing synthesis time (already near-minimum with PCE) or running multiple synthesis tasks in parallel — a significant engine architecture change." This honest assessment of the diminishing returns of further optimization was a hallmark of the session's disciplined approach.

Memory: The Partitioned Path's True Value Proposition

The e2e benchmarks also revealed a troubling discrepancy in memory measurements. The in-process benchmarks had shown the partitioned path achieving 71 GiB peak RSS, but the daemon benchmarks showed ~265 GiB for all partitioned modes and 370 GiB for the standard path. The assistant noted in [msg 1801]: "Peak RSS of ~265 GiB for partitioned modes is way too high — the in-process bench showed 71 GiB. Something in the daemon is not releasing memory between proofs, or the -j 2 is causing two proofs' worth of data to be live."

This discrepancy highlighted an important lesson: in-process benchmarks and daemon benchmarks measure different things. The daemon's gRPC layer, scheduler queue, and concurrent request handling all contribute to memory pressure that doesn't appear in isolated in-process tests. The partitioned path's memory advantage — while real — is less dramatic in the daemon context than the in-process numbers suggested.

Nevertheless, the fundamental trade-off remained clear. The standard path offered superior throughput (47.7s vs 72s per proof) at the cost of higher peak memory (370 GiB vs ~265 GiB). For memory-constrained deployments — machines with, say, 256 GiB of RAM — the partitioned path's lower peak memory could be the difference between running and not running at all. For throughput-optimized deployments with abundant memory, the standard path was the clear winner.

This reframing was the session's most important conceptual contribution. The partitioned pipeline had been designed and evaluated primarily as a throughput optimization. The in-process benchmarks had shown it as a clear win: 3.2× memory reduction with only 16% latency overhead. But the daemon benchmarks revealed that the latency overhead was actually 51% in the multi-proof scenario, because the partitioned path forfeited the inter-proof overlap that the standard path naturally exploited.

The partitioned path was not wrong; it was solving a real problem (memory) that the standard path did not address. The insight was that throughput and memory are different axes, and optimizing for one can harm the other. The partitioned path's 3.2× memory reduction remained valuable — just not for throughput. And the standard path's inter-proof overlap, once recognized, became a feature to preserve and enhance rather than a baseline to surpass.

The Strategic Reorientation

Perhaps the most important outcome of this segment was the strategic reorientation it triggered. The partitioned pipeline had been the focus of Phase 6 development, consuming significant engineering effort across multiple sub-sessions. The benchmark results could have been demoralizing — a sophisticated optimization that turned out to be a regression in the primary metric (throughput).

Instead, the assistant treated the data as information, not judgment. The partitioned path was not abandoned; its value proposition was clarified. It became a tool for memory-constrained deployments rather than a universal improvement. The standard path was recognized as already having a form of pipeline — just one that operates at a different level (inter-proof rather than intra-proof).

This reorientation is visible in the assistant's reflection at the end of [msg 1801]: "But actually, there's a simpler insight: the standard path (slot_size=0) already achieves better throughput. The partitioned path's value is memory reduction for memory-constrained machines. For throughput, the standard path wins."

This is the voice of an engineer who has learned something fundamental about the system they are optimizing. The partitioned pipeline was designed to solve a problem (memory) but was evaluated on a metric (throughput) where the existing solution was already superior. The clarification of which metric each path optimizes is the segment's most lasting contribution.

The assistant also identified a potential architectural fix at [msg 1803]: "The fix is to make the partitioned path's GPU work go through the engine-level GPU channel, so the synthesis task can start on the next proof while the current one's GPU work is happening." This would involve splitting the partitioned path into two phases — a synthesis phase that runs in the synthesis task (producing individual partition proofs) and a GPU phase that sends those proofs through the engine's GPU channel. The synthesis task would then be free to start on the next proof while the GPU works on the current one's partitions.

However, the assistant also recognized the complexity of this approach. The engine GPU channel expects SynthesizedJob objects containing the full SynthesizedProof plus SuprasealParameters, and the GPU worker handles proof assembly and completion. Restructuring the partitioned path to produce individual partition jobs and send them through this channel would require significant refactoring of the engine's internals.

The Architecture of the Standard Pipeline

To fully appreciate the standard path's advantage, it's worth understanding the engine's pipeline architecture in detail. The assistant's exploration at [msg 1803]–[msg 1805] revealed the key components:

The Synthesis Task Loop: A Tokio spawn_blocking task that runs in a dedicated OS thread. It polls the scheduler for ready batches, synthesizes them (constraint evaluation + witness generation), and produces SynthesizedJob objects.
The GPU Channel: A bounded sync_channel that connects the synthesis task to the GPU worker pool. The channel capacity is controlled by synthesis_lookahead — with lookahead=1, the channel can hold at most one pre-synthesized proof.
The GPU Worker Pool: A set of worker threads (one per GPU device) that pick up SynthesizedJob objects from the channel and execute the GPU proving pipeline (MSM, FFT, etc.).
The Scheduler: A Tokio-based async component that receives proof requests via gRPC, queues them by type, and dispatches them to the synthesis task in batches. The beauty of this architecture is that the synthesis task and the GPU workers are decoupled by the channel. The synthesis task can run ahead of the GPU, building up a backlog of pre-synthesized proofs. When the GPU finishes one proof, it immediately picks up the next from the channel. This is classic producer-consumer pipelining, and it works beautifully when synthesis time exceeds GPU time — the producer is the bottleneck, but the consumer never stalls. The partitioned path, by running both synthesis and GPU inside a single spawn_blocking call, collapses this decoupling. The synthesis task becomes blocked on GPU work, and the producer-consumer pipeline becomes sequential. The elegant intra-proof pipelining of the partitioned path (parallel partition synthesis → sequential partition GPU proving) is defeated by the loss of inter-proof pipelining.

Lessons in Systems Engineering

This segment of the opencode session offers several enduring lessons for systems engineering:

Lesson 1: End-to-end testing reveals what component testing cannot. The in-process benchmarks showed the partitioned path as a clear win (3.2× memory reduction, 16% latency overhead). Only daemon-integrated testing exposed the inter-proof overlap advantage of the standard path, which widened the gap from 16% to 51%. Optimizations must be validated at the system level, not just the component level.

Lesson 2: The architecture you have is often better than the one you're building. The standard engine pipeline, which had been treated as the baseline to beat, already achieved near-optimal GPU utilization through its two-stage synthesis→GPU architecture. The partitioned path's elegant intra-proof pipelining was defeated by the simple fact that it blocked the synthesis task, preventing inter-proof overlap.

Lesson 3: Memory and throughput are different optimization axes. The partitioned path's 3.2× memory reduction remains valuable — just not for throughput. Recognizing which metric each path optimizes is the key to making correct deployment decisions. A deployment with 256 GiB of RAM might choose the partitioned path; a deployment with 512 GiB would choose the standard path.

Lesson 4: GPU fixed costs are everywhere. The 12s GPU idle gap — caused by synthesis time exceeding GPU time — is a structural bottleneck that no amount of concurrency tuning can eliminate. Further gains would require either reducing synthesis time (already near-minimum with PCE) or running multiple synthesis tasks in parallel — a significant engine architecture change.

Lesson 5: Measure twice, optimize once. The entire Phase 6 effort was predicated on in-process benchmarks that did not capture the daemon's inter-proof overlap behavior. A single end-to-end benchmark at the start of Phase 6 would have revealed the standard path's advantage and potentially redirected the optimization effort toward different goals. This is not to say the partitioned path was wasted effort — the memory reduction is real and valuable — but the throughput expectations were misplaced.

The Path Forward

The segment concluded with a clear understanding of the system's true nature. The standard pipeline path was the throughput champion, achieving 1.3 proofs/min with 57% GPU utilization. The partitioned path was the memory champion, reducing peak RSS from 370 GiB to ~265 GiB (and potentially to 71 GiB with proper memory management). The synthesis_lookahead=2 experiment showed that the 12s GPU idle gap could be partially closed, but synthesis remained the fundamental bottleneck.

The assistant's final analysis at [msg 1811] laid out the options for increasing GPU utilization:

Reduce synthesis time — already near-minimum with PCE at ~38s. Further gains would require algorithmic innovations or hardware improvements.
Run multiple synthesis tasks in parallel — a significant engine architecture change that would allow the GPU to be fed by multiple synthesis producers. This would require careful coordination to avoid memory blowup.
Increase synthesis_lookahead to 2 — would eliminate the 12s gap, raising GPU utilization to 68% and throughput to 1.58 proofs/min. This was tested and confirmed to work, but at the cost of increased memory footprint (two full synthesized proofs in memory). Option 2 — parallel synthesis — was identified as the most promising path for significant throughput gains, but also the most complex. It would require restructuring the engine's task loop to support multiple concurrent synthesis operations, potentially using a thread pool rather than a single spawn_blocking task. This was recognized as a Phase 7 or later effort.

Conclusion: The Value of Honest Measurement

Segment 20 of the opencode session is a masterclass in the discipline of systems engineering. The assistant approached the end-to-end benchmarks with the same rigor and curiosity that had characterized all the preceding optimization work. When the data contradicted expectations, the assistant did not rationalize or dismiss — it analyzed, understood, and reframed.

The partitioned pipeline was not a failure. It was a solution to a real problem (memory pressure) that had been evaluated on the wrong metric (throughput). The end-to-end benchmarks revealed its true value proposition, and the strategic reorientation that followed was a direct consequence of honest measurement.

The standard pipeline, meanwhile, was revealed to be more sophisticated than the team had given it credit for. Its two-stage synthesis→GPU architecture, with the decoupling channel and lookahead mechanism, was already achieving near-optimal GPU utilization. The partitioned path's attempt to improve upon it through intra-proof pipelining inadvertently destroyed the inter-proof pipelining that made the standard path work.

This is the kind of learning that only comes from end-to-end measurement. Component-level benchmarks can validate individual optimizations, but they cannot capture the emergent behaviors of a complex system under realistic load. The daemon benchmarks — with their gRPC layer, scheduler queue, concurrent request handling, and real GPU hardware — revealed truths that no amount of in-process testing could have uncovered.

The cuzk proving engine emerged from this segment not weaker, but stronger — because the team now understands its true nature. The standard path is the throughput champion. The partitioned path is the memory champion. And the path forward is clear: preserve both, deploy according to resource constraints, and consider parallel synthesis as the next frontier for throughput improvement.

In the end, this segment is not about failure. It is about learning. And the willingness to learn — to let the data speak, to reframe assumptions, to find value where it actually exists rather than where it was expected — is what distinguishes great engineering from merely good engineering.## References

[1] Chunk article: "The Great Unwinding: How End-to-End Testing Revealed the True Value of a Pipelined Proving Architecture" — the companion article covering the same segment's events from a different analytical angle.