The Benchmark That Revealed the GPU Gap: Analyzing Phase 7 Throughput in the cuzk SNARK Proving Engine

Introduction

In the iterative optimization of a high-performance SNARK proving engine, few moments are as revealing as the first multi-proof throughput benchmark. On February 18, 2026, the assistant running the cuzk SNARK proving engine for Filecoin PoRep executed precisely such a benchmark. Message <msg id=2110> captures the raw output of a 5-proof throughput test at concurrency 3, running on the freshly implemented Phase 7 per-partition dispatch architecture. This single bash command and its output would become the catalyst for the next major optimization phase, exposing a critical GPU utilization gap and driving the design of a dual-GPU-worker interlock strategy.

The Message

The assistant executed the following command and captured its output:

cd /home/theuser/curio/extern/cuzk && ./target/release/cuzk-bench batch \
  --type porep --c1 /data/32gbench/c1.json -c 5 -j 3 2>&1

The benchmark tool loaded the C1 output from disk and began dispatching five PoRep proofs through the daemon with a concurrency of three:

=== Batch Benchmark ===
proof type:  porep
count:       5
concurrency: 3

  [1/5] COMPLETED — 121.0s (prove=43464 ms, queue=249 ms)
  [2/5] COMPLETED — 125.3s (prove=44817 ms, queue=691 ms)
  [3/5] COMPLETED — 177.2s (prove=50862 ms, queue=1148 ms)
  [4/5] COMPLETED — 114.5s (prove=38848 ms, queue=322 ms)
  [5/5] COMPLETED — 128.1s (prove=36461 ms, queue=...)

The summary line, referenced in the subsequent message <msg id=2111>, reported approximately 50.7 seconds per proof wall-clock throughput — a meaningful improvement over the single-proof cold-start latency of 72.8 seconds measured in <msg id=2106>, but far short of the theoretical ~30 seconds per proof that the Phase 7 design had predicted.

Context and Motivation: Why This Message Was Written

This message sits at a critical juncture in a months-long optimization campaign targeting the SUPRASEAL_C2 Groth16 proof generation pipeline for Filecoin's Proof-of-Replication (PoRep). The broader project had already progressed through six major phases:

• Phase 1–5: Established the baseline pipeline, identified nine structural bottlenecks, and characterized the ~200 GiB peak memory footprint.
• Phase 6: Implemented slotted partition proving, reducing peak memory from 228 GiB to 71 GiB but achieving only 47.7s per proof (versus the standard pipeline's 72s per proof for single-sector work).
• Phase 7: Fundamentally rearchitected the engine to treat each of the 10 PoRep partitions as an independent work unit flowing through the synthesis→GPU pipeline, enabling cross-sector pipelining. The Phase 7 architecture, committed as f5bfb669 on the feat/cuzk branch, replaced the old model where all 10 partitions of a single proof were synthesized together and then GPU-proved together. Instead, each partition became an independent job: synthesized individually by a pool of 20 spawn_blocking workers gated by a semaphore, then GPU-proved with num_circuits=1 (achieving the predicted ~0.4s b_g2_msm per partition), and finally assembled into a complete proof by a ProofAssembler. The single-proof test in <msg id=2106> had validated correctness: 72.8 seconds total, with 38.8 seconds of GPU time spread across 10 partitions averaging ~3.9 seconds each. But the true test of Phase 7 was multi-proof throughput. The architecture's central promise was that while one proof's partitions occupied the GPU, another proof's partitions could be synthesized concurrently on CPU cores — eliminating the "thundering herd" pattern where all synthesis happened before any GPU work began. The user's instruction was simple: "Do some test runs!" (<msg id=2089>). The assistant responded by methodically setting up a Phase 7 configuration, starting the daemon, running a single-proof validation, and then — in this message — executing the throughput benchmark that would reveal the architecture's real-world behavior.

Technical Analysis: What the Numbers Reveal

The five individual proof times tell a nuanced story. Proofs 1 and 2 completed in 121.0s and 125.3s respectively, with prove (GPU) times of 43.5s and 44.8s. These are the "warm" proofs, where the pipeline is still stabilizing. Proof 3 spiked dramatically to 177.2s with a 50.9s prove time and a queue wait of 1.15 seconds — the longest of the batch. Proofs 4 and 5 then recovered to 114.5s and 128.1s, with prove times dropping to 38.8s and 36.4s.

The high variance is the first red flag. The 177.2s outlier suggests that proof 3 encountered significant congestion: its partitions likely arrived at the GPU worker while multiple other proofs' partitions were already queued, creating a backlog. The queue times tell a parallel story: proof 1 waited only 249ms, proof 2 waited 691ms, proof 3 waited 1,148ms, and then proof 4 dropped back to 322ms. This sawtooth pattern is characteristic of a system where the GPU worker processes partitions sequentially while synthesis workers produce them in bursts.

More telling is the relationship between total time and prove time. For proof 1: 121.0s total minus 43.5s prove = 77.5s of non-GPU time (synthesis + queue + assembly). For proof 3: 177.2s total minus 50.9s prove = 126.3s of non-GPU time. This asymmetry reveals that synthesis is not overlapping cleanly with GPU work — when multiple proofs compete for synthesis threads, the CPU becomes a bottleneck, and partitions arrive at the GPU in irregular bursts rather than a steady stream.

Assumptions and Their Validity

The Phase 7 design rested on several key assumptions, and this benchmark tested them against reality:

Assumption 1: The GPU is the bottleneck. The design assumed that keeping the GPU busy was the primary goal, and that CPU-side synthesis could be effectively overlapped. The benchmark results partially validate this — the GPU is indeed busy, but not continuously so. The subsequent analysis in <msg id=2115> would quantify GPU efficiency at only 64.3%, with 251.9 seconds of total gap time across 109 inter-partition intervals versus 453.1 seconds of GPU wall time.

Assumption 2: Concurrency 3 is appropriate. The choice of -j 3 was reasonable given the single GPU, but the results suggest that concurrency management needs finer tuning. The 177.2s outlier indicates that three concurrent proofs can create enough synthesis pressure to cause queuing delays.

Assumption 3: Per-partition overhead is negligible. The Phase 7 design predicted that individual partition GPU calls would be fast (~3.5s each) and that the overhead between them would be small. The benchmark confirmed the per-partition GPU time (averaging ~4.1s), but the inter-partition gaps — ranging from a few milliseconds to over 125 seconds — proved to be the critical weakness.

Assumption 4: The pipeline flows smoothly in steady state. The design document predicted "~100% GPU utilization, zero cross-sector GPU idle gaps." The reality was far messier. The 64.3% GPU utilization and the presence of multiple multi-second gaps between GPU calls revealed that the pipeline was flowing, but with significant stutters.

The Critical Mistake: Static Mutex Contention

The most important incorrect assumption was hidden in the implementation details. As the assistant would discover in the subsequent investigation (<msg id=2113> through <msg id=2120>), the static std::mutex in generate_groth16_proofs_c held its lock for the entire ~3.5s GPU function call, even though only ~2.1s was actual CUDA kernel execution. The remaining ~1.3s was CPU-side work — proof serialization, b_g2_msm computation, and memory management — that could theoretically overlap with another partition's GPU time.

This single design flaw explained the 64.3% GPU efficiency. The GPU worker was spending over a third of its time doing CPU work while holding a mutex that prevented any other worker from submitting CUDA work. The result was a GPU that sat idle while the CPU crunched numbers, then burst into activity when the next partition finally reached the CUDA kernels.

Input Knowledge Required

To fully understand this message, one needs:

1. The Phase 7 architecture: Knowledge that each of the 10 PoRep partitions is now an independent work unit, synthesized by a pool of 20 workers and GPU-proved individually with num_circuits=1.
2. The benchmark tool: Understanding that -c 5 requests 5 proofs, -j 3 limits concurrent requests to 3, --type porep selects PoRep proof type, and --c1 points to the pre-computed circuit input.
3. The baseline performance: The single-proof latency of 72.8s (38.8s GPU) from <msg id=2106> provides the reference point for judging throughput improvement.
4. The hardware context: The system has a single GPU (NVIDIA, likely an A100 or similar datacenter card based on the memory and compute requirements), 192 rayon threads configured, and sufficient CPU cores to run 20 synthesis workers.
5. The optimization history: Understanding that Phase 6 achieved 47.7s/proof with 71 GiB memory, and that Phase 7 was designed to improve on this by enabling cross-sector pipelining.

Output Knowledge Created

This message produced several critical pieces of knowledge:

1. Throughput baseline for Phase 7: ~50.7s/proof at concurrency 3, representing a ~30% improvement over single-proof latency but still well below the theoretical maximum.
2. Variance characterization: Proof times range from 114.5s to 177.2s (a 55% spread), indicating systemic instability in the pipeline flow.
3. Queue time pattern: The increasing queue times (249ms → 1,148ms) reveal that the system backs up under sustained load.
4. GPU vs. CPU time ratio: The prove times (36.4s–50.9s) versus total times (114.5s–177.2s) show that GPU execution accounts for only 28–38% of wall-clock time, with the remainder consumed by synthesis, queueing, and assembly.
5. Evidence for Phase 8: The variance and GPU efficiency data directly motivated the design of the dual-GPU-worker interlock (Phase 8), documented in <chunk seg=23 chunk=1> as c2-optimization-proposal-8.md.

The Thinking Process

The assistant's reasoning is visible in the sequence of actions leading to this message. After committing Phase 7, the assistant:

1. Checked available test configurations (<msg id=2090>) to understand what benchmark setups already existed.
2. Verified the test data and binaries (<msg id=2091>) to ensure the C1 input and daemon binary were ready.
3. Rebuilt the bench binary (<msg id=2092>) since it predated the Phase 7 changes.
4. Created a Phase 7-specific config (<msg id=2096>) with partition_workers=20.
5. Started the daemon (<msg id=2103>) and waited for SRS preloading to complete.
6. Ran a single-proof validation (<msg id=2106>) to confirm correctness before scaling up.
7. Analyzed the single-proof timeline (<msg id=2107> through <msg id=2109>) to verify that all 10 partitions were flowing through the pipeline correctly. Only after this careful validation did the assistant run the throughput benchmark in <msg id=2110>. The methodical approach — validate correctness first, then measure throughput — reflects a disciplined engineering mindset. The assistant knew that a broken pipeline would produce meaningless throughput numbers, so it verified the basic mechanism before stressing the system. The choice of -c 5 -j 3 was deliberate. Five proofs provide enough samples to observe variance and warm-up effects. Concurrency 3 is high enough to stress the pipeline without overwhelming the single GPU. The assistant likely expected to see steady ~30s/proof throughput based on the design predictions; the actual 50.7s/proof was a clear signal that something was amiss.

From Benchmark to Breakthrough

The true significance of <msg id=2110> lies not in the numbers themselves but in what they triggered. The user's observation in <msg id=2112> — "looks like gpu use is pretty jumpy, generally high, but maybe there is some per-job overhead that could be cut?" — showed that the benchmark results were immediately interpretable to an experienced eye. The jumpy GPU utilization was the visible symptom of the underlying mutex contention problem.

The assistant responded by diving deep into the timeline data (<msg id=2113>), computing precise inter-partition gaps (<msg id=2114>), and ultimately quantifying GPU efficiency at 64.3% (<msg id=2115>). This analysis identified the eight gaps exceeding 500ms — including a 125.9s gap between the first proof's completion and the second proof's first GPU call — as the primary targets for optimization.

The root cause investigation (<msg id=2117>) traced the problem through the call stack from Rust's prove_from_assignments through the FFI boundary into C++ CUDA code, pinpointing the static std::mutex in generate_groth16_proofs_c. This diagnosis directly motivated the Phase 8 design: a dual-GPU-worker interlock where two workers share a fine-grained mutex that brackets only the CUDA kernel region, allowing one worker's CPU preamble and epilogue to execute concurrently with the other worker's GPU kernels.

The Phase 8 proposal, committed as 71f97bc7, promised to boost GPU efficiency from ~64% to ~98%, yielding a 3–10% throughput improvement. This entire optimization chain — from benchmark to diagnosis to design — was triggered by the five numbers printed in <msg id=2110>.

Conclusion

Message <msg id=2110> is a masterclass in the value of measurement-driven optimization. A single benchmark run, carefully designed and executed, revealed that the elegant Phase 7 architecture was being undermined by a subtle locking issue in the C++ CUDA layer. The 50.7s/proof throughput and 64.3% GPU efficiency were not failures — they were data points that pointed directly to the next bottleneck.

The message exemplifies the iterative engineering approach that characterizes the entire cuzk optimization project: design an architecture, implement it, benchmark it, analyze the results, identify the next bottleneck, and design the next phase. Each benchmark produces not just performance numbers but diagnostic insights that drive the next cycle of improvement. The Phase 7 throughput test was a success precisely because it revealed the Phase 8 problem — and in doing so, brought the proving engine one step closer to its goal of continuous, memory-efficient, GPU-saturated proof generation.