The Benchmark That Validated Phase 8: Dual-Worker GPU Interlock in the cuzk SNARK Proving Engine

In the high-stakes world of Filecoin proof generation, every second counts. The cuzk SNARK proving engine, responsible for generating Groth16 proofs for Filecoin's Proof-of-Replication (PoRep) protocol, had been through a grueling optimization journey spanning multiple phases. By the time the team reached Phase 8, they had already implemented a slotted partition pipeline (Phase 6), a per-partition dispatch architecture (Phase 7), and diagnosed a critical GPU utilization gap caused by static mutex contention in the C++ CUDA kernel code. Message 2231 represents the pivotal moment when all that work was put to the test: the first multi-proof throughput benchmark of the Phase 8 dual-worker GPU interlock.

The Message

The message itself is deceptively simple — a single bash command and its output:

[bash] cd /home/theuser/curio/extern/cuzk && time ./target/release/cuzk-bench -a "http://127.0.0.1:9820" batch -t porep --c1 /data/32gbench/c1.json -c 5 -j 3 2>&1
2026-02-19T01:01:41.130801Z  INFO cuzk_bench: loading C1 output path=/data/32gbench/c1.json
=== Batch Benchmark ===
proof type:  porep
count:       5
concurrency: 3

  [1/5] COMPLETED — 109.5s (prove=65695 ms, queue=247 ms)
  [2/5] COMPLETED — 106.1s (prove=77578 ms, queue=716 ms)
  [3/5] COMPLETED — 155.0s (prove=79474 ms, queue=1177 ms)
  [4/5] COMPLETED — 110.5s (prove=70520 ms, queue=267 ms)
  [5/5] COMPLETED — 110.5s (prove=64966 ms...

This is a batch benchmark: five PoRep proofs, run with concurrency 3 (meaning up to three proofs are processed simultaneously). The output is truncated — we see individual proof completion times but not the final summary line. However, from the subsequent message ([msg 2232]), we learn the critical result: 44.0 seconds per proof, a 13.2% improvement over Phase 7's 50.7 seconds.

The Context: Why Phase 8 Existed

To understand the significance of this message, we must trace the problem that led to Phase 8. The Phase 7 per-partition dispatch architecture had already improved throughput by eliminating the "slot" abstraction and dispatching individual partitions directly to the GPU. However, benchmarking revealed a persistent GPU utilization gap. The root cause was a static C++ mutex in the generate_groth16_proofs_c function within the CUDA kernel code. This mutex, intended to protect GPU state during CUDA kernel launches, was locking for the entire duration of the C++ function — including CPU preprocessing steps like data marshaling, host-to-device transfers, and the b_g2_msm computation. During these CPU-side operations, the GPU sat idle while the mutex was held, preventing any other worker from launching CUDA kernels.

The Phase 8 insight was elegantly simple: narrow the mutex scope to cover only the CUDA kernel region (NTT+MSM, batch additions, tail MSMs), allowing CPU preprocessing and b_g2_msm to run outside the lock. This enabled a dual-worker architecture where two GPU workers per device could interleave their work — one performing CPU preprocessing while the other ran CUDA kernels on the GPU.

The implementation spanned seven files and roughly 195 lines of changes. The C++ CUDA kernel was refactored to accept a passed-in mutex pointer with narrowed scope. FFI plumbing threaded the mutex through supraseal-c2 → bellperson → pipeline.rs. The engine was modified to spawn gpu_workers_per_device workers (default 2) sharing a per-GPU C++ mutex allocated via new create_gpu_mutex/destroy_gpu_mutex helpers. The build process alone required multiple iterations to resolve Send trait issues with raw pointer passing across async boundaries — a classic Rust concurrency challenge where *mut c_void pointers needed to be cast to usize to satisfy the compiler's safety checks.## The Reasoning Behind This Specific Benchmark

Why this particular benchmark configuration — 5 proofs with concurrency 3 (-c 5 -j 3)? This was not an arbitrary choice. The c=5 j=3 configuration had been used consistently throughout the optimization journey as the standard multi-proof throughput test. It represents a realistic workload: five proofs is enough to measure steady-state throughput beyond cold-start effects, and concurrency 3 keeps multiple proofs in flight to stress-test the pipeline's ability to overlap work across proofs. Using the same configuration as Phase 7 allowed a direct apples-to-apples comparison.

The assistant's reasoning, visible in the surrounding messages, reveals a methodical approach. Before running this multi-proof benchmark, the assistant had already validated the Phase 8 implementation with a single-proof test ([msg 2226]) that achieved 100.0% GPU efficiency — zero idle gaps between partitions. That single-proof result was the prerequisite; if the dual-worker interlock couldn't saturate the GPU for a single proof, there was no point testing throughput. With that validation complete, the multi-proof benchmark was the real test: could the dual-worker architecture sustain its advantage under load, or would new bottlenecks emerge?

The assistant also ran a comparison with concurrency 2 (j=2) in the subsequent message ([msg 2232]), yielding 49.5 seconds per proof — a 17.2% improvement over Phase 7's 59.8 seconds at the same concurrency. This dual-benchmark strategy reveals a deliberate attempt to understand the interaction between the dual-worker interlock and the existing concurrency mechanism. The larger improvement at j=2 (17.2% vs 13.2%) suggests that the dual-worker interlock is particularly effective when concurrency is lower, because the GPU has more idle cycles to reclaim.

Assumptions and Knowledge Required

Understanding this message requires significant domain knowledge. The reader must know that Filecoin PoRep proofs are generated in a pipeline: C1 output (the result of a previous proving phase) is consumed by C2 (the Groth16 proof generation), which involves both CPU-intensive synthesis (constraint system evaluation) and GPU-intensive operations (NTT, MSM, and other elliptic curve operations). The partition_workers=20 configuration tells us the system is using 20 CPU threads for synthesis, each producing a partition of the constraint system. The GPU then processes these partitions, performing the heavy cryptographic math.

The benchmark also assumes a specific hardware environment: the daemon logs and earlier messages reveal a machine with 96 CPU cores and at least one NVIDIA GPU with CUDA support. The SRS (Structured Reference String) parameters are preloaded from /data/zk/params, and the C1 input is stored at /data/32gbench/c1.json. These paths and configurations are specific to this test environment.

A key assumption embedded in the Phase 8 design is that the GPU is the bottleneck, not the CPU. The dual-worker interlock only helps if GPU compute is the scarce resource — if CPU synthesis were the bottleneck, adding more GPU workers would just create more contention. The benchmark results validate this assumption: the 13.2% improvement confirms that GPU utilization was indeed the limiting factor.

What the Output Reveals

The individual proof timings tell a nuanced story. Proof 3 took 155.0 seconds — significantly longer than the others (which cluster around 106-110 seconds). This is the "first proof" effect: proof 3 was likely the first to be submitted, and its long queue time (1177 ms vs ~250 ms for others) suggests it waited for SRS preloading or other initialization. The prove time of 79,474 ms for proof 3 is also higher than the others, possibly because it includes the cost of warming up GPU state.

The other four proofs show prove times ranging from 64,966 ms to 77,578 ms. This variability is expected in a concurrent system — proofs compete for GPU time, and the dual-worker interlock creates complex scheduling dynamics. The key metric is the average throughput: 44.0 seconds per proof, calculated from the total wall time divided by five proofs.

The Thinking Process

The assistant's thinking, visible in the surrounding messages, reveals a careful analytical process. After the single-proof test, the assistant immediately noticed that per-partition GPU time had increased from ~3.3-4.0 seconds (Phase 7) to ~6.4-6.7 seconds (Phase 8). Rather than panicking, the assistant correctly diagnosed this as a measurement artifact: the gpu_prove function now measured the entire C++ function (including CPU preprocessing), not just the CUDA kernel region. The actual CUDA kernel time remained ~3.3 seconds per partition. The extra ~3.2 seconds was CPU preprocessing that was now correctly overlapped with the other worker's CUDA work — precisely the goal of Phase 8.

This diagnostic reasoning — distinguishing between a real regression and a measurement artifact — is characteristic of performance engineering. The assistant didn't just run benchmarks; it interpreted them in the context of the implementation changes.

The Significance

Message 2231 represents the empirical validation of an optimization hypothesis. The Phase 8 dual-worker GPU interlock was a surgical change: narrow a mutex, spawn an extra worker, and let the GPU stay busy. The 13.2% throughput improvement (from 50.7s to 44.0s per proof) is substantial for a production proving system where every second of GPU time costs money.

But the deeper significance lies in what this benchmark enabled next. The subsequent partition_workers sweep (Chunk 1 of Segment 24) systematically tested pw values from 10 to 20, finding that pw=10-12 produced even better results (43.5s/proof). This sweep was only meaningful because Phase 8 had established a new baseline — without the dual-worker interlock, the optimal partition_workers setting might have been different.

The message also demonstrates a crucial principle of systems optimization: the best optimizations are those that change the bottleneck. Phase 8 didn't just make the code faster in isolation; it shifted the bottleneck from GPU idle time to CPU contention, opening new avenues for improvement. The partition_workers sweep that followed was a direct response to this shifted bottleneck.

In the end, this single bash command — a batch benchmark run at 1:01 AM on February 19, 2026 — captured the moment when months of optimization work crystallized into a measurable result. The 44.0 seconds per proof wasn't just a number; it was proof that the dual-worker interlock worked, that the mutex narrowing was correct, and that the cuzk proving engine was now 13.2% closer to production efficiency.