The Benchmark That Broke Phase 10: How a Failed GPU Interlock Design Revealed Fundamental CUDA Constraints

Introduction

In the high-stakes world of Filecoin proof generation, every second counts. The cuzk SNARK proving engine, a custom Groth16 implementation for Filecoin's Proof-of-Replication (PoRep), had been through nine optimization phases—each squeezing more throughput from a single NVIDIA GPU. Phase 8 introduced a dual-worker GPU interlock that improved throughput by 13–17%. Phase 9 optimized PCIe transfers for another 14.2% gain. But Phase 10, an ambitious two-lock GPU interlock design, was about to crash head-first into fundamental CUDA device-global synchronization constraints. The message that revealed this failure is a single benchmark command and its output, but it tells a story of architectural assumptions colliding with hardware reality.

The Message: A Benchmark Run

The message at index 2659 is deceptively simple. The assistant, having just confirmed the daemon is ready, executes a concurrent benchmark:

Daemon is ready. Now run c=3 j=3:

>

`` FIL_PROOFS_PARAMETER_CACHE=/data/zk/params /home/theuser/curio/extern/cuzk/target/release/cuzk-bench batch --type porep --c1 /data/32gbench/c1.json --count 3 --concurrency 3 2>&1 | tee /home/theuser/cuzk-p10-c3j3-bench.log ``

>

The output tells a grim story:

>

``` === Batch Benchmark === proof type: porep count: 3 concurrency: 3

>

[1/3] FAILED — 41.5s (prove=0 ms, queue=0 ms) [2/3] COMPLETED — 116.1s (prove=125737 ms, queue=700 ms) [3/3] COMPLETED — 154.2s (prove=118346 ms, queue=1133 ms)

>

=== Batch Summary === total time: 154.3s completed: 2 failed: 1 wall time: avg=135.2s min=116.1s max=154.2s prove time: avg=122.0s min=118.3s max=125.7s throughput: 1.17 proofs/min (51.4s/proof) ```

One proof failed outright. The two that completed took 116 and 154 seconds respectively—far worse than the Phase 9 baseline, which achieved ~32 seconds for a single proof and ~41 seconds at high concurrency. The throughput of 1.17 proofs per minute was abysmal compared to expectations.

The Context: Phase 10's Ambitious Design

To understand why this message matters, we must understand what Phase 10 attempted to do. The cuzk proving engine generates Groth16 proofs in a pipelined fashion: CPU-based synthesis produces circuit partitions, which are then dispatched to GPU workers for the MSM (Multi-Scalar Multiplication) and NTT (Number Theoretic Transform) operations that dominate proof time.

Phase 8 had introduced a dual-worker interlock: two GPU workers per device, each holding a mutex to serialize access to the GPU. This improved throughput by allowing one worker to prepare data while the other held the GPU, but it left CPU-side overhead from mutex contention.

Phase 9 optimized PCIe transfers by pre-staging NTT uploads, reducing GPU idle time. This achieved 14.2% improvement in single-worker mode, but dual-worker mode revealed PCIe bandwidth contention.

Phase 10 attempted to solve the remaining contention by splitting the GPU lock into two separate locks: a compute_mtx for GPU kernel execution and a separate lock for memory management operations. The idea was that while one worker was running GPU kernels, another could be freeing memory or preparing buffers—overlapping operations that previously serialized.

What the Benchmark Revealed

The benchmark results were devastating, and they revealed multiple problems simultaneously:

First, a proof failure. The first of three proofs failed entirely with prove=0 ms. This indicated a crash or fatal error in the GPU pipeline, not a timeout. The two-lock design had introduced a race condition or resource conflict severe enough to kill a proof.

Second, catastrophic throughput regression. At 51.4 seconds per proof, Phase 10 was 25% slower than the Phase 9 baseline at similar concurrency (~41 seconds). The two-lock design, intended to improve throughput, had made things dramatically worse.

Third, extreme wall-time variance. The three proofs completed at 41.5s (failed), 116.1s, and 154.2s. The 38-second gap between the two successful proofs suggested severe serialization—workers were blocking each other far more than in the single-lock design.

The Root Cause: Device-Global Synchronization

The assistant's subsequent investigation (visible in later messages) revealed the fundamental flaw. CUDA memory management APIs like cudaDeviceSynchronize and cudaMemPoolTrimTo are device-global operations. They affect the entire GPU, not just the calling thread or stream. When the two-lock design allowed one worker to call cudaMemPoolTrimTo while another held the compute lock, the trim operation would stall the active GPU kernels, defeating the entire purpose of the lock split.

Worse, the 16 GB VRAM on the target GPU could not accommodate pre-staged buffers from multiple workers simultaneously. When Worker A tried to pre-stage its NTT buffers while Worker B was running kernels, the allocation would fail (or worse, trigger OOM), forcing the expensive fallback path of allocating inside the compute lock—exactly what the two-lock design was supposed to avoid.

The prestage_setup=skip_vram fallback seen in earlier single-proof tests (msg 2645) was the canary in the coal mine: every partition was falling back to the slow path because VRAM was exhausted by competing workers.

Assumptions That Failed

Phase 10 rested on several assumptions that proved incorrect:

Assumption 1: CUDA memory operations are per-context, not device-global. The design assumed that cudaMemPoolTrimTo and similar APIs could be called concurrently with kernel execution on the same device. In reality, these operations synchronize the entire device, serializing what was meant to be parallel.

Assumption 2: VRAM is sufficient for multi-worker pre-staging. The design assumed that 16 GB VRAM could hold pre-staged buffers for multiple workers simultaneously. With each worker needing ~6-8 GB for its partition data, two workers' pre-staged buffers plus the active worker's working set would exceed 16 GB.

Assumption 3: Lock splitting naturally improves throughput. The assumption that finer-grained locking always improves performance ignored the reality that GPU resources are fundamentally single-device resources. The GPU is a shared, serial resource at the device level—no amount of lock splitting changes that.

Assumption 4: The single-proof test (c=1 j=1) was representative. The earlier single-proof test showed 73.8 seconds, which was already slow, but the assistant attributed this to 3 workers competing for 1 proof's 10 partitions. The assumption was that concurrent proofs would overlap better. The c=3 j=3 test proved this assumption wrong.

Input Knowledge Required

To fully understand this message, one needs:

• Knowledge of Groth16 proof generation: Understanding that proof generation involves CPU synthesis followed by GPU-heavy MSM/NTT operations, and that these can be pipelined across partitions.
• Understanding of CUDA device model: Knowing that CUDA devices have global synchronization primitives that affect all threads, not just the calling context.
• Familiarity with the cuzk architecture: The pipeline has synthesis workers (CPU) producing partitions, which are consumed by GPU workers. The partition_workers setting (10 in this config) controls how many synthesis threads run concurrently.
• Knowledge of prior optimization phases: Phase 8's dual-worker interlock and Phase 9's PCIe optimization established the baseline that Phase 10 was trying to beat.
• Understanding of memory pressure: The 16 GB VRAM constraint and how pre-staging buffers for multiple workers competes for this limited resource.

Output Knowledge Created

This message produced several critical insights:

Quantitative proof that Phase 10 was broken. The 1.17 proofs/minute throughput and 1 failure out of 3 provided unambiguous evidence that the two-lock design was not just suboptimal but fundamentally broken.

Diagnostic data for root cause analysis. The failure of the first proof (0 ms prove time) suggested a crash rather than a performance issue. The wide variance between the two successful proofs (116s vs 154s) pointed to severe serialization.

A forcing function for architectural reconsideration. This benchmark was the moment the assistant realized the two-lock approach was fundamentally flawed. It led directly to the Phase 10 post-mortem and the design of Phase 11, which abandoned the two-lock approach entirely and instead targeted DDR5 memory bandwidth contention with three targeted interventions.

Validation of the Phase 9 baseline. The fact that Phase 10 was 25% worse than Phase 9 confirmed that the single-lock approach, despite its simplicity, was actually closer to optimal for this hardware than the more complex two-lock design.

The Thinking Process Visible in the Message

The message itself is a tool call—a bash command execution. But the thinking process is visible in the choice of benchmark parameters. The assistant chose c=3 j=3 (3 concurrent proofs, 3 jobs each) rather than a higher concurrency like c=15 j=10. This was a deliberate escalation: the single-proof test (c=1 j=1) had shown 73.8 seconds, and the assistant wanted to test whether concurrent proofs would overlap better before committing to a full sweep.

The use of tee to log the output to a file shows the assistant's disciplined approach to benchmarking—every result is captured for later analysis. The assistant is building a body of evidence, not just running ad-hoc tests.

The fact that the assistant ran this benchmark immediately after confirming the daemon was ready, without any warm-up or pre-flight checks, suggests confidence that the Phase 10 code was correct. This confidence was about to be shattered.

The Aftermath

This benchmark was the turning point. In the following messages, the assistant would:

1. Abandon the Phase 10 two-lock design entirely
2. Revert the code to Phase 9's proven single-lock approach
3. Run a comprehensive concurrency sweep (c=5 through c=20)
4. Perform a detailed waterfall timing analysis from daemon logs
5. Identify DDR5 memory bandwidth contention as the true bottleneck
6. Design Phase 11 with three targeted interventions: bounding async deallocation to a single thread, reducing the groth16_pool thread count, and adding a lightweight semaphore interlock The Phase 10 failure was expensive—hours of design, coding, debugging, and benchmarking—but it produced invaluable knowledge. It revealed that CUDA's device-global synchronization model makes per-resource locking largely illusory for GPU memory management. It demonstrated that the GPU is a fundamentally serial resource at the device level, and no amount of software lock splitting can change that. And it forced a deeper investigation that uncovered the true bottleneck: DDR5 memory bandwidth contention between CPU synthesis and GPU data preparation.

Conclusion

Message 2659 is a single benchmark command, but it represents the moment a carefully designed optimization hypothesis collided with hardware reality. The Phase 10 two-lock GPU interlock was a reasonable engineering idea—split a contended lock to allow more parallelism—but it failed because CUDA's device-global synchronization model made the lock split meaningless. The 16 GB VRAM constraint made multi-worker pre-staging impossible. And the resulting benchmark, with its 1 failure and 25% regression, provided the unambiguous evidence needed to abandon the approach and search for the real bottleneck.

In the broader narrative of the cuzk optimization project, this message is the pivot point between a failed architectural approach and a successful one. It's a reminder that in systems optimization, the most valuable experiments are often the ones that fail—because they reveal the true constraints of the system.