The 73.8-Second Canary: How a Single Benchmark Exposed the Fundamental Flaw in Phase 10's Two-Lock GPU Interlock

The Message

In the course of a deep optimization campaign for the SUPRASEAL_C2 Groth16 proof generation pipeline, the assistant executed the following command to test a newly implemented design:

FIL_PROOFS_PARAMETER_CACHE=/data/zk/params /home/theuser/curio/extern/cuzk/target/release/cuzk-bench batch --type porep --c1 /data/32gbench/c1.json --count 1 --concurrency 1 2>&1 | tee /home/theuser/cuzk-p10-c1j1-bench.log

The output was devastating:

=== Batch Benchmark ===
proof type:  porep
count:       1
concurrency: 1

  [1/1] COMPLETED — 73.8s (prove=102381 ms, queue=244 ms)

=== Batch Summary ===
total time:    73.8s
completed:     1
failed:        0
wall time:     avg=73.8s min=73.8s max=73.8s
prove time:    avg=102.4s min=102.4s max=102.4s
throughput:    0.813 proofs/min (73.8s/proof)

A single proof took 73.8 seconds. The Phase 9 baseline — the very code this new design was supposed to improve upon — completed the same proof in 32.1 seconds. The new design was 2.3× slower than the old one. This message, [msg 2644], is the moment the Phase 10 two-lock GPU interlock design was first confronted with reality, and reality was brutal.

Context: What Was Phase 10 Trying to Do?

To understand why this message was written, we must understand what Phase 10 was attempting to accomplish. The optimization campaign had progressed through nine phases, each targeting a different bottleneck in the Groth16 proof generation pipeline. Phase 9 had achieved a breakthrough: by pre-staging NTT uploads over PCIe and implementing a deferred sync pattern in the Pippenger MSM kernel, GPU kernel time had been reduced from 3.75s to 1.82s per partition. But a new bottleneck had emerged: the CPU-side critical path (prep_msm at 1.91s + b_g2_msm at 0.48s = 2.39s) now exceeded GPU kernel time. The GPU was spending nearly half its time waiting for CPU work to complete.

Phase 10's design, documented in c2-optimization-proposal-10.md, proposed a two-lock GPU interlock. Instead of a single std::mutex protecting the entire GPU compute + memory region, the design split the lock into compute_mtx (for GPU kernel execution) and mem_mtx (for VRAM allocation and pre-staging). The idea was that three GPU workers could overlap: while Worker A held compute_mtx running kernels, Worker B could hold mem_mtx to pre-stage its buffers, and Worker C could be doing CPU work (b_g2_msm, epilogue) outside any lock. In theory, this would eliminate the CPU-waits-for-GPU and GPU-waits-for-CPU serialization, allowing throughput to approach the GPU's true silicon limit.

The Assumptions Behind the Design

The two-lock design rested on several critical assumptions, none of which survived contact with reality:

Assumption 1: VRAM capacity is sufficient for overlapping allocations. The design assumed that 16 GB of VRAM could accommodate one worker's active kernel buffers (~12 GB during H-MSM) plus another worker's pre-staged buffers (~12 GB). This would require ~24 GB on a 16 GB card — a physical impossibility.

Assumption 2: CUDA memory management APIs are per-context or per-stream. The design assumed that cudaDeviceSynchronize, cudaMemPoolTrimTo, and cudaMemGetInfo could be called under mem_mtx without interfering with kernels running under compute_mtx. In reality, these APIs are device-global — they synchronize or query the entire GPU device, not a specific stream or context. Calling cudaDeviceSynchronize under mem_mtx would block until all streams complete, including those running under another worker's compute_mtx.

Assumption 3: Pre-staging would succeed most of the time. The design assumed that by the time a worker acquired mem_mtx, the previous worker's VRAM allocations would have been freed by CUDA's asynchronous memory pool. In practice, the previous worker's kernel buffers were still live, so cudaMalloc for pre-staging would fail, triggering a fallback path that was even slower than the original single-lock approach.

Assumption 4: Three workers on one GPU would improve throughput. The design implicitly assumed that GPU compute was the bottleneck and that adding more workers would allow better overlap. But with a single GPU, all workers queue on the same hardware. The only thing multiple workers can overlap is CPU work (b_g2_msm, epilogue) that runs after the GPU lock is released — and Phase 9 already did that.

What the 73.8s Result Actually Revealed

The 73.8 seconds for a single proof (with concurrency=1 and count=1) was a worst-case scenario for the two-lock design. With only one proof and three GPU workers, all three workers were competing for the same GPU to process the same proof's 10 partitions. The timeline looked like this:

  1. Worker 0 acquires mem_mtx, attempts pre-staging. With no previous worker's buffers on the GPU, pre-staging succeeds (~12 GB allocated). Worker 0 releases mem_mtx and queues for compute_mtx.
  2. Worker 1 acquires mem_mtx, attempts pre-staging. Worker 0's 12 GB of pre-staged buffers are still on the GPU. cudaMalloc fails. Worker 1 falls back to the slow path, releases mem_mtx, queues for compute_mtx.
  3. Worker 2 acquires mem_mtx, same story — pre-staging fails, falls back.
  4. Worker 0 acquires compute_mtx, runs GPU kernels (~3.8s), releases compute_mtx.
  5. Worker 1 acquires compute_mtx, runs fallback DeviceSync+pool trim, then GPU kernels (~3.8s + overhead).
  6. Worker 2 acquires compute_mtx, same pattern. The result: every partition took ~11.5s instead of ~3.8s, because the pre_destructor_ms timer (measuring total function duration) included ~7s of waiting for compute_mtx while other workers ran their GPU kernels. With 10 partitions, the proof took 73.8s instead of the expected ~32s.

The Thinking Process Visible in This Message

This message is remarkable for what it doesn't contain: any analysis, any commentary, any reasoning. It is a single, bare command execution — a benchmark invocation with no prefatory explanation or post-hoc interpretation. The assistant simply ran the test and logged the output.

This terseness itself tells us something about the assistant's state of mind. The assistant had just spent multiple rounds building, debugging, and refining the Phase 10 two-lock design. It had added timing instrumentation, diagnosed OOM errors, and made last-minute edits to remove cudaDeviceSynchronize from the mem_mtx path. The build had succeeded. The daemon was running. The moment of truth had arrived.

The assistant ran the simplest possible test: a single proof at concurrency 1. This is the minimal correctness check — if the design is fundamentally broken, this test will show it immediately. And it did. The 73.8s result is presented without comment, without spin, without explanation. It is what it is.

In the very next message ([msg 2645]), the assistant begins the investigation: "Proof passes correctness (completed, not failed). But 73.8s is very slow — let me check the daemon logs to understand what happened." This is where the real thinking begins — the grep through TIMELINE events, the discovery that every partition falls back (prestage_setup=skip_vram), the realization that pre_destructor_ms is 11-12s per partition because workers are queueing on compute_mtx.

The Broader Significance

This message represents a critical juncture in the optimization campaign. It is the moment when a theoretically elegant design — the two-lock GPU interlock — collided with the physical constraints of the hardware. The design was not wrong in principle; on a multi-GPU system with abundant VRAM, splitting memory management from compute management could yield real benefits. But on a single RTX 5070 Ti with 16 GB VRAM, the design was doomed from the start.

The 73.8s result forced a fundamental re-evaluation. Over the next several messages, the assistant would:

Input Knowledge Required

To understand this message, one needs to know:

Output Knowledge Created

This message created:

Mistakes and Incorrect Assumptions

The primary mistake was the assumption that VRAM capacity and CUDA's device-global API semantics would not be limiting factors. The assistant had noted earlier that "pre-staging will almost always fail with gw≥2" (discovery #11 in the goal message), but proceeded with the implementation anyway, hoping that the fallback path would be fast enough. It was not.

A secondary mistake was testing with concurrency=1 and count=1 — the worst possible workload for gw=3, since all three workers compete for the same single proof's partitions. A more informative first test might have been concurrency=3 count=3, which would have shown whether the design could at least handle independent proofs in parallel. But the assistant chose the minimal correctness test first, which is methodologically sound even if it produced the most damning result.

Conclusion

Message [msg 2644] is a study in the scientific method applied to systems engineering. A hypothesis was formed (two-lock GPU interlock will improve throughput), an experiment was designed (benchmark a single proof with gw=3), the experiment was executed, and the hypothesis was falsified — decisively. The 73.8-second proof time was not just a bad result; it was a diagnostic signal that revealed fundamental design flaws. The assistant's response — immediate investigation, root cause analysis, reversion to known-good state, and formulation of a new hypothesis — is a textbook example of disciplined optimization work. Sometimes the most valuable experiment is the one that fails, because failure teaches you where the real constraints are.