The Instrumentation That Revealed a Design's Demise: Timing the Fallback Path in Phase 10

A Single Edit, a World of Context

At first glance, the message at <msg id=2651> appears unremarkable — a single line confirming that an edit to a CUDA source file was applied successfully. The assistant writes:

[edit] /home/theuser/curio/extern/supraseal-c2/cuda/groth16_cuda.cu Edit applied successfully.

There is no diff, no explanation, no fanfare. Yet this tiny message sits at a pivotal moment in the optimization journey of the cuzk SNARK proving engine. It represents the moment when the assistant, having just discovered that its ambitious Phase 10 two-lock GPU interlock design was fundamentally flawed, pivoted from building the future to diagnosing the present. The edit added timing instrumentation for the fallback path inside the compute_mtx — a path that the assistant had just realized would be taken every single time under the two-lock design, rendering the entire optimization moot. This article unpacks the reasoning, assumptions, and knowledge embedded in that single edit command.

The Broader Context: Phase 10's Ambitious Promise

To understand why this edit matters, one must understand what Phase 10 was supposed to achieve. The cuzk proving engine generates Groth16 proofs for Filecoin's Proof-of-Replication (PoRep) protocol. By Phase 9, the engine had been optimized to the point where GPU kernels took only 1.82 seconds per partition, but the overall proof time was still bottlenecked by CPU-side work — specifically prep_msm (1.91s) and b_g2_msm (0.48s) — that ran sequentially with GPU execution under a single global mutex.

Phase 10 proposed a clever two-lock architecture: split the single std::mutex guarding GPU access into two locks — compute_mtx for GPU kernel execution and mem_mtx for VRAM allocation and pre-staging. The idea was that while one worker held compute_mtx running GPU kernels, another worker could simultaneously hold mem_mtx to pre-stage buffers for the next partition, effectively overlapping CPU-side memory work with GPU computation. The design was documented in c2-optimization-proposal-10.md and implemented in the CUDA code at extern/supraseal-c2/cuda/groth16_cuda.cu.

The Discovery of Fundamental Flaws

The assistant had just completed a correctness test of Phase 10 with gpu_workers_per_device = 3 (three workers sharing one GPU). The results were sobering. A single proof took 73.8 seconds — more than double the Phase 9 baseline of 32.1 seconds. Digging into the daemon logs (see <msg id=2645>), the assistant found the smoking gun: every partition was falling back to the non-prestaged path (prestage_setup=skip_vram), and the pre_destructor_ms metric — measuring total function duration — was 11.4 seconds per partition compared to just 3.8 seconds of actual GPU time.

The root cause was a cascade of fundamental issues with the two-lock design:

1. VRAM exhaustion: The RTX 5070 Ti has only 16 GB of VRAM. During H-MSM (the most memory-intensive GPU phase), peak allocation reaches ~13.8 GB. There is simply no room for a second worker's pre-staged buffers. When Worker A holds compute_mtx with 12+ GB allocated, Worker B's mem_mtx pre-staging attempt always fails the VRAM check.
2. Device-global CUDA APIs: The critical discovery was that cudaDeviceSynchronize, cudaMemPoolTrimTo, and cudaMemGetInfo are all device-global operations. Calling them under mem_mtx blocks on all GPU streams, including those running under another worker's compute_mtx. This meant the two locks were not truly independent — they serialized through the CUDA runtime itself.
3. The fallback penalty: When pre-staging failed, the code fell back to allocating VRAM inside compute_mtx, which required its own cudaDeviceSynchronize + pool trim to reclaim async pool memory from the previous worker. This added latency directly to the critical path, making the two-lock design strictly worse than the single-lock approach.

Why This Specific Edit Matters

The edit at <msg id=2651> was the second of two timing instrumentation edits. The first (at <msg id=2649>) added a compute_wait_ms metric to measure how long each worker waited to acquire compute_mtx. The second — the subject of this article — added timing for the fallback DeviceSync+trim operations that run inside compute_mtx when pre-staging has failed.

The assistant's reasoning, visible in the surrounding messages, was methodical. Having identified that every partition was hitting the fallback path, the assistant needed to quantify the cost of that fallback. The key question was: how much of the 7.7-second gap between gpu_total_ms (3.8s) and pre_destructor_ms (11.5s) was attributable to waiting for compute_mtx (other workers' GPU time) versus the fallback's DeviceSync overhead?

This is a classic performance debugging pattern: when you suspect a code path is being taken unexpectedly, instrument it. Measure it. Make the invisible visible. The assistant was not just adding a print statement — it was building the diagnostic framework needed to decide whether to salvage Phase 10 or abandon it entirely.

Assumptions and Knowledge Required

To understand this edit, one needs significant domain knowledge spanning multiple layers of the system:

CUDA runtime semantics: The assistant had to understand that cudaDeviceSynchronize is device-global, not stream-local. This is a subtle but critical distinction — many CUDA developers assume synchronization can be isolated to a particular stream or context. The assistant's earlier discovery (documented in <msg id=2628>) that "cudaDeviceSynchronize in mem_mtx causes serialization" reflects a hard-won understanding of CUDA's actual behavior under the hood.

Memory allocation patterns: The assistant knew that H-MSM peaks at ~13.8 GB on a 16 GB card, leaving insufficient headroom for dual-worker pre-staging. This required understanding the specific memory footprint of each GPU kernel phase — d_a (4 GB), d_bc (8 GB), MSM temp buffers (1.8 GB) — and how they interact with CUDA's async memory pool.

The cuzk pipeline architecture: The edit targets generate_groth16_proofs_c in groth16_cuda.cu, a function that orchestrates the entire GPU-side proof generation pipeline. Understanding which code paths are inside versus outside each mutex scope, how prep_msm_thread interacts with GPU work, and where b_g2_msm runs relative to kernel execution is essential to interpreting the timing data.

The Phase 10 codebase state: The assistant was working with a dirty working tree containing uncommitted Phase 10 changes. The edit was applied to a file that had already been heavily modified — 84 lines added and 90 removed relative to the Phase 9 baseline. The assistant had to ensure the new timing code was consistent with the existing instrumentation framework (the CUZK_TIMING: log prefix, the TIMELINE event system) and didn't introduce compilation errors.

What the Edit Actually Changed

While the exact diff is not visible in the message, the surrounding context reveals what was added. In <msg id=2650>, the assistant read the file around lines 763-769, which contain the fallback path:

// Phase 10: If pre-staging failed (VRAM was occupied by previous worker),
// the fallback path below will allocate inside compute_mtx. We need to
// ensure previous worker's async pool frees are resolved first.
if...

The edit added timing instrumentation around the cudaDeviceSynchronize and cudaMemPoolTrimTo calls in this fallback path, measuring how long they take. This would produce output like CUZK_TIMING: fallback_sync_ms=XX trim_ms=YY in the daemon logs, allowing the assistant to quantify the penalty of the fallback path in subsequent benchmark runs.

The Output Knowledge Created

This edit produced diagnostic data that would be consumed in the very next benchmark run. The assistant rebuilt the daemon immediately after (see <msg id=2652>) and ran concurrent tests. The timing data would reveal:

• How much time the fallback's DeviceSync adds to each partition
• Whether the fallback overhead grows with concurrency (more workers = more async pool fragmentation = longer trim times)
• Whether the fallback path's cost is constant or scales with the number of competing workers More importantly, the edit contributed to a decision that crystallized over the next several messages: Phase 10's two-lock design was fundamentally unsalvageable. The combination of VRAM exhaustion, device-global CUDA APIs, and fallback overhead meant the design could never achieve its goal of overlapping CPU and GPU work. The assistant would eventually revert to Phase 9's single-lock approach (documented in the chunk summary for segment 28) and pivot to Phase 11, which targeted DDR5 memory bandwidth contention instead of GPU mutex granularity.

The Thinking Process Visible in the Edit

The assistant's reasoning chain leading to this edit reveals a disciplined approach to performance debugging:

1. Observe the symptom: 73.8s proof time vs 32.1s baseline (msg 2644)
2. Gather data: Extract timing logs, identify the prestage_setup=skip_vram pattern (msg 2645)
3. Form a hypothesis: The fallback path is the culprit, and the wait for compute_mtx is excessive (msg 2646)
4. Verify the hypothesis: Check the code to confirm what pre_destructor_ms measures (msg 2647-2648)
5. Instrument to quantify: Add timing for the specific sub-components of the suspected bottleneck (msg 2649-2651)
6. Run experiments: Rebuild and benchmark with concurrent proofs to collect the new timing data (msg 2652 onward) This is textbook performance engineering: measure before you optimize, and when you hit a wall, measure more granularly to understand why.

A Broader Lesson in Systems Engineering

The edit at <msg id=2651> is a microcosm of a larger lesson about GPU programming and systems optimization. The Phase 10 two-lock design was elegant on paper — split a contended resource into two independent locks, each guarding a different phase of work. But it failed because it violated an implicit constraint: CUDA's device model treats the GPU as a single, monolithic resource. You cannot independently lock "memory operations" and "compute operations" on the same device because the CUDA runtime itself serializes them.

This is a recurring pattern in systems engineering. Designs that assume clean separation of concerns often fail when the underlying substrate enforces tighter coupling than expected. The PCIe bus, the memory controller, the CUDA driver — all of these create implicit dependencies that defeat attempts at fine-grained parallelism. The assistant's decision to instrument the fallback path, collect data, and then abandon the design rather than continue to fight against fundamental hardware constraints demonstrates intellectual honesty and engineering discipline.

Conclusion

A single edit command — "Edit applied successfully" — is easy to overlook. But in the context of the cuzk proving engine's optimization journey, <msg id=2651> represents the moment when the assistant stopped building a flawed design and started measuring why it was flawed. The timing instrumentation added in this edit would provide the data needed to make the difficult decision to revert to Phase 9 and pursue a different optimization strategy. It is a reminder that in performance engineering, the most valuable code you can write is often the code that tells you what's actually happening — not what you think is happening.