The Moment of Failure: Diagnosing Phase 10's Two-Lock Design in the cuzk SNARK Proving Engine

Introduction

In the high-stakes world of GPU-accelerated zero-knowledge proof generation, every millisecond counts. The cuzk SNARK proving engine, designed for Filecoin's Proof-of-Replication (PoRep) protocol, had been through nine optimization phases—each one peeling back a layer of bottleneck to reveal the next. Phase 9 had shifted the critical path from PCIe transfers and GPU kernel execution to CPU memory bandwidth contention, where the prep_msm and b_g2_msm operations on the CPU dominated the per-partition wall time at ~2.4 seconds, leaving the GPU idle for ~600ms per partition. The natural next step was Phase 10: a two-lock design that would allow CPU-side memory management and GPU kernel execution to overlap, hiding the CPU overhead behind GPU compute.

Message <msg id=2612> captures the exact moment this carefully designed optimization collapsed. It is a brief, almost terse message—a single line of acknowledgment that a correctness test had failed, followed by a bash command to inspect the daemon log. Yet within this sparse output lies the first evidence of a fundamental conflict between software abstraction and hardware reality: the two-lock design, intended to increase throughput by 30–38%, had instead produced a failure that would take several rounds of debugging to fully understand.

Context: The Phase 10 Two-Lock Architecture

To appreciate what went wrong, one must understand what Phase 10 was trying to achieve. The cuzk proving engine's Groth16 proof generation pipeline had a well-characterized structure. Each partition required three major phases: VRAM allocation and pre-staging (uploading circuit data to the GPU), GPU kernel execution (the actual NTT and MSM computations), and CPU-side post-processing (prep_msm and b_g2_msm). In Phase 9, the pre-staging had been optimized to take only ~18ms, but the CPU post-processing still consumed ~2.4s per partition. The GPU, meanwhile, finished its kernels in ~1.8s. The result was a ~600ms gap where the GPU sat idle waiting for the CPU to finish.

Phase 10's insight was that if multiple GPU workers could be pipelined—one worker running kernels while another worker pre-staged data for the next partition—the CPU overhead could be hidden. The design introduced a gpu_locks struct containing two mutexes: mem_mtx (for the short VRAM allocation and pre-staging region, ~18ms) and compute_mtx (for the longer GPU kernel execution region, ~1.8s). With gpu_workers_per_device set to 3, the expectation was that workers would cycle through the locks in a staggered fashion: Worker 0 would acquire mem_mtx, pre-stage its data, release mem_mtx, then acquire compute_mtx and run kernels. Meanwhile, Worker 1 could acquire mem_mtx (since Worker 0 had released it) and begin pre-staging for the next partition, overlapping its ~18ms memory management with Worker 0's ~1.8s kernel execution.

The design document, c2-optimization-proposal-10.md, had been carefully reasoned. The assistant had analyzed deadlock risks, verified that d_a and d_bc were freed synchronously inside compute_mtx (ensuring VRAM was released before the next worker's mem_mtx phase), and concluded that the protocol was sound. The implementation followed: a new gpu_locks struct, restructured lock regions in generate_groth16_proofs_c, and the FFI interface updated to pass the opaque lock pointer. The build succeeded after fixing a type-casting error (the nvcc compiler rejected a static_cast from std::mutex* to gpu_locks*, requiring the parameter type to be changed to void*).

The Failure: Message <msg id=2612>

With the build succeeding and the daemon started (three GPU workers confirmed running), the assistant ran a correctness test: a single proof at concurrency 1, the simplest possible workload. The result was unambiguous failure:

=== Phase 10 correctness test gw=3 c=1 j=1 ===
  [1/1] FAILED — 41.1s (prove=0 ms, queue=0 ms)

A 41.1-second failure for a single proof that should have taken roughly 10–15 seconds was a clear regression. The prove=0 ms metric was particularly telling—it meant the benchmark client never received a successful proof response, only an error. The assistant's response in <msg id=2612> is immediate and focused: "Failed. Let me check the daemon log."

This is not a moment of panic or confusion. It is the disciplined reaction of an engineer who knows that the log contains the truth. The bash command tail -40 /home/theuser/cuzk-p10-daemon.log is chosen deliberately: tail -40 retrieves the last 40 lines of the log file, capturing the most recent events. The assistant expects to find error messages, OOM indicators, or timing anomalies that explain the failure.

The log output that follows is sparse but revealing:

CUZK_TIMING: b_g2_msm_ms=828 num_circuits=1
TIMELINE,91601,GPU_PICKUP,34d74ffa-32ee-4f3b-af54-e0cd9b354beb,worker=1,partition=1
[2026-02-19T17:08:33.580016Z INFO gpu_worker{worker_id=1 gpu=0 ...}]: GPU worker picked up synthesized...

Only one b_g2_msm_ms timing line appears. Only one GPU_PICKUP event is logged. In a system with three GPU workers running correctly, we would expect to see interleaved timing lines from multiple workers, showing the overlapping execution that Phase 10 was designed to achieve. Instead, the log suggests serialization: only one worker (worker 1) is making progress, and even that worker's b_g2_msm took 828ms—a reasonable time for a single partition, but the overall proof still failed.

The Deeper Problem: What the Log Doesn't Say

The log output in <msg id=2612> is truncated—the tail -40 command produced more output than what appears in the message, but the conversation data only captures the first few lines. However, even this fragment is enough to begin the diagnosis. The absence of multiple workers' timing lines is the first clue that the two-lock design is not achieving its intended overlap.

What the assistant knows at this point, based on the preceding messages, is that:

  1. Three GPU workers were successfully started (confirmed in <msg id=2610>)
  2. The first correctness test failed in 41.1s (<msg id=2611>)
  3. The daemon log shows only sparse activity The assistant's next step, visible in subsequent messages (<msg id=2613>), is to examine the timing logs more carefully. There, the full extent of the problem becomes clear: prestage_vram free_mib=1501 shows that only 1.5 GiB of VRAM is free when Worker 1 tries to pre-stage, because Worker 0's 12 GiB allocation is still live. The cudaDeviceSynchronize call inside mem_mtx is a device-global operation that blocks while another worker holds compute_mtx and runs kernels. This effectively serializes the two locks, destroying the intended overlap and forcing all subsequent partitions to use the slow fallback path.

The Assumptions That Failed

The Phase 10 design rested on several assumptions that proved incorrect:

Assumption 1: Memory management operations on a single CUDA device can be isolated from compute operations. The mem_mtx region contained cudaDeviceSynchronize and cudaMemPoolTrimTo calls, which are device-global operations. When Worker 1 held mem_mtx and called cudaDeviceSynchronize, it blocked until all pending operations on the device completed—including kernels running under Worker 0's compute_mtx. This turned the two mutexes into a single serialized pipeline.

Assumption 2: VRAM freed inside compute_mtx would be available immediately for the next worker's mem_mtx phase. While d_a and d_bc were freed synchronously via cudaFree inside the per-GPU threads, the internal MSM and batch_add allocations used cudaMallocAsync/cudaFreeAsync, which go to a stream-ordered pool. These async frees are not resolved until the stream is synchronized—and the cudaDeviceSynchronize that would resolve them was inside mem_mtx, which couldn't run until compute_mtx was released.

Assumption 3: The fallback path would handle VRAM contention gracefully. When pre-staging failed (the skip_vram path), the fallback code inside compute_mtx attempted to allocate VRAM from scratch. But without a cudaDeviceSynchronize and pool trim at the start of compute_mtx, the async pool still held cached memory from previous allocations, leading to OOM failures even in the fallback path.

The Input Knowledge Required

To understand <msg id=2612>, a reader needs substantial context about the cuzk proving pipeline. They must know that:

The Output Knowledge Created

This message produces several critical pieces of knowledge:

  1. The Phase 10 implementation has a correctness bug. The simplest possible test (one proof, concurrency 1) fails. This is not a performance regression that might be acceptable—it is a correctness failure that blocks any further progress.
  2. The daemon log is the primary diagnostic tool. The assistant immediately turns to the log, establishing a pattern of log-driven debugging that will characterize the subsequent investigation.
  3. The failure mode involves serialization. The sparse log output—only one b_g2_msm_ms timing line, only one GPU_PICKUP event—suggests that the three workers are not running concurrently as intended.
  4. The b_g2_msm operation completed successfully for one worker. The timing line b_g2_msm_ms=828 num_circuits=1 shows that the CPU-side MSM for the G2 curve took 828ms for a single circuit. This is a normal value, indicating that the CPU-side code is not the source of the failure.

The Thinking Process Visible in the Message

The assistant's reasoning in <msg id=2612> is compressed but discernible. The sequence is:

  1. Observe failure: The benchmark output shows FAILED — 41.1s. The assistant does not speculate about the cause; instead, it immediately moves to data collection.
  2. Formulate hypothesis: The failure could be caused by an OOM error, a deadlock, a CUDA API error, or a logical bug in the lock protocol. The daemon log is the most direct source of evidence.
  3. Select diagnostic tool: tail -40 is chosen to capture the most recent log entries. The assistant knows that the daemon produces structured timing lines (CUZK_TIMING:) and timeline events (TIMELINE,), and that errors would appear near the end of the log.
  4. Interpret results: The log output confirms that at least one worker reached the GPU kernel phase (b_g2_msm_ms=828) and that the GPU worker picked up a synthesized partition. But the absence of other workers' activity is noted. The thinking is disciplined and methodical. There is no wasted motion, no premature debugging. The assistant collects data first, then will analyze it in the next message.

Conclusion

Message <msg id=2612> is a study in diagnostic discipline. In just two lines of text and a bash command, it captures the moment when a carefully designed optimization meets the unforgiving reality of CUDA hardware constraints. The two-lock design of Phase 10 was theoretically sound—two independent mutexes protecting two independent resources—but it failed because the resources were not truly independent. cudaDeviceSynchronize and cudaMemPoolTrimTo are device-global operations that cannot be isolated per-lock, and the async memory pool's behavior defied the simple synchronous-free model the design assumed.

The message also illustrates a broader lesson about GPU programming: hardware constraints override software abstractions. No matter how cleanly you split your locks, a single CUDA device has a single memory space, a single stream-ordered pool, and device-wide synchronization primitives. The two-lock design was a valiant attempt to work around these constraints, but it ultimately required a deeper understanding of CUDA's memory management model—a lesson that would be fully absorbed in the subsequent debugging rounds, leading to the removal of device-wide synchronization from the mem_mtx region and a redesigned locking strategy.