The Moment of Reckoning: Abandoning a Flawed GPU Interlock Design

Introduction

In the high-stakes world of Filecoin proof generation optimization, not every experiment yields progress. Some yield hard-won knowledge instead. Message [msg 2662] captures one such pivotal moment: the instant when the assistant, after investing significant effort into building and testing a sophisticated two-lock GPU interlock design (Phase 10), confronted the evidence that the design was fundamentally flawed and could not be salvaged. This message is not merely a bug report or a debugging session—it is a case study in engineering judgment, where the willingness to abandon a failed approach is as important as the creativity that conceived it.

The message sits at the intersection of CUDA memory management, mutex contention, and the unforgiving constraints of 16 GB VRAM. It reveals the assistant working through a cascade of realizations: first identifying an OOM crash, then tracing it to a race condition inherent in the two-lock architecture, then considering three possible remedies, and finally settling on the most pragmatic—and most painful—option: reverting to the previous, proven design and accepting that this optimization path was a dead end.

The Context: What Phase 10 Was Trying to Achieve

To understand why this message matters, one must understand what Phase 10 was attempting to accomplish. The cuzk SNARK proving engine processes Groth16 proofs for Filecoin's Proof-of-Replication (PoRep) protocol. Each proof requires processing 10 partitions, each partition undergoing a pipeline of CPU synthesis followed by GPU computation. The GPU phase includes pre-staging data to VRAM, running NTT (Number Theoretic Transform) and MSM (Multi-Scalar Multiplication) kernels, then performing b_g2_msm (a CPU-side MSM on the G2 curve) and epilogue work after the GPU finishes.

Earlier optimization phases (Phases 5–9) had already reduced per-partition GPU time from over 5 seconds to approximately 1.8 seconds, and the team had achieved throughput of ~38 seconds per proof at high concurrency. The bottleneck had shifted: analysis showed that the GPU lock was held for the entire duration of pre-staging plus kernel execution, preventing other workers from starting their GPU work. Phase 10's two-lock design aimed to split this into a mem_mtx (for VRAM allocation and pre-staging) and a compute_mtx (for kernel execution), allowing one worker to pre-stage while another was computing, theoretically increasing GPU utilization.

The design was elegant on paper. It failed catastrophically in practice.

The OOM Crash: First Evidence of Trouble

The message opens with the assistant examining the aftermath of a benchmark run. The Phase 10 daemon had been started with gpu_workers_per_device = 3 (three GPU workers sharing a single NVIDIA GPU with 16 GB VRAM), and a concurrent benchmark with three proofs (c=3 j=3) had been launched. The results were disastrous:

[1/3] FAILED — 41.5s (prove=0 ms, queue=0 ms)
[2/3] COMPLETED — 116.1s (prove=125737 ms, queue=700 ms)
[3/3] COMPLETED — 154.2s (prove=118346 ms, queue=1133 ms)

One proof failed outright, and the two that completed were dramatically slower than the Phase 9 baseline of ~41 seconds per proof. The daemon logs revealed the cause: a panic deep inside the CUDA kernel pipeline, at sppark-0.1.14/sppark/util/gpu_t.cuh:331, with the message cudaMalloc failed: "out of memory".

The assistant immediately recognized this as an OOM crash, but the exact mechanism required careful reconstruction. The first worker had successfully pre-staged its buffers under mem_mtx, allocating approximately 12 GB of VRAM. It then released mem_mtx and waited for compute_mtx. But another worker—Worker 1—entered compute_mtx first (the order of acquisition is not guaranteed), and when it tried to allocate memory for its own GPU kernels, it found Worker 0's 12 GB of pre-staged buffers still occupying VRAM. The cudaDeviceSynchronize and cudaMemPoolTrimTo calls at the start of the compute phase were supposed to reclaim memory from the previous worker's asynchronous frees, but they could not reclaim buffers that were still live—still owned by Worker 0, which had not yet entered its compute phase to consume them.

This is the critical insight that the assistant articulates in the message: pre-staged VRAM allocated under mem_mtx persists until consumed under compute_mtx, but another worker may enter compute_mtx first and find VRAM exhausted. The two locks, intended to enable overlap, instead created a window where VRAM was double-booked.

Tracing the Race Condition

The assistant's reasoning in this message is methodical and precise. It walks through the exact sequence of events that leads to the OOM:

  1. Worker 0 acquires mem_mtx, pre-stages 12 GB of VRAM successfully, releases mem_mtx, then waits for compute_mtx.
  2. Worker 1 acquires mem_mtx, tries to pre-stage but finds VRAM occupied by Worker 0's buffers, falls back to the in-compute-mutex allocation path, releases mem_mtx, waits for compute_mtx.
  3. Worker 0 acquires compute_mtx, runs GPU kernels (which internally allocate additional temporary buffers).
  4. Worker 1 acquires compute_mtx after Worker 0 releases, runs cudaDeviceSynchronize + pool trim, then tries to allocate for its own kernels—and crashes because the pool trim cannot reclaim Worker 0's still-live pre-staged buffers. The assistant initially considers a subtle variation: "Wait — that shouldn't happen because compute_mtx serializes the GPU kernel phase." But then it realizes the race is more nuanced. The pre-staged buffers are allocated under mem_mtx but persist across the lock boundary. They are not freed until Worker 0's per_gpu threads join, which happens inside compute_mtx. So even though compute_mtx serializes kernel execution, the VRAM footprint of Worker 0's pre-staged buffers overlaps with Worker 1's allocation attempts. The assistant then checks whether the crash is in the pre-staging code or deeper in the kernel pipeline, and confirms it is inside gpu_t.cuh:331—the internal MSM workspace allocation. This means even the fallback path (which tries to allocate inside compute_mtx after a device sync) cannot succeed because the VRAM is genuinely exhausted by two workers' overlapping allocations.

The Fundamental Design Flaw

The message reaches its most important conclusion: the two-lock approach is fundamentally flawed for a 16 GB GPU. The design assumed that mem_mtx and compute_mtx could be held by different workers simultaneously, with VRAM allocation and kernel execution overlapping. But VRAM is a device-global resource—it does not respect lock boundaries. A buffer allocated by Worker 0 under mem_mtx occupies physical VRAM until Worker 0 frees it, regardless of which lock Worker 1 holds.

This is not a bug in the implementation; it is a flaw in the conceptual model. The assistant identifies the root cause with clarity: "pre-staging allocates ~12 GB that persists through the entire compute phase." With 16 GB total VRAM and each worker needing ~12 GB for its pre-staged buffers plus additional temporary allocations during kernel execution, there is simply no room for two workers' allocations to coexist.

The assistant also recognizes a second, subtler issue: CUDA memory management APIs like cudaDeviceSynchronize and cudaMemPoolTrimTo are device-global operations. Calling them from one worker's compute phase affects the entire GPU device, potentially interfering with another worker's ongoing kernel execution. This means the two-lock split cannot actually achieve true parallelism on the memory management path—the device-global nature of these APIs defeats the purpose of splitting the locks.

Evaluating the Options

Having identified the fundamental flaw, the assistant does not immediately abandon the effort. Instead, it considers three possible remedies:

  1. Abandon pre-staging entirely with gw>1: Accept that pre-staging cannot work when multiple workers share a GPU, and fall back to allocating inside the compute lock. This would eliminate the VRAM double-booking but also eliminate the theoretical benefit of overlapping allocation with computation.
  2. Use a single lock but release it earlier: Return to the Phase 9 single-lock design, but release the lock immediately after GPU kernel execution completes, before b_g2_msm and epilogue. This is essentially what Phase 9 already did—the lock was already released before b_g2_msm. The assistant verifies this by checking the Phase 9 code.
  3. Add a VRAM reservation/handshake: Implement a mechanism where only the worker whose pre-staged buffers are live can enter compute_mtx. This would prevent the race condition but add complexity and potentially serialization. The assistant evaluates each option against the constraints of the system. Option 3 is rejected as overly complex for a single-GPU configuration. Option 1 is considered but ultimately unnecessary—the assistant realizes that Phase 9 already achieves the key benefit (releasing the lock before b_g2_msm) without the complexity of two locks. Option 2 is the winner, but it is not really an option at all—it is a reversion to the status quo.

The Decision to Revert

The message culminates in a clear decision: revert to Phase 9's single-lock approach. The assistant states this with the kind of measured finality that comes from thorough analysis: "Let me simplify: skip pre-staging entirely, use compute_mtx only for the GPU kernel region, and let b_g2_msm + epilogue + host unregister run outside the lock."

But the assistant does not stop at reversion. It performs a crucial sanity check by examining the Phase 9 code to confirm that the single lock is already released before b_g2_msm. This check reveals something important: Phase 9 already achieves the overlap that Phase 10 was trying to create. In Phase 9, the gpu_lock is acquired at line 641, held through pre-staging and GPU kernel execution, then released at line 960—before prep_msm_thread.join() at line 969, which waits for b_g2_msm to complete. This means the next worker can start its GPU phase while the previous worker's b_g2_msm is still running on the CPU.

The assistant's realization is almost philosophical: "Phase 10's two-lock split doesn't save anything over Phase 9—Phase 9 already hides b_g2_msm." The entire Phase 10 effort, with its sophisticated lock splitting and pre-staging logic, was attempting to solve a problem that had already been solved by a simpler design.

The Broader Lesson: When Optimization Paths Are Dead Ends

This message is valuable not just for its technical content but for what it reveals about the optimization process itself. The assistant had invested significant effort in Phase 10: designing the two-lock architecture, implementing the pre-staging logic, adding timing instrumentation, building and testing the code, and running benchmarks. The temptation to salvage something from this investment—to find a way to make the two-lock approach work—must have been strong.

Instead, the assistant chose to confront the evidence honestly. The OOM crash was not a minor bug to be fixed; it was a symptom of a fundamental incompatibility between the design and the hardware constraints. The 16 GB VRAM limit was not going to change. The device-global nature of CUDA memory management was not going to change. The only variable the assistant could change was the design itself, and the correct change was to abandon it.

This is a pattern that recurs throughout engineering: the most valuable optimization is sometimes the decision to stop optimizing in a particular direction and redirect effort elsewhere. The assistant's subsequent actions—running a comprehensive benchmark sweep across concurrency levels, performing waterfall timing analysis, and designing Phase 11 with three targeted interventions to reduce DDR5 memory bandwidth contention—all stem from this decision. By accepting that Phase 10 was a dead end, the assistant freed itself to discover the real bottleneck.

The Thinking Process Visible in the Message

The message's reasoning structure is worth examining in detail. It follows a pattern that experienced engineers will recognize:

  1. Observe the symptom: OOM crash during concurrent benchmark.
  2. Form a hypothesis: Pre-staged buffers from Worker 0 occupy VRAM when Worker 1 tries to allocate.
  3. Test the hypothesis: Trace through the lock acquisition sequence, check whether the crash is in pre-staging or kernel code.
  4. Refine the understanding: Realize the race is more subtle—Worker 1 enters compute_mtx before Worker 0, finding VRAM exhausted.
  5. Identify the root cause: Pre-staged VRAM persists across lock boundaries; device-global CUDA APIs defeat lock isolation.
  6. Consider alternatives: Three options evaluated against hardware constraints.
  7. Make the decision: Revert to Phase 9, which already achieves the intended benefit.
  8. Verify the decision: Check Phase 9 code to confirm lock release timing. Each step builds on the previous one, and the assistant is careful to distinguish between what it knows (the OOM crash, the lock sequence) and what it infers (the race condition, the device-global API limitation). The message also shows the assistant correcting its own assumptions in real time: "Wait — that shouldn't happen because compute_mtx serializes..." followed by the realization that the race is more nuanced.

Assumptions and Their Consequences

The Phase 10 design rested on several assumptions that proved incorrect:

Conclusion

Message [msg 2662] captures a moment of clarity in the optimization journey. It is the message where the assistant confronts the failure of an ambitious design, traces its root cause with precision, evaluates alternatives honestly, and makes the difficult decision to revert to a simpler approach. The Phase 10 two-lock GPU interlock was not a waste—it was an experiment that produced valuable negative knowledge. The assistant learned that CUDA device-global synchronization defeats lock-based isolation, that 16 GB VRAM cannot support overlapping pre-staged allocations, and that Phase 9's single-lock design was already optimal for the single-GPU configuration.

This knowledge directly informed the next phase of optimization. By understanding why Phase 10 failed, the assistant could redirect its attention to the real bottleneck: DDR5 memory bandwidth contention between CPU synthesis and GPU preprocessing. The Phase 11 interventions—bounding async deallocation, reducing thread pool oversubscription, and adding a lightweight synthesis throttle—all target this bottleneck with surgical precision, informed by the hard-won lessons of Phase 10.

In engineering, the most valuable experiments are sometimes the ones that fail. Message [msg 2662] is a testament to that truth.