The Build That Passed Into a Trap: Phase 10's Moment of False Confidence

A Single Line of Bash That Changed Everything

On the surface, message [msg 2608] is one of the most unremarkable moments in the entire optimization saga. It contains nothing but a bash command and its output:

[assistant] [bash] rm -rf target/release/build/supraseal-c2-* && cargo build --release -p cuzk-daemon 2>&1 | tail -10
   Compiling storage-proofs-core v19.0.1
   Compiling storage-proofs-porep v19.0.1
   Compiling storage-proofs-post v19.0.1
   Compiling storage-proofs-update v19.0.1
   Compiling filecoin-proofs v19.0.1
   Compiling filecoin-proofs-api v19.0.0
   Compiling cuzk-core v0.1.0 (/home/theuser/curio/extern/cuzk/cuzk-core)
   Compiling cuzk-server v0.1.0 (/home/theuser/curio/extern/cuzk/cuzk-server)
   Compiling cuzk-daemon v0.1.0 (/home/theuser/curio/extern/cuzk/cuzk-daemon)
    Finished `release` pr...

A build succeeds. The compiler is satisfied. The linker resolves every symbol. The binary is written to disk. And yet this moment of apparent success is the precise inflection point where a carefully architected optimization begins its collision course with physical reality. The build passes, but the design is already broken — and nobody knows it yet.

The Context: Phase 10's Ambitious Two-Lock Design

To understand why this build matters, one must understand what Phase 10 was trying to achieve. The optimization journey had been progressing through a series of increasingly sophisticated bottlenecks. Phase 9 had identified that the bottleneck had shifted from GPU kernel execution and PCIe transfers to CPU memory bandwidth contention — the prep_msm (1.9s) and b_g2_msm (0.48s) CPU operations were now dominating the per-partition wall time at ~2.4s, leaving the GPU idle for ~600ms per partition.

The Phase 10 solution, designed in the immediately preceding messages, was elegant in theory. The existing architecture used a single std::mutex* gpu_mtx that serialized all GPU operations — both memory management (VRAM allocation, pre-staging uploads) and compute (NTT kernels, MSM operations). The insight was that these two phases had very different durations: memory management took only ~18ms, while compute took ~1.8s. By splitting the single lock into two — a mem_mtx for the short memory operations and a compute_mtx for the long compute operations — the design aimed to let one worker's memory phase overlap with another worker's compute phase, effectively hiding the CPU overhead.

The implementation required replacing the opaque std::mutex* pointer with a new struct gpu_locks { std::mutex compute_mtx; std::mutex mem_mtx; }, restructuring the lock regions in generate_groth16_proofs_c, and increasing gpu_workers_per_device from 2 to 3. The expected improvement was dramatic: per-partition wall time dropping from ~3.7s to ~1.8–2.0s, a 30–38% throughput improvement.

The Build: A Deliberate Clean Compilation

Message [msg 2608] is the first successful build of this new architecture. It follows immediately after message [msg 2607], where the assistant fixed a type-casting error: the function parameter generate_groth16_proofs_c had been declared as taking std::mutex* gpu_mtx, but the new code was passing a gpu_locks* pointer. The fix was to change the parameter type to void*, matching the FFI contract where the Rust side passes it as *mut c_void.

The build command itself reveals deliberate engineering choices. The rm -rf target/release/build/supraseal-c2-* is not accidental — it clears the cached build artifacts for the supraseal-c2 package, forcing a full recompilation of the CUDA code. This is necessary because incremental builds sometimes fail to detect changes in .cu files compiled through the cc-rs/nvcc pipeline. The 2>&1 | tail -10 pipes stderr into stdout and shows only the last 10 lines, focusing attention on the final compilation result rather than the thousands of lines of intermediate output.

The build output tells a story of its own. The compiler recompiles the entire dependency chain: storage-proofs-core, storage-proofs-porep, storage-proofs-post, storage-proofs-update, filecoin-proofs, filecoin-proofs-api, cuzk-core, cuzk-server, and finally cuzk-daemon. This is not just the CUDA code — changing the FFI interface in the C++ file triggers a cascade of recompilations through the Rust build system because the Rust side links to the C++ object files. The build succeeds, and the daemon binary is ready.

The Hidden Assumption: Device-Global Independence

The build's success masks a critical flaw in the Phase 10 design. The two-lock architecture rests on an assumption that memory management operations on a CUDA device can be isolated from compute operations on the same device. Specifically, the mem_mtx region was designed to handle VRAM allocation and pre-staging uploads (~18ms) independently of the compute_mtx region that runs GPU kernels (~1.8s).

This assumption is wrong. The mem_mtx region calls cudaMemPoolTrimTo to reclaim cached memory from the stream-ordered allocation pool, and the design initially removed cudaDeviceSynchronize from this region to keep it fast. But cudaMemPoolTrimTo and cudaDeviceSynchronize are device-global operations — they affect the entire CUDA device, not just the calling thread's stream. When one worker holds compute_mtx and runs kernels, and another worker tries to call cudaMemPoolTrimTo under mem_mtx, the trim operation cannot fully reclaim memory because the kernels running under compute_mtx still have outstanding stream-ordered allocations.

The result, which will be discovered in the very next message ([msg 2611]), is catastrophic: the first worker pre-stages its 12 GiB of VRAM, then enters compute_mtx to run kernels. The second worker acquires mem_mtx, calls cudaMemPoolTrimTo, finds only ~5.5 GiB free, and falls back to the non-prestaged path. But even the fallback path fails with OOM because the device-global synchronization required to resolve pool frees never happens — it's blocked by compute_mtx.

The Dramatic Irony of a Clean Build

There is profound dramatic irony in message [msg 2608]. The assistant has just spent several messages designing and implementing what appears to be a sound optimization. The type-casting error was fixed. The code compiles cleanly. The build output shows Finished \release\ profile [optimized] target(s) — the compiler's stamp of approval.

But the compiler cannot validate the runtime semantics of CUDA device-global synchronization. It cannot detect that two mutexes on the same device will effectively serialize because their operations are not truly independent. It cannot warn that cudaMemPoolTrimTo without a preceding cudaDeviceSynchronize will leave pool memory in an inconsistent state. The compiler sees only types, not hardware constraints.

This is a recurring theme in GPU programming: the abstraction boundary between software concurrency models and hardware execution models is porous. A mutex can protect a critical section from concurrent access, but it cannot protect against implicit device-wide synchronization effects. The CUDA programming model has a "hidden" global state — the memory pool, the stream-ordered allocator, the device-wide synchronization barrier — that no amount of lock splitting can isolate.

Input Knowledge Required

Understanding this message requires substantial context from the preceding optimization pipeline:

  1. The Phase 9 bottleneck analysis: The realization that CPU memory bandwidth contention (specifically prep_msm at 1.9s and b_g2_msm at 0.48s) had become the dominant factor, leaving the GPU idle for ~600ms per partition.
  2. The two-lock design proposal ([msg 2586]–[msg 2588]): The gpu_locks struct with mem_mtx and compute_mtx, the restructuring of lock regions, and the expected performance improvements.
  3. The CUDA memory model: Understanding stream-ordered allocations (cudaMallocAsync/cudaFreeAsync), the memory pool (cudaMemPoolTrimTo), and device-global synchronization (cudaDeviceSynchronize).
  4. The FFI boundary: The Rust side passes the mutex as *mut c_void, and the C++ side casts it to the appropriate type. The type-casting error in message [msg 2605]–[msg 2607] reveals the fragility of this opaque-pointer pattern.
  5. The VRAM budget: Each worker requires ~12 GiB of VRAM for d_a and d_bc allocations. With 3 workers on a single GPU, the total requirement exceeds available memory unless allocations are carefully orchestrated.

Output Knowledge Created

The build produces a cuzk-daemon binary that embodies the Phase 10 two-lock architecture. This binary is immediately deployed for correctness testing in the next message ([msg 2609]–[msg 2610]). The output is not just a binary, however — it is a testable hypothesis about GPU concurrency. The build transforms the design document into executable code, making the assumptions of the design falsifiable.

The build output also reveals the dependency structure of the project. The recompilation chain — from storage-proofs-core through cuzk-daemon — shows that changing the C++ FFI interface has ripple effects through the entire Rust build graph. This is important engineering knowledge: the FFI boundary is not a clean abstraction layer but a coupling point that can trigger widespread recompilation.

The Thinking Process Visible in the Build

While message [msg 2608] itself contains no explicit reasoning — it is a bare command and its output — the thinking process is visible in what precedes it. The assistant's reasoning in messages [msg 2605]–[msg 2607] shows a systematic debugging approach to the compilation error:

  1. Error detection: The build fails with error: invalid type conversion at line 161, where static_cast<gpu_locks*>(gpu_mtx) tries to cast from std::mutex* to gpu_locks*.
  2. Root cause analysis: The function parameter gpu_mtx is declared as std::mutex* in the function signature, but the new code creates gpu_locks objects and passes them through the same opaque pointer interface.
  3. Fix design: Change the parameter type from std::mutex* to void*, matching the FFI contract. The static_cast<gpu_locks*> then becomes valid because the pointer is void*.
  4. Verification: The assistant greps for any remaining references to the old gpu_lock or mtx_ptr names to ensure no stale code remains. This debugging process reveals a careful, methodical approach to refactoring. The assistant doesn't just fix the immediate error — it verifies that no other references to the old types remain, ensuring the codebase is internally consistent.

The Broader Lesson: Hardware Constraints Trump Software Abstraction

Message [msg 2608] is a case study in the fundamental tension between software abstraction and hardware reality. The two-lock design is a beautiful software abstraction: two independent mutexes protecting two independent resources (memory management and compute). But on a single CUDA device, these resources are not independent. The memory pool is shared. Device-wide synchronization is global. Stream-ordered allocations are tied to streams that may be running on the same hardware queues.

The build's success is therefore a false signal. It tells the engineer that the code is syntactically correct, type-safe, and linkable. But it says nothing about whether the code will work correctly at runtime. The real validation — the correctness test — will come in the next message and will fail spectacularly, with a 41-second prove time and an OOM error.

This is a pattern that repeats across GPU programming: the compiler cannot protect you from hardware semantics. Thread safety (mutexes) does not imply device safety (global synchronization). The two-lock design would work perfectly on a system with two independent GPUs, or on a CPU-only system where memory management and compute are truly independent. But on a single CUDA device, the abstraction leaks.

Conclusion

Message [msg 2608] is the quiet before the storm. A build succeeds, a binary is produced, and an incorrect assumption about CUDA device semantics is about to be empirically refuted. The message is a reminder that in systems programming — and especially in GPU programming — the compiler's approval is only the first gate. The second gate is the hardware, and it is far less forgiving.

The build that passed into a trap teaches a lesson that no amount of static analysis can replace: understand the hardware's implicit synchronization model before designing your concurrency architecture. The two-lock design was clever, but it was clever in the wrong dimension. The real solution, which the assistant will discover through debugging, is to remove device-global synchronization from the mem_mtx region entirely and rely on the fallback path — a humbler design that acknowledges the hardware's constraints rather than trying to abstract over them.