The Build That Succeeded Into a Dead End

A Single Compilation Command That Marked the Turning Point in GPU Optimization

On the surface, message [msg 2635] appears to be one of the most mundane moments in any software engineering conversation: a build command. The assistant issues a single bash tool call:

cargo build --release -p cuzk-daemon 2>&1

The output is equally unremarkable—a single compiler warning about an unexpected cfg condition value groth16 in an unrelated dependency (bellpepper-core/src/lc.rs), followed by the implicit success of a completed compilation. No errors, no failures. The build simply worked.

But this message is far from mundane. It sits at a critical inflection point in a months-long optimization campaign for the cuzk SNARK proving engine, a system that generates Groth16 proofs for Filecoin's Proof-of-Replication (PoRep) protocol. The build in [msg 2635] is the first successful compilation of Phase 10, a two-lock GPU interlock design that the assistant had spent hours designing, implementing, and debugging. And within minutes of this build completing, the entire Phase 10 approach would be revealed as fundamentally flawed—abandoned, reverted, and replaced with a completely different strategy.

This article examines that moment: why the build was necessary, what assumptions it carried, what knowledge it required, and how its quiet success set the stage for one of the most instructive failures in the optimization pipeline.

Context: The Optimization Campaign So Far

To understand message [msg 2635], one must understand the trajectory that led to it. The cuzk project had progressed through nine optimization phases, each targeting a specific bottleneck in the Groth16 proof generation pipeline. Phase 9 had achieved a significant breakthrough: by pre-staging NTT (Number Theoretic Transform) uploads over PCIe gen5 and implementing a double-buffered deferred sync in the Pippenger MSM (Multi-Scalar Multiplication) kernel, the assistant had reduced per-partition GPU kernel time from 3.75 seconds to 1.82 seconds. However, this exposed a new bottleneck: the CPU-side critical path (prep_msm at 1.91 seconds plus b_g2_msm at 0.48 seconds) now exceeded GPU time, leaving GPU silicon utilization at only 48.5%.

Phase 10 was designed to address this by splitting the single GPU mutex into two locks: a compute_mtx for GPU kernel execution and a mem_mtx for VRAM allocation. The idea was that with three GPU workers per device, one worker could hold compute_mtx and run kernels while another worker held mem_mtx to pre-stage buffers for the next partition, overlapping CPU work with GPU execution. The design was documented in c2-optimization-proposal-10.md, the CUDA code was restructured with a gpu_locks struct replacing the single std::mutex, and the Rust FFI layer was updated to pass opaque *mut c_void pointers throughout.

But the implementation had hit problems. An early attempt that placed cudaDeviceSynchronize inside mem_mtx caused catastrophic serialization—cudaDeviceSynchronize is a device-global operation that blocks until all GPU streams complete, including those running under another worker's compute_mtx. A subsequent edit removed the device-wide synchronization calls from mem_mtx, replacing them with a simpler "try cudaMalloc directly, fail fast on OOM" approach. That edit—the last modification to groth16_cuda.cu—had never been compiled.

That is the state captured in message [msg 2635]. The assistant has just finished describing the situation in a lengthy planning message ([msg 2628]), noting that "the build needs to be done and tested." The previous message ([msg 2634]) explicitly states: "The code has Phase 10 two-lock changes that haven't been compiled yet. Let me build, test, and benchmark." Message [msg 2635] is the execution of that first step.

The Build Itself: What It Reveals

The build command is straightforward: cargo build --release -p cuzk-daemon 2>&1. The 2>&1 redirects stderr to stdout, ensuring all output—including warnings—is captured. The --release flag enables optimizations. The -p cuzk-daemon targets only the daemon package, not the entire workspace.

The output shows a single warning from bellpepper-core/src/lc.rs:

warning: unexpected `cfg` condition value: `groth16`
   --> /home/theuser/curio/extern/bellpepper-core/src/lc.rs:487:17
    |
487 | #[cfg(all(test, feature = "groth16"))]
    |                 ^^^^^^^^^^^^^^^^^^^ help: remove the condition
    |
    = note: no expected values for `feature`
    = help: consider adding `groth16` as a feature in `Cargo.toml`

This warning is about a Rust cfg-checking lint. The bellpepper-core crate uses #[cfg(feature = "groth16")] but hasn't declared groth16 as an expected feature in its Cargo.toml. This is a pre-existing issue in a dependency, not something introduced by Phase 10. The assistant does not act on this warning—it's harmless, and the build succeeds.

The fact that the build succeeds is significant. It confirms that the complex CUDA code changes—the gpu_locks struct, the two-mutex locking strategy, the restructured pre-staging and fallback paths—are syntactically correct and link properly. The C++ compiler accepted the template instantiations, the CUDA device compiler compiled the kernels, and the Rust FFI layer connected to the compiled C++ objects without symbol resolution errors.

The Assumptions Embedded in This Build

Message [msg 2635] carries several implicit assumptions, some of which would prove incorrect:

Assumption 1: The two-lock design is viable. The entire Phase 10 effort assumes that splitting the GPU mutex into compute_mtx and mem_mtx can yield throughput improvements. The reasoning is that overlapping CPU work (b_g2_msm, prep_msm, epilogue) with GPU kernel execution will reduce the per-partition wall-clock time. This assumption is not tested by the build—it's a design assumption that the build merely enables testing.

Assumption 2: Pre-staging under mem_mtx can succeed with multiple workers. The design depends on Worker B being able to allocate VRAM buffers (pre-staging) while Worker A holds compute_mtx with ~12 GiB of active allocations. On a 16 GiB GPU, this is extremely tight. The assistant had already noted this concern in [msg 2628]: "Pre-staging will almost always fail with gw≥2." The build doesn't test this—only runtime will reveal whether cudaMalloc succeeds or returns OOM.

Assumption 3: The build is worth doing. There is an implicit judgment that the Phase 10 approach, despite known concerns, is sufficiently promising to warrant compilation and testing. The assistant could have abandoned the two-lock design at the planning stage, but instead chose to build it—a decision that reflects a disciplined experimental methodology: implement, test, measure, and let data guide decisions.

Assumption 4: The warning is ignorable. The groth16 cfg warning is treated as a pre-existing issue in a dependency, not something that needs fixing. This is a reasonable judgment—the warning doesn't affect correctness or performance—but it's still an assumption that the assistant doesn't investigate further.

What Knowledge Was Required to Understand This Message

To fully grasp message [msg 2635], a reader needs substantial context:

The optimization pipeline architecture. The cuzk system is a pipelined SNARK proving engine that processes Filecoin PoRep proofs through multiple stages: synthesis (constraint system evaluation), MSM (multi-scalar multiplication on the GPU), NTT (number theoretic transforms), and Groth16 proof assembly. The "partition" concept is central—each proof is divided into partitions that can be processed independently, with partition_workers controlling parallelism.

The CUDA locking problem. GPU programming introduces unique synchronization challenges. CUDA streams can execute kernels concurrently on the same device, but memory management APIs like cudaDeviceSynchronize, cudaMemPoolTrimTo, and even cudaMalloc can have device-global effects. A mutex split that works for CPU-side resources may not work for GPU-side resources because the GPU device itself is a shared, non-partitionable resource.

The hardware constraints. The RTX 5070 Ti has 16 GB VRAM, and peak allocation during H-MSM reaches ~13.8 GiB. With only ~2.2 GiB of headroom, pre-staging buffers for a second worker is nearly impossible. The AMD Threadripper PRO 7995WX has 96 Zen4 cores and 8-channel DDR5 memory (~300 GB/s theoretical bandwidth), which becomes a contention bottleneck when 192 rayon synthesis threads compete with 192 groth16_pool threads for memory access.

The Phase 9 baseline. The 32.1 seconds per proof in isolation and 41.3 seconds at c=15 concurrency provide the benchmark against which Phase 10 would be measured. Without this baseline, the build in [msg 2635] has no purpose—it's just compiling code.

The Knowledge Created by This Build

Message [msg 2635] produces one critical piece of knowledge: the Phase 10 code compiles. This is non-trivial. The CUDA code involves template-heavy MSM kernels, complex locking logic across C++ and Rust FFI boundaries, and careful management of CUDA streams, events, and memory pools. A compilation failure could indicate anything from a simple syntax error to a fundamental ABI mismatch between the Rust mutex pointer and the C++ gpu_locks struct.

The successful build also creates the precondition for all subsequent testing. Without it, the assistant cannot run correctness checks, cannot benchmark, cannot measure whether the two-lock design actually improves throughput. The build is the gate that must open before any empirical evaluation can begin.

But the build also creates a subtle form of knowledge that is easy to overlook: it confirms that the code changes are self-consistent. The C++ compiler checked types, template instantiations, and function signatures. The linker resolved all symbols. The Rust compiler verified that the FFI declarations match the C++ ABI. These are all compile-time guarantees that, while they don't ensure runtime correctness, do eliminate a large class of bugs.

The Thinking Process Visible in This Message

The assistant's reasoning is not explicit in message [msg 2635] itself—the message contains only a bash command and its output. But the thinking is visible in the surrounding context. The previous message ([msg 2634]) lays out the plan explicitly: "The code has Phase 10 two-lock changes that haven't been compiled yet. Let me build, test, and benchmark." This is followed by "Step 1: Build. First, force rebuild of the CUDA code since .cu files changed," and a command to remove cached build artifacts (rm -rf target/release/build/supraseal-c2-*).

The force-rebuild step is important. Cargo's build system caches compiled artifacts, and when .cu files change, the build system may not always detect the need to recompile. By deleting the build cache directory, the assistant ensures a clean rebuild. This is a learned behavior from previous experience with CUDA compilation in this project—the assistant has internalized the quirks of the build system and compensates for them.

The choice to build only the daemon (-p cuzk-daemon) rather than the entire workspace is also deliberate. It minimizes compilation time while still producing the binary needed for testing. The bench tool (cuzk-bench) is built separately in the next message ([msg 2636]) with different features (--no-default-features).

The Aftermath: Why This Build Matters

What makes message [msg 2635] truly interesting is not the build itself but what follows. The next messages show the assistant starting the daemon with the gw=3 (3 GPU workers) configuration, running a single proof to test correctness, and discovering that the two-lock design performs terribly—72.5 seconds for the first proof, far worse than the Phase 9 baseline.

The root cause is exactly what the assistant had feared in [msg 2628]: pre-staging under mem_mtx almost always fails because the GPU's 16 GB VRAM is occupied by the worker holding compute_mtx. When pre-staging fails, the code falls back to the old non-prestaged path, which takes 3.3-3.9 seconds per partition instead of the hoped-for 1.8 seconds. The two-lock split provides no benefit and actually adds overhead from the additional mutex contention and fallback logic.

Within a few messages, the assistant abandons Phase 10 entirely. The code is reverted to Phase 9's single-lock approach. A comprehensive benchmark sweep across concurrency levels (c=5 through c=20) reveals that the real bottleneck is not GPU locking at all, but DDR5 memory bandwidth contention between synthesis threads and MSM threads competing for the same memory channels. This insight leads to Phase 11, a completely different optimization strategy targeting memory bandwidth rather than GPU interlock.

Message [msg 2635] thus marks the moment when the Phase 10 approach was still viable—when the code compiled, when testing was about to begin, when the design still had a chance to prove itself. It is the calm before the storm of negative results, the build that succeeded into a dead end. In the narrative of the optimization campaign, it is the turning point where one strategy ends and another, more successful one begins.

Lessons for Optimization Methodology

The story of message [msg 2635] and the Phase 10 failure it enabled teaches several lessons about systematic optimization:

Build early, build often. The assistant could have spent more time analyzing the Phase 10 design theoretically, trying to predict whether it would work. Instead, the approach was to implement, compile, and test—letting empirical measurement, not speculation, drive decisions. The build in [msg 2635] is a commitment to that methodology.

Fail fast is a feature, not a bug. The Phase 10 failure was discovered within minutes of the build completing. The assistant ran a single correctness test, saw 72.5 seconds instead of the expected ~32 seconds, and immediately diagnosed the root cause. The total investment in the failed approach was measured in hours, not days or weeks.

Negative results are valuable data. The Phase 10 failure taught the team something important: CUDA memory management APIs are device-global and cannot be partitioned by user-space mutexes. This knowledge, hard-won through implementation and testing, informed the Phase 11 design and prevented similar mistakes in future optimization phases.

The build is never just a build. Every compilation is a hypothesis test. It tests whether the code is syntactically valid, whether the types are consistent, whether the interfaces align. But it also tests—implicitly—whether the design is worth pursuing. A build that succeeds is not a design that works, but it is a design that can be tested. And testing, as Phase 10 demonstrated, is where the real learning happens.