A Build That Speaks Volumes: The Phase 11 Intervention 3 Milestone

"Build succeeded. Let me also build the bench"

In the relentless pursuit of shaving seconds off Filecoin Groth16 proof generation, a successful build is rarely the end of a story — it is the quiet pivot point where months of analysis, iterative experimentation, and cross-language engineering converge into something measurable. Message [msg 2802] in this opencode session is exactly such a moment. On its surface, it is a two-line status update: the C++/CUDA and Rust code for Phase 11's three memory-bandwidth interventions compiles cleanly, and the benchmark binary follows suit. But beneath this terse report lies a dense web of reasoning, architectural decisions, and debugging that spans dozens of earlier messages and reaches across C++, CUDA, Rust FFI, and the Linux memory subsystem.

This article unpacks that single message — what it meant, why it was written, what decisions it crystallized, and what knowledge it both consumed and produced.

The Road to Phase 11: Memory Bandwidth as the Hidden Bottleneck

To understand why a mere "build succeeded" warrants analysis, one must appreciate the journey that led to it. The SUPRASEAL_C2 Groth16 proof generation pipeline for Filecoin PoRep is a beast: it consumes approximately 200 GiB of peak memory and orchestrates a complex dance between CPU-based synthesis (sparse matrix-vector multiplication, or SpMV) and GPU-accelerated multi-scalar multiplication (MSM) and NTT kernels. Earlier phases of optimization had already addressed GPU interlock contention (Phase 8), PCIe transfer overhead (Phase 9), and a failed two-lock architecture (Phase 10) that was abandoned after discovering fundamental CUDA device-global synchronization conflicts.

Phase 11 emerged from a comprehensive waterfall timing analysis that identified DDR5 memory bandwidth contention as the primary bottleneck. The CPU's SpMV synthesis and the GPU's b_g2_msm computation were fighting over the same memory bus, causing TLB shootdowns, L3 cache thrashing, and utilization dips. The Phase 11 design specification proposed three targeted interventions:

  1. Intervention 1: Serialize async_dealloc calls with a static mutex to prevent concurrent deallocation storms.
  2. Intervention 2: Reduce the groth16_pool thread count from 192 to 32 threads, matching the two-CCD topology of the AMD EPYC CPU.
  3. Intervention 3: Implement a global memory-bandwidth throttle — a cooperative flag that C++ sets before launching b_g2_msm and that Rust's SpMV checks, yielding CPU time to reduce contention. By the time the assistant reached message [msg 2802], Interventions 1 and 2 were already implemented and benchmarked. Intervention 2 alone had delivered a 3.4% improvement (36.7 s/proof vs. 38.0 s baseline), though with an unexpected side effect: b_g2_msm slowed from ~0.5 s to ~1.7 s because the reduced thread pool starved the Pippenger MSM of parallelism. The user had explicitly chosen to press forward with Intervention 3 rather than tuning the thread count further.

Designing the Memory-Bandwidth Throttle

The core challenge of Intervention 3 was architectural: how can C++ code running inside groth16_cuda.cu signal Rust code running in cuzk-pce's spmv_parallel function to temporarily yield? These are separate crates, separate compilation units, and separate thread pools — the synthesis SpMV runs on rayon's par_iter while b_g2_msm runs on a std::thread spawned inside the CUDA code. There is no shared object, no common channel, no existing communication path.

The assistant's initial design in the Phase 11 spec had proposed aliasing a C++ std::atomic<int32_t> through Rust's unsafe pointer dereference. But upon reading the actual code paths in messages [msg 2789] and [msg 2790], the assistant recognized a cleaner approach: use a global static AtomicI32 in the Rust cuzk-pce crate, expose it as an extern "C" function, and call that function from C++. This avoids any unsafe pointer aliasing while achieving the same effect — a single integer in the process's address space that both sides can read and write atomically.

This decision reveals an important design principle: prefer language-native synchronization primitives over cross-language pointer sharing when both sides live in the same process. The Rust AtomicI32 is guaranteed to be lock-free on x86-64, and the extern "C" function call adds only a few cycles of overhead. The C++ side calls set_membw_throttle(1) before entering b_g2_msm and set_membw_throttle(0) after. The Rust side, in its inner SpMV loop, reads the flag and calls std::thread::yield_now() when the throttle is active, voluntarily relinquishing CPU time to reduce memory bus contention.

The Compilation Hurdle

The implementation was not without friction. The assistant initially placed the extern "C" declaration for set_membw_throttle inside a lambda body in groth16_cuda.cu ([msg 2792]). CUDA C++ does not allow extern "C" declarations at block scope — they must appear at file (namespace) scope. The build failed with a cryptic error, and the assistant diagnosed the issue by reading the relevant section of the file ([msg 2798]), then moved the declaration to the top of the file alongside the existing extern "C" functions like create_gpu_mutex and destroy_gpu_mutex ([msg 2799]). The local declaration inside the lambda was then removed ([msg 2800]).

This is a classic FFI pitfall: the C++ compiler accepts extern "C" in many positions, but CUDA's nvcc is stricter, and the interaction between host-device compilation and linkage specifiers creates edge cases. The assistant's fix — consolidating all extern "C" declarations at file scope — is the idiomatic solution and mirrors the existing pattern in the same file.

What the Build Success Actually Means

Message [msg 2802] reports two successful builds:

Build succeeded. Let me also build the bench:
[bash] cargo build --release -p cuzk-bench --no-default-features 2>&1 | tail -5
...
    Finished `release` profile [optimized] target(s) in 0.07s

The first build (cuzk-daemon) is the production binary that links together cuzk-core, cuzk-pce, bellperson, and supraseal-c2 (which compiles the CUDA code). Its success confirms that the set_membw_throttle symbol exported from cuzk-pce resolves at link time against the extern "C" declaration in groth16_cuda.cu. The second build (cuzk-bench) is a separate benchmark harness; its success confirms that the new FFI symbol does not break any downstream consumers.

The 0.07 s build time for the bench binary indicates that only the final linking step was needed — all dependencies were already compiled. This is typical of an incremental build where only the CUDA code and cuzk-pce were recompiled.

Assumptions Embedded in the Design

The throttle design makes several assumptions worth examining:

  1. The throttle flag is read frequently enough to matter. The Rust SpMV loop checks the flag on each iteration. If the loop body is long (processing many rows per iteration), the throttle response may be sluggish. The assistant's implementation in eval.rs places the check in the innermost worker loop, so threads yield within milliseconds of the flag being set.
  2. yield_now() is the right mechanism. Calling yield_now() tells the OS scheduler to reschedule the thread, which reduces CPU time consumed by the yielding thread. However, if all synthesis threads yield simultaneously, the scheduler may immediately reschedule them on the same cores, causing pointless context switches. The assumption is that the OS scheduler will distribute the yielding threads across available cores in a way that leaves room for b_g2_msm's threads.
  3. The throttle is purely cooperative. There is no enforcement mechanism — if the SpMV threads ignore the flag (e.g., because they are in a tight loop that doesn't check), b_g2_msm still contends for memory bandwidth. The design assumes that the SpMV threads are well-behaved and that the cost of checking an atomic flag on each iteration is negligible (~1 ns).
  4. Memory bandwidth contention is symmetric. The throttle assumes that both SpMV and b_g2_msm compete for the same resource (DDR5 bandwidth) and that reducing SpMV's activity will free bandwidth for b_g2_msm. This is true on the AMD EPYC architecture where all cores share the same memory channels, but the degree of contention depends on cache hit rates, NUMA domain boundaries, and the memory access patterns of each workload.

Input and Output Knowledge

To understand this message, a reader needs knowledge of:

The Broader Arc

Message [msg 2802] sits at the boundary between implementation and measurement. The assistant had just completed the third and most architecturally intricate intervention of Phase 11. The next messages would launch the daemon, run benchmarks, and analyze the results — ultimately discovering that Intervention 3 had negligible impact and that the real gains came from Intervention 2 alone. But at this moment, all three interventions are live, the build is green, and the stage is set for the experiment.

This is the rhythm of systems optimization: analyze, design, implement, build, measure, iterate. A successful build is not the goal — it is the prerequisite for the next cycle. The assistant's terse "Build succeeded" masks the hours of reading code, tracing dependency chains, fixing CUDA compilation errors, and reasoning about cross-language synchronization that preceded it. In that sense, message [msg 2802] is a monument to the invisible labor of making complex distributed systems compile correctly.