The Moment of Truth: Launching Phase 10's Two-Lock Architecture

In the relentless pursuit of GPU throughput for Filecoin PoRep proof generation, every optimization eventually meets reality. Message [msg 2610] captures that exact collision — the first launch of a newly implemented two-lock architecture designed to break through a CPU memory bandwidth bottleneck, only to discover that hardware constraints do not respect clean software abstractions.

The Message: A Deceptively Simple Launch

The subject message is, on its surface, mundane:

[assistant] [bash] /home/theuser/curio/extern/cuzk/target/release/cuzk-daemon --config /tmp/cuzk-p10-gw3.toml > /home/theuser/cuzk-p10-daemon.log 2>&1 &
echo "PID=$!"
sleep 30 && tail -5 /home/theuser/cuzk-p10-daemon.log && pgrep -a cuzk-daemon
PID=3081693
2026-02-19T17:07:27.254232Z  INFO cuzk_daemon: cuzk-daemon ready, serving on 0.0.0.0:9820
2026-02-19T17:07:27.254266Z  INFO cuzk_core::engine: synthesis dispatcher started max_batch_size=1 max_batch_wait_ms=10000 slot_size=0 synthesis_concurrency=1 partition_workers=10
2026-02-19T17:07:27.254303Z  INFO cuzk_core::engine...

A bash command, a process ID, a few log lines confirming the daemon is alive. But this message is the culmination of an entire optimization phase — Phase 10 of a multi-week investigation into the SUPRASEAL_C2 Groth16 proof generation pipeline. The daemon being launched carries the first implementation of a carefully designed two-lock architecture intended to solve a bottleneck that had been identified and characterized over the preceding nine phases of optimization work.

Why This Message Was Written: The Context of Phase 10

To understand why this message exists, one must trace the optimization journey that led here. The preceding Phase 9 had implemented PCIe transfer optimization and achieved a 14.2% throughput improvement in single-worker mode. But benchmarking revealed a critical bottleneck shift: the CPU critical path — specifically prep_msm (1.9s) and b_g2_msm (0.48s) — now dominated the per-partition wall time at approximately 2.4 seconds, leaving the GPU idle for roughly 600ms per partition waiting for the CPU thread. At high concurrency, the ten synthesis workers competed with the CPU MSM operations for the 8-channel DDR5 memory bandwidth, inflating CPU times by 2–12×. The bottleneck had moved from PCIe transfers and GPU kernel execution to CPU memory bandwidth contention.

The user and assistant designed Phase 10 to address this: a two-lock design that splits the single std::mutex protecting GPU access into two separate mutexes — mem_mtx for VRAM allocation and pre-staging upload (a short ~18ms operation), and compute_mtx for GPU kernel execution (the long ~1.8s operation). The key insight was that by releasing mem_mtx before entering compute_mtx, another worker could begin its VRAM pre-staging while the first worker's kernels were still running. With three workers per GPU device (increased from two), the hope was to hide the CPU overhead entirely, reducing per-partition wall time from ~3.7s to an estimated ~1.8–2.0s — a projected 30–38% throughput improvement.

The assistant had spent the preceding messages ([msg 2589] through [msg 2608]) implementing this design. A gpu_locks struct was defined in groth16_cuda.cu, the FFI interface was updated to pass an opaque pointer, the lock regions were restructured, and a build error (a type-casting issue where nvcc rejected casting from std::mutex* to gpu_locks*) was fixed. The build succeeded. A configuration file /tmp/cuzk-p10-gw3.toml was written with gpu_workers_per_device = 3. Message [msg 2610] is the first deployment of this newly built daemon — the moment of truth.

Assumptions Embedded in the Launch

This message, and the implementation it launches, rests on several critical assumptions:

First, the assumption that memory management operations on a CUDA device can be cleanly separated from compute operations. The two-lock design depends on the idea that mem_mtx can be held briefly for VRAM allocation and upload, then released, allowing another worker to acquire it while the first worker runs kernels under compute_mtx. This assumes that cudaMalloc, cudaMemPoolTrimTo, and the async upload operations inside mem_mtx do not implicitly synchronize with or depend on the device state affected by operations inside compute_mtx.

Second, the assumption that cudaDeviceSynchronize could be safely removed from the pre-staging path. The Phase 9 code had called cudaDeviceSynchronize before pool trimming to ensure all prior stream-ordered allocations were resolved. The Phase 10 design removed this sync, reasoning that pool trim alone would suffice because by the time compute_mtx was released by the previous worker, all stream ops were synced. This was a calculated risk, documented in the implementation plan's risk assessment table with "Medium" likelihood.

Third, the assumption that three workers per device would not exhaust VRAM. Each worker's pre-staging path allocates approximately 12 GiB (4 GiB for d_a plus 8 GiB for d_bc). With a single NVIDIA RTX A6000 (48 GiB VRAM), three workers would need 36 GiB simultaneously — feasible, but only if the allocations are properly serialized and freed between workers. The design relied on the gpu_ptr_t destructor freeing d_a via cudaFree inside the per-GPU thread (under compute_mtx), and d_bc being freed early at line 815 of the original code, ensuring VRAM was released before compute_mtx was unlocked.

Fourth, the assumption that the Rust-side FFI layer needed no changes. The opaque pointer approach — where create_gpu_mutex returns void* and the C++ side casts it to gpu_locks* — was designed to keep the Rust bindings unchanged. This was correct in principle but introduced the type-casting error that had to be fixed during implementation.

The Thinking Process Visible in the Message

While the message itself is a simple bash command, its structure reveals the assistant's thinking. The command starts the daemon in the background, captures the PID, waits 30 seconds for initialization, then checks the log and confirms the process is running. This is a careful, methodical approach: launch, wait, verify. The assistant is not assuming the daemon will start correctly — it explicitly checks.

The log output shown in the message confirms the daemon is ready: "cuzk-daemon ready, serving on 0.0.0.0:9820". The synthesis dispatcher started with partition_workers=10 (unchanged from Phase 9). The pipeline GPU workers are being created. Everything looks clean.

But the real story lies in what happens next. The very next message ([msg 2611]) runs the correctness test and gets a FAILED result with a 41.1s prove time — far worse than the Phase 9 baseline. The daemon launched in this message is about to reveal a fundamental flaw in the two-lock design.

The Hidden Flaw: Device-Global Synchronization

The debugging that follows ([msg 2612] through [msg 2620]) reveals the root cause. The sequence of events is:

  1. Worker 0 acquires mem_mtx, successfully pre-stages 12 GiB (taking ~1.3s for cudaMalloc), releases mem_mtx.
  2. Worker 1 acquires mem_mtx, sees only 1.5 GiB free (Worker 0's 12 GiB is still live on the GPU), falls back to skip_vram, releases mem_mtx.
  3. Worker 2 similarly fails pre-staging.
  4. Worker 0 acquires compute_mtx, runs kernels.
  5. Worker 1 enters compute_mtx (after Worker 0 releases it), tries the fallback path, and hits OOM because the pool hasn't been trimmed. The critical insight is that cudaDeviceSynchronize and cudaMemPoolTrimTo are device-global operations. When Worker 1 calls cudaDeviceSynchronize inside mem_mtx, it blocks until Worker 0's kernels (running under compute_mtx) complete. This effectively serializes the two locks, destroying the intended overlap. Furthermore, even after synchronization, the pool trim may not reclaim memory that was allocated with cudaMalloc (synchronous) rather than cudaMallocAsync (stream-ordered), because the pre-staging path uses synchronous cudaMalloc for the large allocations. The assistant's initial fix — adding cudaDeviceSynchronize back into mem_mtx — actually makes things worse, because it forces the mem_mtx holder to wait for the compute_mtx holder's kernels to finish. The two locks become effectively one, with the added overhead of device-wide synchronization.

What This Message Teaches

Message [msg 2610] is a perfect example of the gap between software abstraction and hardware reality. The two-lock design was elegant in theory: separate the short memory-management operations from the long compute operations, allow pipelining between workers. But CUDA's device model does not support this separation. Memory management operations like cudaMalloc, cudaMemPoolTrimTo, and especially cudaDeviceSynchronize are device-global — they interact with all streams, all contexts, all allocations on the device. You cannot have one thread managing memory while another runs kernels on the same device without implicit synchronization points.

The input knowledge required to understand this message includes: the Groth16 proof generation pipeline for Filecoin PoRep, the CUDA programming model (streams, pools, synchronous vs. async allocation), the C++/Rust FFI boundary, the mutex-based worker dispatch architecture, and the specific timing characteristics of each pipeline stage (pre-staging ~18ms, GPU kernels ~1.8s, CPU MSM ~2.4s). The output knowledge created by this message and its aftermath is a deep understanding of the limitations of lock-based GPU pipelining on a single device — a lesson that will inform the design of Phase 11 and beyond.

The daemon launched in this message runs, but it does not perform. The beautiful abstraction of two independent locks collides with the monolithic reality of a CUDA device, where memory and compute are not separable concerns. This is the moment where theory meets practice, and practice wins.