The Commit That Closed the GPU Idle Gap: Phase 8 Dual-Worker Interlock

In the high-stakes world of Filecoin proof generation, every millisecond of GPU idleness is wasted capital. The SUPRASEAL_C2 Groth16 proving pipeline, responsible for producing the cryptographic proofs that underpin Filecoin's Proof-of-Replication (PoRep), had been chasing a persistent performance ceiling. Through six optimization phases, the team had progressively reduced memory footprint, improved throughput, and refined the partition dispatch architecture. But a stubborn problem remained: GPU idle gaps. Between successive partition proofs, the GPU sat idle while CPU-side preprocessing completed. The Phase 8 dual-worker GPU interlock, captured in a single git commit at message index 2245, was the surgical strike that eliminated those gaps and delivered a 13-17% throughput improvement.

The Message: A Git Commit as a Technical Milestone

The subject message is deceptively simple on its face: a git commit command. The assistant types:

cd /home/theuser/curio && git add extern/bellperson/src/groth16/mod.rs extern/bellperson/src/groth16/prover/supraseal.rs extern/cuzk/cuzk-core/src/config.rs extern/cuzk/cuzk-core/src/engine.rs extern/cuzk/cuzk-core/src/pipeline.rs extern/cuzk/cuzk.example.toml extern/supraseal-c2/cuda/groth16_cuda.cu && git commit -m "feat(cuzk): Phase 8 — dual-worker GPU interlock

This is not merely a record-keeping action. It is the culmination of an entire session's worth of implementation, debugging, benchmarking, and analysis — a session that spanned daemon restarts, timeline analysis scripts, Python gap calculations, and multiple benchmark runs at different configuration settings. The commit message itself is a miniature technical document, containing the rationale, the architectural change, the file-by-file breakdown, and the benchmark results that validate the approach.

Why This Message Was Written: The GPU Idle Gap Problem

To understand the motivation behind this commit, one must trace back through the optimization journey. Phase 7 (<msg id=2219-2244 context>) had introduced per-partition dispatch, where each of the 10 Groth16 partitions in a PoRep proof was submitted as an independent GPU job. This allowed the GPU to begin working on partitions as soon as they were ready, rather than waiting for all partitions to complete synthesis. However, Phase 7 revealed a critical bottleneck: a C++ static mutex in generate_groth16_proofs_c that serialized all GPU access.

The mutex had been originally placed to protect CUDA kernel launches from concurrent access — a reasonable safety measure. But its scope was too broad. It locked the entire generate_groth16_proofs_c function, including CPU-side preprocessing steps (vector splitting, MSM preparation) and the b_g2_msm computation. This meant that when one worker held the mutex to run CUDA kernels, the other worker could not even begin its CPU preprocessing. The GPU would finish its kernels, release the mutex, and then sit idle while the next worker's CPU preprocessing ran. The timeline analysis from Phase 7 showed GPU efficiency hovering around 85-90%, with idle gaps of hundreds of milliseconds between partitions.

The insight that drove Phase 8 was elegantly simple: narrow the mutex to cover only the CUDA kernel region — the NTT+MSM operations, batch additions, and tail MSMs — and let CPU preprocessing and b_g2_msm run outside the lock. This would allow two GPU workers per device to interleave their work: one worker runs CPU preprocessing while the other holds the mutex for CUDA kernels, and vice versa.## How the Decision Was Made: From Analysis to Implementation

The decision to narrow the mutex was not made in a vacuum. It emerged from careful analysis of the Phase 7 timeline data. In the messages preceding this commit (<msg id=2227-2230>), the assistant had extracted the GPU_START and GPU_END timestamps from the daemon logs and run a Python analysis script that computed merged intervals, cross-worker gaps, and per-partition durations. The Phase 7 data showed that while per-partition GPU kernel time was only ~3.3 seconds, the total gpu_prove function time was ~6.5 seconds — meaning ~3.2 seconds per partition was spent on CPU preprocessing that could be overlapped.

The assistant's reasoning, visible in the thinking behind the tool calls, was methodical. First, it confirmed the dual-worker architecture was working by checking that partitions alternated between worker 0 and worker 1. Then it computed the merged GPU intervals and found that in the Phase 8 single-proof test, GPU efficiency had already reached 100.0% — zero idle time. But the per-partition GPU time had increased from ~3.3s to ~6.4-6.7s, which the assistant correctly diagnosed as the full function time now being measured, not just the CUDA kernel region. The actual CUDA kernel time was still ~3.3s, confirmed by the CUZK_TIMING log lines showing gpu_total_ms=3165-3372.

This diagnostic reasoning was critical. Without it, the increased per-partition time could have been mistaken for a regression. Instead, the assistant recognized it as a measurement artifact — the gpu_prove function was now timing the entire C++ call including CPU preprocessing, whereas previously the mutex had hidden that time inside the locked region. The real win was that the two workers could now overlap: while worker 0 ran CUDA kernels on the GPU, worker 1 could simultaneously run CPU preprocessing for its next partition.

Assumptions Made by the Assistant

Several assumptions underpin this commit. The first is that the CUDA kernel region is safe to execute concurrently with CPU-side vector operations on the same GPU. This is a reasonable assumption — modern CUDA streams and GPU hardware support asynchronous kernel launches, and the CPU preprocessing (splitting vectors, preparing MSM inputs) operates on host memory, not GPU memory. However, the assistant implicitly trusts that the CUDA driver and the specific GPU (RTX 5070 Ti) can handle the interleaving pattern without resource contention or memory corruption.

A second assumption is that two workers per GPU is the right default. The commit message notes gpu_workers_per_device defaults to 2, and the config was set to 2 for all benchmarks. The assistant assumes that more than 2 workers would not help because the GPU can only execute one kernel at a time per device, and the CPU preprocessing is fast enough that two workers can keep the GPU fully saturated. The partition_workers=30 experiment (<msg id=2238-2239>) validated this assumption indirectly — too many workers caused CPU contention that actually regressed performance.

A third assumption is that legacy callers passing null for the mutex pointer will fall back to an internal mutex, maintaining backward compatibility. This is a pragmatic engineering decision, but it assumes that no legacy caller is running in a multi-worker context where the fallback mutex would cause serialization. For the existing single-worker use cases, this is safe.

Mistakes and Incorrect Assumptions

The most notable mistake was not in the commit itself but in the operational execution surrounding it. When the user asked to test partition_workers=30 ([msg 2233]), the assistant encountered a sed substitution that failed to persist the config change, requiring a manual rewrite of the config file and a daemon restart in separate steps. This was a workflow hiccup rather than a design error, but it highlights the complexity of managing a distributed proving system where configuration changes require coordinated restarts.

A more subtle issue is the assumption that the optimal partition_workers setting is 20. The commit message reports benchmark results at partition_workers=20, but the subsequent sweep in the next chunk (<msg id=2246+>) would reveal that partition_workers=10-12 actually achieves better throughput (43.5s/proof vs 44.9s/proof at pw=20). The commit's benchmark numbers are accurate for pw=20, but they represent a slightly suboptimal configuration. This is not a mistake per se — the assistant was reporting what it had measured — but it means the "13.2% improvement" figure is conservative relative to the best possible configuration.

Input Knowledge Required to Understand This Message

To fully grasp this commit, one needs substantial context about the SUPRASEAL_C2 pipeline. The Groth16 proof for Filecoin PoRep involves 10 partitions, each requiring CPU synthesis (constraint evaluation to produce a/b/c vectors) followed by GPU proving (NTT, MSM, batch additions). The pipeline had evolved through seven prior phases: from the original monolithic proof generation, through slotted partition pipelines (Phase 6), parallel synthesis with semaphores (Phase 6.5), per-partition dispatch (Phase 7), and now the dual-worker interlock (Phase 8).

One also needs to understand the FFI chain: Go Curio calls Rust supraseal-c2, which calls Rust bellperson, which calls C++ CUDA code via FFI. The mutex had lived in the C++ layer (groth16_cuda.cu), and threading it through the Rust layers required adding a gpu_mutex parameter to the FFI declaration, wrapper functions, and the prove_from_assignments call chain. The SendableGpuMutex wrapper in supraseal.rs was necessary because Rust's Send trait requires proof that a type can be safely transferred between threads — a raw pointer to a C++ mutex does not satisfy this without explicit wrapping.

Output Knowledge Created by This Message

This commit produces several forms of knowledge. First, it creates a reproducible benchmark result: on an RTX 5070 Ti with a 96-core AMD Zen4 CPU, the Phase 8 dual-worker interlock achieves 44.0s/proof at c=5 j=3 with partition_workers=20, representing a 13.2% improvement over Phase 7. The single-proof GPU efficiency of 100.0% is a particularly strong result — it demonstrates that the interlock pattern can completely eliminate GPU idle gaps.

Second, the commit establishes a new architectural pattern for GPU proving engines: the narrow-scope mutex that protects only the CUDA kernel region while allowing CPU preprocessing to proceed concurrently. This pattern is generalizable beyond Filecoin PoRep to any GPU workload where CPU preprocessing and GPU kernel execution can be overlapped.

Third, the commit documents the negative result at partition_workers=30, which regressed to 60.4s/proof. This is valuable negative knowledge — it confirms that there is a sweet spot for partition worker count, and that exceeding it causes CPU contention that starves the GPU preprocessing threads. The subsequent sweep in the next chunk would refine this to pw=10-12 as the optimal range.

The Thinking Process Visible in the Reasoning

The assistant's reasoning is most visible in the commit message itself, which is structured as a mini-design document. It opens with the core insight: "Narrow the C++ static mutex in generate_groth16_proofs_c to cover only the CUDA kernel region." This is the thesis statement. Then it enumerates the file-by-file changes, each with a specific rationale. The groth16_cuda.cu change is described with precise lock/unlock placement: "acquire lock before per-GPU thread launch, release after per-GPU join (before prep_msm_thread join)." This level of detail reveals a deep understanding of the CUDA thread hierarchy and the need to avoid deadlocks with the prep_msm thread.

The benchmark section is presented as evidence, not just numbers. The assistant compares Phase 8 against Phase 7 at two concurrency levels (j=3 and j=2), showing consistent improvement. The negative result at pw=30 is included honestly, with a clear explanation: "CPU contention from 30 simultaneous synthesis workers starving GPU preprocessing." This is the hallmark of good engineering communication — presenting failures alongside successes to build a complete picture.

The commit also reveals the assistant's awareness of backward compatibility. Legacy callers passing null for the mutex pointer get a fallback internal mutex, ensuring that existing single-worker deployments continue to work without modification. This is a thoughtful touch that demonstrates production-minded engineering.

Conclusion

The Phase 8 commit at message 2245 is more than a git commit — it is the culmination of a deep optimization journey that required understanding CUDA thread synchronization, Rust FFI safety, Go-to-Rust-to-C++ call chains, and the economics of GPU rental markets. By narrowing a single mutex's scope, the assistant eliminated the GPU idle gap that had persisted through seven prior optimization phases. The 13-17% throughput improvement is real, but the deeper achievement is the architectural insight: sometimes the biggest performance gains come not from adding more hardware or parallelism, but from removing unnecessary serialization that hides behind a seemingly innocuous mutex.