The Dual-Worker GPU Interlock and the Partition Workers Sweep: Phase 8 of the cuzk SNARK Proving Engine

Introduction

In the relentless pursuit of faster Filecoin proof generation, the cuzk SNARK proving engine had already undergone seven phases of optimization. Each phase targeted a specific bottleneck in a pipeline that spans Go, Rust, and CUDA code, consuming ~200 GiB of peak memory while generating Groth16 proofs for Filecoin's Proof-of-Replication (PoRep) protocol. Phase 7 had introduced per-partition dispatch, which improved throughput but revealed a critical structural inefficiency: a C++ static mutex guarding the entire GPU function, including CPU preprocessing work that did not need exclusive access to the GPU.

Phase 8 was designed to solve this problem. The core insight was deceptively simple: by narrowing the mutex scope to cover only the CUDA kernel region, two GPU workers per device could interleave their work — one performing CPU preprocessing while the other ran CUDA kernels on the GPU. This eliminated the GPU idle gaps that had plagued Phase 7, achieving 100% GPU efficiency in single-proof benchmarks and 13–17% throughput improvement in multi-proof workloads.

But the story does not end with the implementation. Immediately after committing Phase 8, the user requested a systematic sweep of the partition_workers configuration parameter across five values (10, 12, 15, 18, 20). This sweep, executed over roughly 45 messages and spanning several hours of wall time, would reveal that the previously assumed sweet spot of pw=20 was slightly suboptimal, with pw=10 and pw=12 tying for best throughput at 43.5 seconds per proof.

This article examines the full arc of this segment: the reasoning behind the Phase 8 design, the implementation journey across multiple layers of the software stack, the benchmark validation that confirmed the design's correctness, and the disciplined parameter sweep that closed the loop on configuration optimization. Together, these two phases of work represent a complete optimization cycle — from problem diagnosis through design, implementation, validation, and finally parameter tuning.

Part I: The Problem That Phase 8 Solved

The GPU Idle Gap

To understand the significance of Phase 8, one must first understand the problem it solved. The cuzk proving engine generates Groth16 proofs for Filecoin's PoRep protocol. Each proof is divided into 10 partitions, each requiring CPU synthesis (generating circuit constraints from witness data) followed by GPU proving (Number Theoretic Transforms, Multi-Scalar Multiplications, batch additions, and tail MSMs).

In Phase 7, the engine had achieved per-partition dispatch — each partition could be sent to the GPU independently as soon as its synthesis completed. However, a structural bottleneck remained: a C++ static std::mutex in the generate_groth16_proofs_c function in groth16_cuda.cu locked the entire function body, including CPU-side preprocessing steps that did not touch the GPU at all.

The timeline analysis from Phase 7 revealed the cost of this coarse locking. The mutex was held for approximately 3.5 seconds per partition, but only about 2.1 seconds of that was actual CUDA kernel execution. The remaining 1.3 seconds was CPU work — vector splitting, MSM preparation, b_g2_msm computation, and proof assembly epilogue — that could have run concurrently with another partition's GPU execution. The result was predictable GPU idle gaps: between partitions, the GPU sat idle while the next partition's CPU preprocessing completed under the mutex.

The assistant's analysis, documented in the Phase 8 design spec (commit 71f97bc7), quantified the problem with precision. From 110 GPU calls measured across a benchmark run, 87 gaps (80%) were under 50ms, 14 gaps (13%) were between 50–500ms, and 8 gaps (7%) exceeded 500ms. The total gap time was 251.9 seconds against 453.1 seconds of GPU wall time, yielding a GPU efficiency of just 64.3% — meaning the GPU spent over a third of its time idle.

The Core Insight

The Phase 8 design rested on a single, elegant insight: the static mutex did not need to cover the entire function. It only needed to protect the CUDA kernel region — NTT+MSM, batch additions, and tail MSMs — because concurrent CUDA kernel launches from different CPU threads can interfere with each other on the same GPU device. CPU preprocessing (vector splitting, MSM preparation) and the b_g2_msm computation (a CPU-only Pippenger algorithm) are safe to run concurrently because they operate on host memory, not GPU memory.

By narrowing the mutex scope to just the CUDA kernel region, the design enabled a new execution pattern: two GPU workers per device, sharing a per-GPU mutex, interleaving their work. While Worker A holds the mutex and runs CUDA kernels on the GPU, Worker B performs CPU preprocessing for its next partition. When Worker A releases the mutex, Worker B is ready to launch its kernels immediately — no idle gap.

The design document (see [msg 2145]) explored four options for implementing this interlock. Option 1 proposed splitting the C++ function into three phases at the Rust level, but this would require a major refactor of the C++ code. Option 2 proposed passing acquire/release callbacks via FFI, adding callback complexity. Option 3 proposed a C++ internal dual-buffer approach that would be transparent to Rust but was the most complex C++ change. Option 4 — the chosen approach — proposed replacing the static mutex with a passed-in std::mutex* pointer, narrowing the lock scope to just the CUDA kernel region. This was the simplest C++ change, requiring only a few dozen lines of modification across the codebase.

Part II: The Implementation Journey

The C++ CUDA Kernel Refactoring

The heart of the change was in groth16_cuda.cu, where the static mutex was removed and replaced with a std::mutex* parameter passed in from Rust via FFI (see [msg 2151] and [msg 2152]). The lock scope was narrowed from covering the entire generate_groth16_proofs_c function to covering only the region from per-GPU thread launch (where CUDA kernels begin) to per-GPU thread join (where CUDA kernels complete).

A critical detail was the join ordering. In the original code, the prep_msm_thread (which does CPU preprocessing, signals a barrier, then computes b_g2_msm) was joined before the per-GPU threads. In the refactored code, the per-GPU threads are joined first (while still holding the lock), then the lock is released, and then prep_msm_thread is joined. This reordering ensures that b_g2_msm — which runs on the prep_msm_thread — completes without holding the GPU lock, allowing the other worker to acquire the lock and start its CUDA kernels while b_g2_msm is still running.

The design also required careful analysis of the semaphore_t barrier that synchronizes the prep_msm_thread and the per-GPU thread. The assistant verified that the counting semaphore's notify()-before-wait() semantics work correctly when the per-GPU thread is launched after the notification — a critical correctness property that prevented deadlocks.

FFI Plumbing Through Three Layers

Threading the mutex pointer from Rust through to C++ required coordinated changes across three layers of the software stack:

supraseal-c2/src/lib.rs: The FFI wrapper that bridges Rust and C++. New C helper functions create_gpu_mutex and destroy_gpu_mutex were added for allocation, and the generate_groth16_proof wrapper functions were extended to accept and pass the mutex pointer.
extern/bellperson/src/groth16/: The Rust bellperson library, which implements the Groth16 proving system. A GpuMutexPtr type and SendableGpuMutex wrapper were introduced to safely pass the mutex address (as a usize) across thread boundaries. The prove_from_assignments and create_proof_batch_priority_inner functions received the new gpu_mutex parameter.
extern/cuzk/cuzk-core/src/pipeline.rs and engine.rs: The engine orchestration layer. The pipeline was updated to pass the per-GPU mutex through the proof generation call chain, and the engine was extended to spawn gpu_workers_per_device workers (default 2) per GPU, each receiving the same mutex address.

The Send Trait Puzzle

One of the most interesting sub-problems encountered during implementation was a Rust compilation error involving the Send trait. The raw mutex pointer (*const std::ffi::c_void) was being passed across an async boundary — from the async task that orchestrates proof generation to the blocking thread that calls the C++ function. Rust's Send trait requires that types transferred across thread boundaries are safe to move between threads, and a raw pointer does not satisfy this requirement by default.

The assistant solved this by extracting the pointer's address as a usize before the async boundary, then reconstructing the pointer on the other side. This approach, while requiring careful reasoning about safety, was the minimal fix that preserved the existing architecture without requiring a larger refactoring of the async code.

Configuration and Backward Compatibility

A new configuration field gpu_workers_per_device was added to config.rs and documented in cuzk.example.toml, defaulting to 2. For backward compatibility, legacy callers passing null for the mutex pointer would fall back to an internal mutex, ensuring that existing single-worker deployments continued to work without modification.

Part III: Benchmark Validation

Single-Proof GPU Efficiency: 100.0%

The first benchmark was a single-proof test designed to measure GPU efficiency — the ratio of time the GPU is actively executing kernels to total wall time. The assistant extracted GPU_START and GPU_END timestamps from the daemon logs and ran a Python analysis script that computed merged intervals, cross-worker gaps, and per-partition durations.

The result was stunning: 100.0% GPU efficiency (see [msg 2229]). The merged GPU intervals showed zero idle time across all 10 partitions. Cross-worker gaps were measured at just 12–22 milliseconds, but crucially, these gaps were always covered by the other worker's GPU execution. The GPU was never idle.

This result validated the core design thesis: by narrowing the mutex and running two workers per device, CPU preprocessing and GPU kernel execution could be perfectly interleaved, eliminating the idle gaps that had plagued Phase 7. The analysis showed Worker 0 handling partitions p2, p0, p3, p7, p8 with durations of 6474ms, 6696ms, 6525ms, 6628ms, and 6440ms respectively, while Worker 1 handled p4, p5, p1, p9, p6 with similar durations. The interleaving was so tight that the merged GPU intervals covered the entire 36,176ms wall time with zero gaps.

Multi-Proof Throughput Improvement

The multi-proof benchmarks confirmed that the GPU efficiency gains translated to real-world throughput improvements (see [msg 2231]). Using the standard c=5 j=3 configuration (5 proofs with 3 concurrent sectors), Phase 8 achieved 44.0 seconds per proof, compared to 50.7 seconds for Phase 7 — a 13.2% improvement. At c=5 j=2, the improvement was even more pronounced: 49.5 seconds vs 59.8 seconds, a 17.2% gain.

The assistant also tested the boundary condition of partition_workers=30 at the user's request. This test revealed a sharp regression to 60.4 seconds per proof (see [msg 2238]). The diagnosis was precise: with 30 concurrent synthesis workers across 3 sectors, CPU contention starved the GPU preprocessing threads, causing individual partition GPU times to balloon from ~6.5 seconds to 22+ seconds. This confirmed that the sweet spot for the 96-core Zen4 test machine was partition_workers=20 — or so it seemed at the time.

The Commit

The Phase 8 implementation was committed as 2fac031f on the feat/cuzk branch with a detailed commit message that documented the rationale, the file-by-file changes, and the benchmark results. The commit message itself served as a mini-design document, capturing the architectural insight, the implementation approach, and the empirical validation. The assistant followed a disciplined pre-commit ritual: updating the todo list, running git status to verify the working tree, running git diff --stat to inspect the scope of changes (7 files, 195 insertions), and checking the commit history to ensure the design document was the most recent commit. This ritual — the pause before the commit — is the hallmark of an engineer who respects version control as a narrative tool, not just a backup mechanism.

Part IV: The Partition Workers Sweep

The User's Request

Immediately after the Phase 8 commit, the user issued a concise but consequential request at [msg 2248]: "sweep 10,12,15,18,20". This five-word message initiated a systematic investigation into the optimal partition_workers setting. The assistant had previously determined that pw=20 was the sweet spot based on comparisons with pw=30, but the user wanted a more thorough exploration of the parameter space.

The assistant acknowledged the request and created a structured todo list with five sweep points (see [msg 2249]). The methodology was clear: restart the daemon for each configuration value, wait for SRS preload, and run the standard c=5 j=3 benchmark.

The Sweep Protocol

The sweep protocol, established at the beginning of the execution, followed a careful, repeatable procedure for each configuration point:

Kill the running daemon — terminate any existing cuzk-daemon process to ensure a clean state.
Write a fresh configuration file — a complete TOML file at /tmp/cuzk-sweep.toml with the target partition_workers value, keeping all other parameters constant (gpu_workers_per_device=2, synthesis_lookahead=3, synthesis_concurrency=1, devices=[] for auto-detection).
Start the daemon — launch cuzk-daemon in the background via nohup, redirecting output to a per-configuration log file (e.g., /tmp/cuzk-sweep-pw10.log).
Wait for SRS preload — poll the log file for the "cuzk-daemon ready" message, indicating that the multi-gigabyte Structured Reference String had been loaded into GPU memory.
Run the benchmark — execute cuzk-bench batch -t porep --c1 /data/32gbench/c1.json -c 5 -j 3, the standard throughput benchmark used throughout the optimization campaign.
Extract the throughput — compute seconds per proof from the batch summary.
Record and proceed — update the todo list and move to the next configuration. This protocol ensured that each measurement was taken under identical conditions, with the only variable being the number of partition workers. The use of five proofs with three concurrent jobs (c=5 j=3) stressed the system with enough work to measure steady-state throughput while being short enough to run repeatedly.

Operational Challenges: When Automation Meets Reality

The sweep was not executed flawlessly. Several operational hiccups occurred that reveal the challenges of automated benchmarking in real environments.

The Sed That Didn't Stick. During the transition from pw=10 to pw=12, the assistant attempted to use a compound bash command combining pkill, sed, nohup, and echo in a single invocation (see [msg 2255]). The command timed out after 120 seconds, and critically, the sed substitution never executed. The config file still read partition_workers = 10 when the assistant checked. This was a classic operational hazard: compound commands where one component fails silently, leaving the system in an unexpected state.

The assistant's debugging process was instructive. It checked the log file, found it empty, checked the config, discovered the sed hadn't taken effect, and then pivoted to a more robust approach: writing the entire config file directly using a heredoc (see [msg 2263]). This eliminated the fragility of the sed in-place substitution, which depended on the exact string match being present.

The Silent Timeout. At multiple points during the sweep, the daemon startup command timed out after 120 seconds. The log file was empty because the daemon had never started — the nohup command was part of a previous timed-out invocation that never completed. The assistant recovered by checking for the log file, verifying daemon processes, and restarting with separate, individually-timed bash calls.

The Diagnostic Pivot. Each operational failure forced the assistant to pivot into debugging mode. The assistant would check for running processes (pgrep -f cuzk-daemon), inspect the config file (cat /tmp/cuzk-sweep.toml | grep partition_workers), verify the log file exists, and then restart from a known state. This systematic debugging approach — checking file contents, verifying process state, and decomposing complex commands into simpler steps — was essential to completing the sweep.

The key lesson the assistant learned was to separate concerns: kill the daemon in one tool call, update the config in another, and start the daemon in a third. This pattern, adopted after the pw=10→12 transition failure, was used successfully for all subsequent sweep points.

The Five Data Points

The sweep produced a remarkably tight cluster of results. Here is the complete dataset:

| partition_workers | Throughput (s/proof) | Total Time (s) | |---|---|---| | 10 | 43.5 | 217.5 | | 12 | 43.5 | 217.5 | | 15 | 44.8 | 223.9 | | 18 | 43.8 | 219.1 | | 20 | 44.9 | 224.3 |

pw=10 was the first data point, establishing a baseline of 43.5s/proof. The individual proof times ranged from 67.0s to 75.2s (prove time), with queue wait times of 261–1167ms. The assistant recorded this result and immediately proceeded to pw=12.

pw=12 produced an identical result: 43.5s/proof. The prove times (64.6s to 77.0s) were consistent with the pw=10 run, confirming that the GPU kernel execution time was the bottleneck, not CPU-side partition synthesis. The assistant noted this plateau with the comment "Same as pw=10" and moved on.

pw=15 yielded 44.8s/proof — a slight regression of about 3%. The total time was 223.9s, with individual proof wall times ranging from 74.0s to 153.4s. This was the first indication that increasing partition workers beyond 12 was not beneficial.

pw=18 came in at 43.8s/proof, nearly matching the pw=10/12 results. The total time was 219.1s, with prove times averaging 70.2s. This result was close enough to the best performers that it could be considered within the noise, but it still favored the lower settings.

pw=20, the final data point, produced 44.9s/proof — the worst result in the sweep. The total time was 224.3s, with prove times averaging 71.4s. This confirmed that the earlier assumption of pw=20 being the "sweet spot" was slightly off: pw=20 was actually 1.4s/proof slower than pw=12.

What the Sweep Revealed

The performance band was remarkably narrow — just 1.4 seconds separating the best (43.5s) from the worst (44.9s). This tight clustering itself was a finding: it suggested that the system was not highly sensitive to partition_workers in the 10–20 range, at least on this 96-core machine.

The optimal values were pw=10 and pw=12, tied at 43.5 seconds per proof. The previously assumed sweet spot of pw=20 was actually the worst performer in this range (44.9s/proof), though still only 3.2% slower than the optimum. pw=18 followed closely at 43.8 seconds per proof.

These results told a deeper story about the system's architecture. The fact that pw=10 and pw=12 performed identically suggested that below a certain threshold, the number of synthesis workers was not the bottleneck — the GPU pipeline was the limiting factor, and as long as enough partitions were available to keep the GPU fed, additional workers provided no benefit. The slight regression at pw=15 and pw=20 suggested the onset of CPU contention, where additional synthesis threads began competing for the same cores needed by the GPU preprocessing threads.

The narrow performance band also validated the Phase 8 design: by decoupling CPU preprocessing from GPU kernel execution via the dual-worker interlock, the system had become robust to moderate variations in synthesis worker count. The GPU was no longer left idle waiting for partitions, so even suboptimal partition_workers settings achieved reasonable throughput.

Part V: The Engineering Mindset

Systematic Investigation Over Intuition

The partition_workers sweep exemplifies a key principle of performance engineering: measure, don't guess. The assistant could have assumed that pw=20 (the previous "sweet spot") would remain optimal, or that pw=15 (the midpoint) would be best. Instead, it methodically tested five values and discovered that the optimum was actually lower than expected.

Without this sweep, production deployments might have used pw=20 and left 1–2% performance on the table — small for a single proof, but significant when multiplied across thousands of proofs in a Filecoin storage mining operation.

The Value of Negative Results

The sweep also produced valuable negative knowledge. The pw=30 test (conducted before the sweep) had already established an upper bound where CPU contention dominated. The sweep confirmed that the optimal range was 10–12, and that values above 12 offered no benefit and could slightly degrade performance. This negative knowledge is just as important as the positive finding — it prevents future engineers from wasting time exploring higher values.

The Ritual of the Commit

The Phase 8 implementation was committed with a level of ceremony that reflects disciplined engineering practice. Before committing, the assistant:

Updated the todo list marking all tasks as completed
Ran git status to verify the working tree
Ran git diff --stat to inspect the scope of changes (7 files, 195 insertions)
Ran git log --oneline -5 to check the commit history and ensure the design document was the most recent commit This ritual — the pause before the commit — is the hallmark of an engineer who respects version control as a narrative tool, not just a backup mechanism. Each commit should tell a coherent story, and the assistant ensured that the Phase 8 implementation commit would follow logically from the Phase 8 design document that preceded it.

Conclusion

This segment of the conversation captures a complete optimization cycle: from problem diagnosis through design, implementation, validation, and finally parameter tuning. The Phase 8 dual-worker GPU interlock eliminated GPU idle gaps by narrowing a single mutex's scope, achieving 100% GPU efficiency and 13–17% throughput improvement. The partition_workers sweep then refined the configuration, discovering that pw=10–12 was the optimal range for the 96-core test machine.

What makes this segment remarkable is not just the technical achievement — though that is significant — but the engineering discipline it reveals. The assistant worked methodically through a todo list, verified each change with benchmarks, documented negative results alongside positive ones, and treated the commit as a narrative act. The user, in turn, provided clear strategic direction and requested the systematic sweep that closed the loop on configuration optimization.

The result is a body of work that is not just faster, but understood. The team now knows why Phase 8 works, how much it improves throughput, and what the optimal configuration is for their hardware. That knowledge, captured in the conversation and in the git history, is the most valuable output of all.

References

[1] "From Implementation to Optimization: The Phase 8 Dual-Worker GPU Interlock and the Partition Workers Sweep" — [chunk 24.0] article on the full Phase 8 implementation arc

[2] "The Systematic Partition Workers Sweep: Finding the Optimal Configuration for Phase 8's Dual-Worker GPU Interlock" — [chunk 24.1] article on the partition workers sweep

[3] Phase 8 design spec — commit 71f97bc7, documented in [msg 2145]

[4] Phase 8 implementation commit — 2fac031f, documented in [msg 2245]

[5] Single-proof GPU efficiency analysis — [msg 2229]

[6] Multi-proof throughput benchmark — [msg 2231]

[7] Partition workers pw=30 regression test — [msg 2238]

[8] User's sweep request — [msg 2248]

[9] Assistant's sweep todo list — [msg 2249]

[10] Compound command failure and recovery — [msg 2255] through [msg 2263]