The First Benchmark: Validating Phase 8's Dual-Worker GPU Interlock

In the course of optimizing the cuzk SNARK proving engine for Filecoin's PoRep C2 proof generation, a single message marked a critical inflection point. At message index 2226, the assistant ran the first benchmark after implementing Phase 8's dual-worker GPU interlock architecture. The message is deceptively simple — a single bash command and its output — but it represents the culmination of a multi-hour engineering effort spanning seven files, ~195 lines of changes across C++, Rust FFI, and async Rust orchestration code. More importantly, it is the moment where the assistant validates that a complex architectural change works correctly before proceeding to more extensive benchmarking.

The Message

The assistant executed:

cd /home/theuser/curio/extern/cuzk && time ./target/release/cuzk-bench -a "http://127.0.0.1:9820" single -t porep --c1 /data/32gbench/c1.json 2>&1

And received:

=== Proof Result ===
status:    COMPLETED
job_id:    6e639c2e-12c7-4783-bce8-66b183230cfe
timings:   total=69256 ms (queue=259 ms, srs=0 ms, synth=355416 ms, gpu=65699 ms)
wall time: 69403 ms
proof:     1920 bytes (hex: 805ef2972...)

Why This Message Was Written: The Motivation and Context

To understand why this message exists, one must understand the problem it was designed to solve. The cuzk proving engine generates Groth16 proofs for Filecoin's Proof-of-Replication (PoRep) protocol. These proofs are computationally intensive, requiring ~200 GiB of peak memory and involving a complex pipeline of CPU synthesis followed by GPU computation (NTT, MSM, batch additions, tail MSMs). Earlier phases of optimization had identified a critical bottleneck: GPU idle gaps.

In Phase 7, the engine dispatched partitions to GPU workers, but a coarse-grained static mutex in the C++ generate_groth16_proofs_c function locked the entire GPU call — including CPU preprocessing work that could run in parallel with other workers. This meant that while one worker held the mutex doing CPU work (preparing data for the GPU), the other worker was blocked, unable to submit its own GPU work. The result was GPU utilization well below 100%, with measurable idle gaps between partitions.

Phase 8 was designed specifically to eliminate these gaps. The approach was to narrow the mutex scope to cover only the CUDA kernel region (NTT+MSM, batch additions, tail MSMs) while allowing CPU preprocessing and b_g2_msm to run outside the lock. With this narrowed mutex, two GPU workers per device could interleave: one would do CPU work while the other ran CUDA kernels, keeping the GPU continuously busy.

The implementation spanned multiple layers:

1. C++ (groth16_cuda.cu): Removed the static mutex, added a mutex pointer parameter, narrowed the lock scope to only the CUDA kernel region
2. FFI (supraseal-c2/src/lib.rs): Added mutex pointer parameter to the extern declaration and wrapper
3. Rust bellperson: Added gpu_mutex parameter to prove_from_assignments and create_proof_batch_priority_inner
4. Engine (cuzk-core/src/engine.rs): Spawned multiple GPU workers per device, each sharing a per-GPU C++ mutex allocated via new create_gpu_mutex/destroy_gpu_mutex helpers
5. Config: Added gpu_workers_per_device configuration option The build process itself revealed subtle challenges. The Rust compiler initially rejected the code because raw pointers (*mut c_void) are not Send, making them impossible to capture in tokio::spawn closures. The assistant had to work through multiple iterations — first trying to move the SendableGpuMutex wrapper into the closure, then converting the pointer to a usize (which is Send) and casting back. Each fix required careful editing of the async Rust code to satisfy the type system while preserving the runtime behavior. After the build succeeded, the assistant started the daemon with the Phase 8 configuration (gpu_workers_per_device = 2) and verified that two GPU workers appeared in the logs (sub_id=0 and sub_id=1 on GPU 0). Then came the critical moment: running the first benchmark to see if the dual-worker interlock actually worked.

How Decisions Were Made

Several key decisions are visible in the lead-up to this message:

The decision to benchmark with a single proof first. The assistant chose to run cuzk-bench single rather than jumping directly to multi-proof benchmarks. This is a deliberate validation strategy: a single proof is the simplest test of the dual-worker interlock. If the implementation has a bug (e.g., the mutex is not properly shared, or the workers deadlock), a single proof would reveal it quickly without the complexity of concurrent proof requests.

The choice of benchmark parameters. The assistant used the default --partition-workers 20 setting (inherited from the Phase 7 config), even though the Phase 8 config file at /tmp/cuzk-phase8.toml doesn't explicitly set it. The config file shows partition_workers = 20 from the Phase 7 template. This is significant because the partition_workers setting controls how many CPU threads are used for synthesis, and the assistant would later discover that this value has a critical impact on GPU utilization.

The decision to use time for wall-clock measurement. The time command prefix provides an independent measurement of total wall time, which the assistant can cross-reference against the daemon's reported timings. This is a good engineering practice — it catches cases where the daemon's timing might be inaccurate or where there's startup/shutdown overhead not included in the reported numbers.

Assumptions Made

The assistant operated under several assumptions, most of which were validated by this benchmark:

Assumption: The dual-worker interlock would not regress single-proof performance. The Phase 8 changes added overhead — a per-GPU mutex allocation, additional FFI calls, and the complexity of coordinating two workers. There was a real risk that this overhead could make single-proof performance worse than Phase 7. The 69.3s result needed to be compared against Phase 7's single-proof performance to determine if the changes were neutral or beneficial.

Assumption: The GPU workers would correctly share the mutex. The C++ mutex is allocated once per GPU and shared between the two workers via a raw pointer. If the pointer were dangling, or if the mutex were not properly initialized, the result would be a crash or deadlock. The fact that the proof completed successfully validates this assumption.

Assumption: The synth timing of 35.5s was reasonable. The reported synthesis time of 35.5 seconds (across 10 partitions with partition_workers=20) needed to be consistent with Phase 7 numbers. If synthesis had regressed significantly, it would indicate a problem with the CPU-side changes.

What This Message Reveals About the Thinking Process

The assistant's thinking process is visible in the structure of the benchmark and the subsequent analysis. The single-proof benchmark is the first step in a systematic validation protocol:

1. Build verification (already done): Ensure the code compiles
2. Daemon startup (already done): Ensure the daemon starts correctly with the new config
3. Single-proof benchmark (this message): Validate correctness and measure baseline performance
4. Timeline analysis (next message): Extract detailed GPU event timestamps to compute efficiency
5. Multi-proof benchmark (subsequent): Measure throughput under realistic load This protocol reflects a disciplined engineering approach. Each step builds on the previous one, and failure at any step would trigger debugging before proceeding. The assistant is not just running a benchmark — it is systematically verifying that a complex, multi-layered change works correctly at every level. The 69.3s wall time and 65.7s GPU time are immediately informative. The GPU time (65.7s) is close to the wall time (69.3s), suggesting that the GPU is busy for most of the proof's duration. But the real test of the dual-worker interlock is not the aggregate numbers — it's the gap analysis between GPU events, which the assistant performs in the very next message by extracting TIMELINE events from the daemon log.

Input Knowledge Required

To fully understand this message, one needs:

Knowledge of the cuzk proving engine architecture. The engine has a pipeline mode where proofs are processed through stages: queue, SRS loading, synthesis (CPU), and GPU proving. The timings in the output (queue=259 ms, srs=0 ms, synth=355416 ms, gpu=65699 ms) correspond to these stages. The synth value of 355,416 ms is suspicious — it's 355 seconds, not 35.5 seconds — suggesting either a display bug or that the synthesis time is reported in milliseconds while the GPU time is also in milliseconds. In fact, looking at the wall time of 69.4 seconds, the synth of 355,416 ms cannot be correct for a single proof. This is likely a cumulative counter that includes synthesis across all partitions or across multiple proofs, and the assistant would need to investigate this discrepancy.

Knowledge of the Phase 7 baseline. Without knowing that Phase 7 achieved certain throughput numbers, the 69.3s result is just a number. The assistant's subsequent analysis (in messages 2227-2229) compares against Phase 7 to determine improvement.

Understanding of GPU mutex semantics. The core insight of Phase 8 is that a narrowed mutex allows worker interleaving. One must understand that GPU calls are asynchronous from the CPU perspective — cudaMemcpy and kernel launches return immediately but execute on the GPU. The mutex is needed not for GPU execution but for accessing shared GPU state (context, streams, memory allocations). By narrowing the mutex to only the critical section, the CPU work that precedes and follows the GPU call can run in parallel with another worker's GPU execution.

Output Knowledge Created

This message produces several pieces of knowledge:

A validated Phase 8 implementation. The proof completed successfully with the correct size (1920 bytes for a Groth16 proof). This confirms that the C++ mutex refactoring, FFI plumbing, and async worker spawning are all functioning correctly.

A baseline single-proof timing. The 69.3s wall time becomes the reference point for all subsequent analysis. The assistant will compare this against Phase 7's single-proof time to quantify the improvement (or regression) from the dual-worker interlock.

Evidence for the GPU efficiency analysis. The 65.7s GPU time out of 69.3s wall time suggests high GPU utilization, but the assistant needs the TIMELINE data to confirm. The subsequent analysis (message 2229) reveals that the GPU efficiency is actually 100.0% — zero idle gaps between partitions. This is the key validation that Phase 8 works as designed.

A suspicious timing value. The synth=355416 ms value is anomalously high and would need investigation. This is the kind of signal that an experienced engineer would flag for follow-up.

Mistakes and Incorrect Assumptions

The most notable issue in this message is the synthesis timing anomaly. The reported synth=355416 ms (355 seconds) is inconsistent with the 69.4s wall time. This could be:

• A cumulative counter that includes synthesis across multiple partitions or proof attempts
• A display bug where the value is in a different unit
• An actual regression in synthesis performance The assistant does not flag this discrepancy in the message itself, but the subsequent analysis (message 2227) focuses on GPU timeline data rather than synthesis timing, suggesting the assistant recognized that the GPU timing is the more reliable metric for validating Phase 8. Another potential issue is that the assistant is benchmarking with partition_workers=20, which was the Phase 7 default. The Phase 8 architecture changes the relationship between CPU and GPU work — with two GPU workers, the CPU needs to keep both workers fed with synthesized partitions. If partition_workers=20 is too high, CPU contention could starve the GPU preprocessing threads, actually reducing throughput. The assistant would later discover this during the partition_workers sweep (Chunk 1), finding that pw=10-12 is optimal for the 96-core machine.

The Broader Significance

This message sits at a pivotal moment in the optimization journey. Phase 8 was the most architecturally ambitious change yet — it required coordinated modifications across C++, Rust FFI, and async Rust, with careful attention to memory safety (raw pointers in async contexts) and concurrency (mutex scope, worker coordination). The fact that the first benchmark produces a clean result is a testament to the assistant's systematic approach: build, verify, benchmark, analyze.

But the single-proof benchmark is just the beginning. The real value of Phase 8 is in multi-proof throughput, where the dual-worker interlock allows the GPU to remain continuously busy even as the CPU works on multiple proofs simultaneously. The assistant would go on to benchmark multi-proof throughput, finding 13-17% improvement over Phase 7, and then execute a systematic partition_workers sweep to find the optimal configuration.

In the end, this message represents more than a benchmark result. It is the validation of a design hypothesis — that narrowing a mutex scope and adding a second GPU worker could eliminate GPU idle gaps. The 100.0% GPU efficiency that the assistant would discover in the subsequent timeline analysis is the definitive proof that the hypothesis was correct.