The Five-Number Sweep: Finding the Optimal partition_workers Setting

Subject Message: [user] sweep 10,12,15,18,20 — Message 2248

At first glance, a user message containing only five comma-separated numbers appears almost trivial: "sweep 10,12,15,18,20." It is five words, four punctuation marks, and a handful of digits. Yet in the context of a months-long optimization campaign for the SUPRASEAL_C2 Groth16 proof generation pipeline — a system with a peak memory footprint of ~200 GiB and a call chain stretching from Go orchestration through Rust FFI into C++/CUDA kernels — this single sentence represents a pivotal moment of empirical rigor. It is the culmination of Phase 8, the dual-worker GPU interlock, and the final act of a systematic search for the optimal configuration parameter that governs how many partition synthesis workers compete for CPU resources.

The Context: Phase 8 Delivers, But Questions Remain

The message arrives immediately after a dramatic series of events. The assistant had just implemented and committed Phase 8 — the dual-worker GPU interlock — across seven files and ~195 lines of C++, Rust, and CUDA code. The core insight of Phase 8 was elegant: narrow the C++ static mutex in generate_groth16_proofs_c to cover only the CUDA kernel region (NTT+MSM, batch additions, tail MSMs), allowing CPU preprocessing and b_g2_msm to run outside the lock. This enabled two GPU workers per device to interleave their work — one performing CPU preprocessing while the other ran CUDA kernels — achieving a perfect 100.0% GPU efficiency in single-proof benchmarks.

The throughput results were impressive: a 13.2% improvement over Phase 7 at concurrency level 3 (44.0s/proof vs 50.7s), and a 17.2% improvement at concurrency level 2 (49.5s vs 59.8s). The commit message, recorded at 2fac031f on the feat/cuzk branch, declared partition_workers=20 as "the sweet spot for this 96-core machine."

But this conclusion was based on only two data points: pw=20 (44.0s/proof) and pw=30 (60.4s/proof). The pw=30 experiment, requested by the user in the preceding message ([msg 2233]), had revealed a sharp cliff: with 30 concurrent partition workers across three sectors, CPU contention starved the GPU preprocessing threads, causing the first partition's GPU time to balloon from ~6.5s to 22-50s. The regression was clear, but the optimal value remained uncertain. Was pw=20 truly optimal, or was it merely the best among the values tested? Could lower values — 10, 12, 15, 18 — perform even better? The user's message is the logical next step: a systematic sweep to map the performance curve.

The Message as a Specification

The message "sweep 10,12,15,18,20" is remarkable for its concision. It communicates, in five words, a complete experimental protocol:

• Action: "sweep" — run a parameter sweep, varying one configuration variable across specified values while holding all other conditions constant.
• Parameter: partition_workers — the number of concurrent synthesis workers. This is implicit, derived from the conversation's immediate history where partition_workers was the variable being tuned.
• Values: 10, 12, 15, 18, 20 — a carefully chosen sequence that brackets the suspected optimum (around 20) but extends lower to test whether fewer workers might reduce contention without starving the pipeline.
• Methodology: Use the established benchmark protocol — the cuzk-bench batch command with c=5 j=3 (5 proofs, concurrency 3), restarting the daemon for each configuration. This too is implicit, established by the preceding benchmarks at pw=20 and pw=30. The user does not need to specify "restart the daemon each time" or "use the same C1 input file" or "wait for SRS preload before benchmarking." These are shared conventions, built over the course of the optimization campaign. The message is a masterful example of efficient communication between domain experts who share deep context.

Assumptions Embedded in the Request

The user's request rests on several assumptions, most of which are well-founded. First, that the partition_workers parameter is the dominant variable affecting throughput at this stage — that Phase 8's dual-worker interlock has eliminated GPU idle gaps as a bottleneck, and that CPU-side synthesis contention is now the primary constraint. This assumption is supported by the pw=30 regression data, which showed CPU starvation directly causing GPU time inflation.

Second, the user assumes the benchmark is reproducible enough that differences of a few percent between configurations are meaningful. The sweep values are spaced closely (10, 12, 15, 18, 20), suggesting the user expects the optimal value to lie within a narrow band and wants to distinguish between, say, 43.5s/proof and 44.9s/proof.

Third, the user implicitly assumes that the optimal value lies within the 10-20 range, not below 10 or above 20. The pw=30 data already rules out higher values, and the reasoning seems to be: if 20 workers cause some contention (as evidenced by the slight regression at pw=30), then reducing workers might reduce contention, but reducing too far might starve the pipeline of synthesized partitions. The sweep is designed to find the Goldilocks zone.

The Execution: Operational Challenges and Empirical Results

The assistant's execution of the sweep, spanning messages [msg 2249] through [msg 2294], reveals the messy reality of empirical systems work. The assistant initially attempted to write a helper script but then executed the sweep manually, restarting the daemon for each configuration. Several operational hiccups occurred: a sed substitution failed to persist because it was bundled with a pkill command that timed out; a daemon failed to start because the nohup command was lost in a timeout; the config file had to be rewritten from scratch rather than patched. These are not failures — they are the normal friction of operating a complex distributed system, and the assistant's methodical debugging (checking the config file, verifying the daemon process, restarting in separate steps) demonstrates a robust operational workflow.

The results were illuminating:

| partition_workers | Throughput (s/proof) | |---|---| | 10 | 43.5 | | 12 | 43.5 | | 15 | 44.8 | | 18 | 43.8 | | 20 | 44.9 |

The performance band was remarkably narrow — just 1.4 seconds separating the best (43.5s) from the worst (44.9s). This tight clustering itself is a finding: it suggests that the system is not highly sensitive to partition_workers in the 10-20 range, at least on this 96-core machine. The optimal values were pw=10 and pw=12, tied at 43.5s/proof, with pw=18 close behind at 43.8s/proof. The previously assumed sweet spot of pw=20 was actually the worst performer in this range (44.9s/proof), though still only 3.2% slower than the optimum.

What the Sweep Reveals About the System

The sweep results tell a deeper story about the system's architecture. The fact that pw=10 and pw=12 perform identically suggests that below a certain threshold, the number of synthesis workers is not the bottleneck — the GPU pipeline is the limiting factor, and as long as enough partitions are available to keep the GPU fed, additional workers provide no benefit. The slight regression at pw=15 and pw=20 suggests the onset of CPU contention, where additional synthesis threads begin competing for the same cores needed by the GPU preprocessing threads. But the effect is small — only ~3% — indicating that the CPU contention is mild at these levels.

The narrow performance band also validates the Phase 8 design: by decoupling CPU preprocessing from GPU kernel execution via the dual-worker interlock, the system has become robust to moderate variations in synthesis worker count. The GPU is no longer left idle waiting for partitions, so even suboptimal partition_workers settings achieve reasonable throughput.

Output Knowledge and Significance

The sweep produced concrete, actionable knowledge: for this specific hardware configuration (96-core AMD Zen4, RTX 5070 Ti), the optimal partition_workers setting is 10-12. This is a production-relevant finding — it means the daemon should be configured with partition_workers = 10 or 12 to maximize throughput, and that values above 12 offer no benefit and may slightly degrade performance.

But the sweep also produced meta-knowledge: it confirmed that the system's performance is plateaued in this region, meaning that further optimization of partition_workers alone will not yield significant gains. The next bottleneck — whether it is CPU synthesis itself, GPU kernel efficiency, or I/O — must be addressed through architectural changes rather than parameter tuning. This understanding sets the stage for the next phases of optimization, which would need to target the synthesis hotpaths or the GPU kernel characteristics rather than configuration knobs.

The user's five-word message, then, is not a simple request — it is a strategic move in a long optimization campaign, designed to close the loop on one variable before moving to the next. It embodies the scientific method applied to systems engineering: form a hypothesis (the optimal pw is around 20), design an experiment (sweep 10-20), execute it systematically, and let the data speak. The data spoke clearly, and the answer — pw=10-12 — is now committed to the project's knowledge base alongside the Phase 8 implementation.