The Art of the Benchmark Transition: One Data Point at a Time

In a sprawling coding session dedicated to optimizing the SUPRASEAL_C2 Groth16 proof generation pipeline for Filecoin's Proof-of-Replication (PoRep), most messages are sprawling affairs — multi-tool orchestration, deep architectural reasoning, bug hunts spanning C++ and Rust. And then there is message 3295, which is almost comically brief:

pw=2 gw=1: 110 GiB peak, 152s/proof throughput, 33.7s prove time. Now pw=5:

That's it. Three numbers, a config label, and a transition. But within this terse utterance lies the culmination of an entire engineering phase: the shift from optimization to characterization, from making things faster to understanding what we've made.

The Context: From Building to Measuring

To understand why this message exists, we must step back. The preceding weeks (in session-time) had been an intense period of architectural iteration. Phase 9 had identified PCIe transfer bottlenecks. Phase 10 attempted a two-lock GPU interlock design that was abandoned after discovering fundamental CUDA device-global synchronization conflicts. Phase 11 introduced three memory-bandwidth interventions. Phase 12 implemented a split GPU proving API with memory backpressure — a complex set of changes including early a/b/c vector deallocation, channel capacity auto-scaling, and a use-after-free fix in the C++ prep_msm_thread.

By the time we reach message 3295, Phase 12 is complete. The engine works. It produces proofs at 37.7 seconds with 12 partition workers and 2 GPU workers, consuming ~400 GiB of RAM. But the user — clearly someone thinking about production deployment — asks a different kind of question (message 3272):

Run cuzk with lower parallelsms to see how low we can get RSS requirements on smaller systems and the timings. Try 1/2/5/7, also lower queues/gpu workers that would also hold a lot of memory.

This is a question about deployment envelopes. The Phase 12 engine at pw=12 gw=2 is a powerhouse, but it requires a 512 GiB or larger system. What about the 128 GiB machine? The 256 GiB machine? The user wants a map — a configuration guide that maps hardware capabilities to performance outcomes.

The Benchmark Methodology Takes Shape

Message 3295 is the second data point in a systematic sweep. The assistant had already run pw=1 gw=1 (message 3288), discovering that a single partition worker achieves only 104 GiB peak RSS but takes ~290 seconds per proof — essentially serializing all 10 partitions through one worker. The prove time (the GPU portion) was a consistent ~33 seconds regardless of pw, confirming that synthesis, not GPU computation, was the throughput bottleneck at low worker counts.

For pw=2, the assistant had written a proper benchmark script (message 3292) that automated the entire workflow: start the daemon, wait for SRS loading, record baseline RSS, run the benchmark with a specified number of proofs and concurrency level, monitor RSS every two seconds during execution, capture peak RSS from /proc/$PID/status (VmHWM), and finally kill the daemon. This script was a necessary piece of infrastructure — the earlier pw=1 run had used ad-hoc monitoring that produced an empty log file due to shell timing issues (message 3285-3286).

The pw=2 run used 3 proofs at concurrency 3, which the assistant estimated would take about 7-8 minutes based on the calculation: with 2 workers handling 10 partitions, each round of parallel work processes 2 partitions, requiring ~5 rounds at ~29 seconds each ≈ 145 seconds per proof. Three proofs at concurrency 3 would overlap, so total wall time would be roughly 3× the per-proof synthesis time divided by concurrency, plus queueing overhead.

What the Numbers Tell Us

The pw=2 result — 110 GiB peak, 152s/proof, 33.7s prove — reveals a critical relationship. Doubling the partition workers from 1 to 2 added only 6 GiB of peak memory (from 104 to 110 GiB), but nearly halved the wall time (from ~290s to 152s). This is a remarkably favorable tradeoff: almost linear throughput scaling for negligible memory cost.

The prove time staying constant at ~33.7s confirms that the GPU pipeline is never the bottleneck at these low worker counts. The GPU finishes its work and then waits — idle — for the CPU synthesis to deliver the next partition's data. This is exactly the behavior the Phase 12 split API was designed to enable: the GPU worker loop decouples proof finalization from synthesis, so prove time reflects only the GPU's actual computation, not queueing delay.

The baseline RSS of ~69 GiB (observed across all runs) represents the SRS (Structured Reference String) loaded into pinned memory, plus the daemon's fixed overhead. The additional ~41 GiB for pw=2 comes from in-flight partition data: evaluation vectors, a/b/c coefficients, and the partition semaphore permits held during synthesis-to-GPU transfer.

The Transition: "Now pw=5"

The second half of the message — "Now pw=5" — is not just a status update. It's a decision point embedded in a todo list update. The assistant updates the todo tracker, marking pw=1 and pw=2 as completed and pw=5 as in-progress. This reveals the systematic, almost scientific nature of the sweep: each configuration is a controlled experiment, and the assistant moves through them in order, collecting data points without skipping ahead.

The choice to skip from pw=2 to pw=5 (rather than pw=3, pw=4) is itself informative. The assistant knows from prior Phase 12 characterization that the throughput/memory curve is roughly linear with diminishing returns. The interesting inflection points are at the low end (pw=1, pw=2 for minimal systems), the middle (pw=5, pw=7 for mid-range systems), and the high end (pw=10, pw=12 for production). Testing every integer would be wasteful — the data would be correlated and the benchmark time would be prohibitive.

The Deeper Significance

Message 3295 is a message about transition — from one experimental condition to the next, from optimization to characterization, from building to deploying. The entire sweep that this message inaugurates would eventually produce a clean memory scaling formula: ~69 GiB baseline + pw × ~20 GiB. It would reveal that gw=2 provides no throughput benefit below pw=10 because synthesis starvation leaves the second GPU worker idle. It would produce concrete deployment guidance: 128 GiB systems can run pw=2 gw=1 at 152s/proof; 256 GiB systems can run pw=7 gw=1 at 53s/proof; 768 GiB systems can run pw=12 gw=2 at 42.5s/proof.

But none of that exists yet in message 3295. At this moment, the assistant has exactly two data points: pw=1 at 104 GiB/290s, and pw=2 at 110 GiB/152s. The pattern is promising — near-linear throughput scaling for minimal memory cost — but the curve could break at any point. The assistant doesn't know yet that pw=5 will hit 170 GiB/68s, or that pw=7 will reach 208 GiB/53s. Each subsequent message will add another point to the graph, building the empirical foundation for the deployment guide that will be committed as 9bb657e5.

This is the essence of systems engineering at scale: you don't design a configuration; you discover it through measurement. Message 3295 is the quiet pivot point where the conversation stops asking "how do we make this faster?" and starts asking "how do we fit this into the real world?"