Restructuring the Benchmark: Eliminating Daemon Restarts for Reliable Throughput Measurement

Introduction

In the high-stakes world of Filecoin storage proving, every millisecond counts. The CuZK proving engine—a GPU-accelerated system for generating Proof-of-Replication (PoRep) proofs—must sustain high throughput to keep up with the demands of the Filecoin network. But measuring that throughput accurately is itself a challenge. In message [msg 3760], the assistant delivered a concise summary of a significant restructuring of the benchmark methodology, one that eliminated a costly daemon restart and introduced a three-phase measurement model designed to capture steady-state performance without the artifacts of startup and shutdown.

The message reads in full:

Done. Changes:

>

benchmark.sh — restructured to 3-phase benchmark, no daemon restart: - Daemon always starts with full settings (no more --warmup / synthesis_concurrency=1 mode) - PCE warmup still runs a single proof if PCE not cached (with retries) - No daemon restart after PCE warmup - Phase 1: 5 warmup proofs at full concurrency (untimed, fills pipeline + calibrates pacer) - Phase 2: 10 timed proofs at full concurrency (measured for throughput) - Phase 3: 3 cooldown proofs at full concurrency (untimed, avoids tail effect in timed batch) - New options: --warmup-proofs N (default 5), --cooldown-proofs N (default 3) - Positional arg N is now number of timed proofs (default 10)

>

entrypoint.sh — BENCH_PROOFS=12 changed to BENCH_PROOFS=10 (timed proofs; total is 5+10+3=18)

>

Image pushed: theuser/curio-cuzk:latest — sha256:28c186a2df705ae98885989dec2ccfb92a4f74908aeb432e591e246d550a6fc1

This message, though brief, represents the culmination of a multi-step reasoning process that grappled with fundamental questions about how to measure a pipeline's true throughput. The changes touch on everything from the shell-level orchestration of proof dispatch to the philosophical question of what constitutes a "fair" benchmark measurement.

The Problem: Daemon Restart Overhead

The original benchmark flow had a critical flaw. It started the CuZK daemon in a "warmup mode" with synthesis_concurrency=1, ran a single proof to trigger Pre-Compiled Constraint Evaluator (PCE) extraction, and then restarted the daemon with full settings before running the actual benchmark. This restart forced the daemon to reload the Structured Reference String (SRS)—a large cryptographic parameter set—from disk, adding minutes of overhead to every benchmark run.

The user identified this problem in [msg 3744], noting that the "much longer warmup" was unacceptable and proposing a new flow: no restart after PCE warmup, with a benchmark structured as 5 warmup proofs, 10 timed proofs, and 3 cooldown proofs. This was not merely a cosmetic change—it reflected a deep understanding of how the proving pipeline behaves under load and what it takes to get reliable measurements.

The Reasoning Journey: How to Measure Steady-State Throughput

The assistant's response in [msg 3760] is the final summary, but the reasoning that produced it is visible in the preceding message ([msg 3745]), where the assistant worked through several design alternatives before settling on the final approach.

The core tension was this: if you run warmup proofs and timed proofs as separate sequential batches, the pipeline may drain between batches, meaning the timed proofs don't actually measure steady-state behavior. The assistant considered several alternatives:

  1. Single batch of 18 proofs, filter the middle 10: This would dispatch all proofs at once at full concurrency, then parse the output to extract timing only for proofs 6-15. This gives true steady-state measurement since the pipeline never drains. However, it requires knowing the exact output format of cuzk-bench batch and whether it provides per-proof timing data—information the assistant didn't have without reading the Rust source code.
  2. Overlapping batches: Start the timed batch while warmup proofs are still in flight, so the pipeline stays busy. This is complex to implement in shell script and risks measurement contamination.
  3. Sequential batches with quick dispatch: Run warmup, timed, and cooldown back-to-back at full concurrency. The pipeline will dip slightly between batches, but the assistant judged that it would stabilize within the first couple of proofs in the timed batch, so the 10 measured proofs should still capture mostly steady-state behavior. The assistant ultimately chose option 3—the sequential approach—as the pragmatic compromise. The reasoning shows a clear awareness of the trade-off: "the pipeline will dip slightly between batches but should stabilize within the first couple of proofs in the timed batch." This is a judgment call that prioritizes implementation simplicity and reliability over theoretical purity.

The Three-Phase Model

The chosen design divides the benchmark into three distinct phases, each with a specific purpose:

Phase 1: Warmup (5 proofs, untimed). These proofs fill the proving pipeline and allow the PI-controlled dispatch pacer (a system developed in earlier segments) to calibrate to the workload. The warmup proofs are dispatched at full concurrency—no throttling—so the pipeline reaches its natural steady state. The assistant's comment that this "fills pipeline + calibrates pacer" reveals an important assumption: that 5 proofs are sufficient to reach steady state. This is an empirical question that may need tuning later, but it's a reasonable starting point given the pipeline depths observed in earlier work.

Phase 2: Timed (10 proofs, measured). These are the proofs that count. Dispatched at full concurrency into an already-warm pipeline, their completion times form the basis for throughput calculations. The choice of 10 is a balance between statistical significance and practical run time—enough proofs to average out variance, not so many that the benchmark takes too long.

Phase 3: Cooldown (3 proofs, untimed). These proofs are dispatched after the timed batch to "avoid tail effect in the timed batch." This is a subtle but important insight: when the pipeline starts to drain, the last few proofs may see less contention for GPU resources and complete faster, artificially inflating the throughput measurement. By dispatching 3 extra proofs that aren't counted, the timed proofs are all in the steady-state region where the pipeline is fully occupied.

Implementation Details

The implementation involved two files and a Docker rebuild:

benchmark.sh received the bulk of the changes. The assistant removed the --warmup mode entirely—the daemon now always starts with full settings. The PCE warmup still runs a single proof if the PCE hasn't been cached yet (with retries), but there's no daemon restart afterward. New command-line options --warmup-proofs (default 5) and --cooldown-proofs (default 3) were added, and the positional argument N now represents the number of timed proofs (default 10 instead of the old total of 12).

entrypoint.sh received a single-line change: BENCH_PROOFS=12 became BENCH_PROOFS=10. This is because the entrypoint's BENCH_PROOFS variable now represents only the timed proofs, not the total. The total number of proofs dispatched is 18 (5 + 10 + 3), but only 10 are measured.

After the edits, the assistant rebuilt the Docker image and pushed it to the registry, producing image theuser/curio-cuzk:latest with SHA 28c186a2df705ae98885989dec2ccfb92a4f74908aeb432e591e246d550a6fc1.

Assumptions and Potential Pitfalls

The design rests on several assumptions that deserve scrutiny:

The pipeline doesn't fully drain between phases. This is the biggest assumption. If the warmup batch completes and the timed batch doesn't start quickly enough, the GPU workers may idle, and the first few timed proofs will see a cold pipeline. The assistant mitigated this by running all phases at full concurrency and keeping them sequential in the same script, but the risk remains.

5 warmup proofs is enough. The proving pipeline has multiple stages (synthesis, GPU proving, etc.), each with its own queue depth. If the pipeline is deeper than 5 proofs, the warmup may not fully saturate it. The assistant's earlier work on the PI-controlled dispatch pacer and synthesis throughput cap suggests the pipeline depth is well-understood, but this is still an empirical question.

The cooldown phase actually helps. The theory is that the cooldown proofs prevent tail effects in the timed batch. But if the cooldown proofs are dispatched after the timed proofs complete (rather than concurrently), they don't actually affect the pipeline state during the timed phase. The assistant's implementation dispatches them sequentially, so the cooldown may be more about draining the pipeline gracefully than about measurement quality.

Input and Output Knowledge

To understand this message, a reader needs familiarity with:

Conclusion

Message [msg 3760] appears at first glance to be a simple status update—"Done. Changes:" followed by a bullet list. But it represents the resolution of a complex design problem: how to measure the throughput of a GPU proving pipeline without the measurement itself distorting the results. The assistant navigated trade-offs between implementation simplicity and measurement accuracy, made explicit assumptions about pipeline behavior, and delivered a solution that eliminates a costly daemon restart while providing cleaner steady-state measurements.

The three-phase model—warmup, timed, cooldown—is a pattern that appears in many benchmarking contexts, from CPU benchmarks to network throughput tests. Its application here shows a maturing understanding of the CuZK system's behavior under load and a commitment to producing reliable, reproducible performance data for production deployment.