The Benchmark That Told a Story: Measuring Phase 11's First Intervention

FIL_PROOFS_PARAMETER_CACHE=/data/zk/params /home/theuser/curio/extern/cuzk/target/release/cuzk-bench batch --type porep --c1 /data/32gbench/c1.json --count 20 --concurrency 15 2>&1 | tee /home/theuser/cuzk-p11-int1-c20j15.log
2026-02-19T20:20:22.280742Z INFO  cuzk_bench: loading C1 output path=/data/32gbench/c1.json
=== Batch Benchmark ===
proof type:  porep
count:       20
concurrency: 15

  [1/20] COMPLETED — 62.4s (prove=51975 ms, queue=300 ms)
  [2/20] COMPLETED — 97.1s (prove=53708 ms, queue=736 ms)
  [3/20] COMPLETED — 134.0s (prove=54498 ms, queue=1172 ms)
  [4/20] COMPLETED — 170.9s (prove=55266 ms, queue=1599 ms)
  [5/20] COMPLETED — 208.6s (prove=60...

At first glance, this message looks mundane — just another benchmark run in a long optimization campaign. A bash command, some log output, a few completed proofs with timing numbers. But this single command represents a critical juncture in a months-long effort to squeeze every drop of performance out of a GPU-accelerated zero-knowledge proof generation pipeline. It is the moment when a carefully formed hypothesis about memory-bandwidth contention meets the cold, unforgiving reality of empirical measurement.

The Context: A Pipeline Under the Microscope

To understand why this message was written, we must step back and appreciate the broader arc of the optimization campaign. The subject is the cuzk proving engine — a system that generates Groth16 zero-knowledge proofs for Filecoin's Proof-of-Replication (PoRep) protocol. The pipeline is a marvel of cross-language engineering: Go orchestration (Curio) delegates to Rust FFI, which calls into C++ CUDA kernels, which in turn drive NVIDIA GPUs through a sequence of number-theoretic transforms (NTTs), multi-scalar multiplications (MSMs), and other cryptographic operations. The peak memory footprint is approximately 200 GiB, and the system operates under extreme memory pressure.

By the time we reach this message, the optimization campaign has already produced ten phases of iterative improvement. Phase 9 achieved 32.1 seconds per proof in isolation but degraded to 38.0 seconds under high concurrency (20 concurrent proofs, 15 jobs each). The Phase 10 two-lock GPU interlock was explored and abandoned after discovering fundamental CUDA device-global synchronization conflicts. Now Phase 11 targets a newly identified bottleneck: DDR5 memory bandwidth contention.

The Phase 11 design spec (committed just moments before this message at [msg 2748]) identifies three interventions:

  1. Intervention 1: Serialize async_dealloc with a static mutex to bound TLB shootdown storms
  2. Intervention 2: Reduce groth16_pool from 192 to 32 threads to cut L3 cache thrashing
  3. Intervention 3: A global atomic throttle flag to coordinate memory bandwidth between CPU and GPU The message we are examining is the first benchmark of Intervention 1 — the very first empirical test of Phase 11's hypotheses.

Why This Message Was Written: The Motivation

The assistant wrote this message to answer a single, crucial question: Does serializing the async deallocation path improve throughput under high concurrency?

The reasoning behind Intervention 1 was rooted in deep memory-subsystem analysis. The waterfall timing analysis from Phase 9 had revealed that GPU per-partition time inflated from 4.9 seconds to 7.5 seconds under load, and synthesis time ballooned from 35 seconds to 54 seconds. The system had shifted from being GPU-bound to being CPU memory-bandwidth-bound. One suspected contributor was TLB shootdown storms — a phenomenon where concurrent page table modifications across multiple cores cause expensive inter-processor interrupts. The async_dealloc path, which frees large GPU-side buffers from a pool of worker threads, was a prime suspect because it touches page tables across many threads simultaneously.

The hypothesis was straightforward: by serializing deallocations behind a single mutex, the system would serialize TLB invalidations, reducing the frequency of shootdown IPIs and freeing memory bandwidth for the actual computational work (synthesis, SpMV, MSM). The benchmark was designed to test this hypothesis under the same conditions that produced the 38.0-second baseline: c=20 j=15 (20 concurrent proofs, 15 jobs per proof) with gpu_workers_per_device = 2 and partition_workers = 10.

How Decisions Were Made: The Benchmark Design

The assistant made several deliberate choices in designing this benchmark:

Choice of parameters: The --count 20 --concurrency 15 flags exactly match the Phase 9 baseline configuration. This is critical for scientific validity — any performance difference can be attributed to the code change, not to different test conditions. The assistant could have chosen different parameters (higher concurrency to stress-test the intervention more aggressively, or lower concurrency to measure best-case overhead), but the immediate goal was a direct apples-to-apples comparison.

Choice of benchmark binary: The cuzk-bench binary with the batch subcommand was chosen over the single subcommand because batch mode measures sustained throughput under realistic pipeline conditions, including queueing effects and resource contention. A single-proof test would not reveal the memory-bandwidth contention that Intervention 1 was designed to mitigate.

Choice of measurement: The benchmark records both wall-clock time per proof (the cumulative "COMPLETED — Xs" column) and the individual prove time (the "prove=Y ms" values). The wall-clock time includes queueing delay, which is essential for understanding system-level throughput, while the prove time isolates the actual computation cost.

Pre-benchmark setup: Before running this command, the assistant had already (1) committed the Phase 10 post-mortem and Phase 11 design spec ([msg 2748]), (2) implemented Intervention 1 in both C++ and Rust ([msg 2750], <msg id=2753-2754>), (3) force-rebuilt the C++ CUDA code and the Rust binaries (<msg id=2756-2757>), (4) killed any existing daemon ([msg 2759]), (5) started a fresh daemon with the new code ([msg 2760]), and (6) verified the daemon was ready ([msg 2761]). This careful setup ensures that the benchmark measures the new code, not a stale build or a contaminated process state.

Assumptions Made

Every benchmark rests on assumptions, and this one is no exception:

The daemon configuration is correct: The assistant assumed that the daemon started with gpu_workers_per_device = 2 and partition_workers = 10 (verified at [msg 2758]) and that no other configuration changes would interfere with the measurement.

The system is in a steady state: The assistant assumed that the 30-second wait before running the benchmark ([msg 2761]) was sufficient for the daemon to initialize SRS parameters and reach a steady operating state.

The Intervention 1 code path is exercised: The assistant assumed that the async_dealloc path is actually triggered during these proofs — that the deallocation serialization mutex is contended enough to make a measurable difference. If the deallocations are infrequent or fast, serializing them would have no effect.

The baseline is reproducible: The 38.0-second Phase 9 baseline was assumed to be stable and reproducible. If the baseline itself is noisy (due to thermal throttling, OS scheduling jitter, or other environmental factors), then a small improvement or regression might be indistinguishable from noise.

The benchmark environment is consistent: The assistant assumed that nothing else on the system was competing for memory bandwidth during the benchmark — no background cron jobs, no other users, no system maintenance tasks.

Mistakes and Incorrect Assumptions

The path to this benchmark was not without its stumbles. Earlier in the session, the assistant attempted to run the benchmark with a --server flag ([msg 2762]) and received an error:

error: unexpected argument '--server' found

Usage: cuzk-bench [OPTIONS] <COMMAND>

For more information, try '--help'.

This mistake reveals an important aspect of the assistant's workflow: it is operating in a codebase with multiple CLI interfaces, and the --server flag belonged to a different version or a different binary. The assistant had to consult the help output (<msg id=2763-2764>) to discover the correct batch subcommand syntax. This is a common pitfall in large, evolving codebases — documentation and actual CLI interfaces can diverge, and the assistant must adapt.

A more subtle issue is the choice of --concurrency 15 versus the original --jobs 15 parameter. The assistant initially tried --concurrency 15 after checking the help, but the Phase 9 baseline used --jobs 15. If --concurrency and --jobs have different semantics (e.g., one controls total parallelism while the other controls per-proof parallelism), then the benchmark might not be comparing apples to apples. The assistant appears to have inferred the correct parameter from the help output, but this inference is an assumption that could introduce measurement error.

Input Knowledge Required

To understand this message, one needs substantial context about the system being benchmarked:

The proving pipeline: Knowledge that Groth16 proof generation involves multiple phases — synthesis (building the circuit from constraints), MSM (multi-scalar multiplication on the GPU), NTT (number-theoretic transforms), and epilogue (final proof assembly). The prove=51975 ms value represents the total time from job submission to proof completion, encompassing all these phases.

The concurrency model: Understanding that c=20 means 20 proofs are submitted concurrently (the batch size), while j=15 means each proof uses 15 parallel jobs during synthesis. The system has gpu_workers_per_device = 2, meaning two GPU worker threads share each physical GPU, competing for VRAM and compute resources.

The memory bandwidth problem: Knowledge that Phase 9 identified DDR5 memory bandwidth as the primary bottleneck under high concurrency, with GPU time inflating from 4.9s to 7.5s per partition and synthesis from 35s to 54s. This context explains why serializing deallocations might help — it reduces memory subsystem contention.

The Phase 11 design spec: Understanding that Intervention 1 is specifically about TLB shootdown mitigation, which requires knowledge of operating system memory management (page tables, TLB invalidation, IPI overhead) and how concurrent deallocation patterns can trigger these costs.

The tooling: Familiarity with the cuzk-bench binary, its CLI interface, and the daemon architecture (the daemon runs as a background service, the bench binary connects to it via HTTP).

Output Knowledge Created

This message produces several forms of knowledge:

Empirical performance data: The benchmark output shows per-proof completion times and prove latencies. The first proof completes in 62.4 seconds total (51.975 seconds prove time, 300 ms queue delay). Subsequent proofs show increasing queue delays (300 ms → 736 ms → 1172 ms → 1599 ms) as the pipeline fills up, while prove times remain relatively stable (~52-55 seconds). This data feeds directly into the next message ([msg 2766]), where the assistant computes an average of 37.9 seconds per proof — essentially identical to the 38.0-second Phase 9 baseline.

A negative result: The most important output of this message is the finding that Intervention 1 alone produces no measurable improvement. This is a valuable scientific result — it tells the team that TLB shootdowns from async deallocation are not a significant contributor to the memory-bandwidth bottleneck, at least at this concurrency level. This negative result prevents wasted effort on further optimizing the deallocation path and redirects attention to Interventions 2 and 3.

Evidence for the next iteration: The benchmark data also reveals that queue delays are increasing steadily (from 300 ms to 1.6 seconds by proof 4), suggesting that the pipeline is still throughput-limited. The prove times themselves are relatively stable, indicating that the GPU workers are not the bottleneck — the bottleneck is elsewhere in the pipeline (likely CPU synthesis or memory bandwidth). This evidence informs the design of Interventions 2 and 3.

A reproducible baseline: By running the benchmark with known parameters and recording the output to a log file (/home/theuser/cuzk-p11-int1-c20j15.log), the assistant creates a permanent record that can be compared against future benchmarks. This is essential for tracking progress across the optimization campaign.

The Thinking Process Visible in the Reasoning

The assistant's reasoning, visible in the surrounding messages, reveals a methodical scientific mindset:

Hypothesis formation: The assistant starts with a clear hypothesis: serializing async deallocations will reduce TLB shootdown storms and improve throughput. This hypothesis is grounded in detailed knowledge of the memory subsystem.

Controlled experimentation: The assistant carefully controls for confounding variables — using the same benchmark parameters as the baseline, rebuilding from scratch, starting a fresh daemon, waiting for initialization.

Measurement and analysis: After running the benchmark, the assistant immediately computes the average throughput (37.9 s/proof) and compares it to the baseline (38.0 s/proof), concluding that the effect is "essentially identical" ([msg 2766]). The assistant also checks the daemon logs for dealloc timing data to understand why the intervention didn't help.

Iterative refinement: The negative result doesn't stop the optimization effort — it redirects it. The assistant moves on to Interventions 2 and 3, and eventually to the Phase 12 split API design, all within the same session.

This thinking process exemplifies the engineering mindset required for systems optimization: form a hypothesis, design a minimal experiment, measure carefully, accept negative results, and iterate. The message is not just a benchmark — it is a scientific instrument deployed to test a theory about how a complex system behaves under load.

Conclusion

Message [msg 2765] is a benchmark that told a story — the story of a hypothesis tested and found wanting. In the broader arc of the optimization campaign, it represents a necessary dead end: the moment when the team learned that TLB shootdowns from async deallocation were not the bottleneck they were looking for. This negative result, far from being a failure, is a success of the scientific method applied to systems engineering. It saved the team from pursuing a low-impact optimization and redirected effort toward the interventions that would eventually yield real improvements (Intervention 2's thread pool reduction, and the Phase 12 split API).

The message also reveals the discipline required for effective optimization work: careful benchmarking setup, reproducible parameters, thorough logging, and honest interpretation of results. In a field where it's tempting to see improvements where none exist, the willingness to measure and accept "no change" is a mark of engineering maturity.