The Moment of Truth: Benchmarking Phase 12's Split GPU Proving API

Introduction

In the high-stakes world of Filecoin proof generation, where every second of proving time translates directly into operational cost, the difference between a successful optimization and a failed experiment is measured in milliseconds per proof. Message [msg 2961] captures one of the most charged moments in any engineering cycle: the first benchmark run after a complex refactor. The assistant has just finished implementing Phase 12 of the cuzk proving engine optimization—a "split GPU proving API" designed to offload the b_g2_msm computation from the GPU worker's critical path—and now it must answer the only question that matters: does it actually work, and does it make things faster?

The Message

The assistant writes:

Daemon is up and running with 2 GPU workers. Let me run the benchmark — c=20 j=15 to match the Phase 11 baseline:

>

``bash /home/theuser/curio/extern/cuzk/target/release/cuzk-bench batch --type porep --c1 /data/32gbench/c1.json --count 20 --concurrency 15 2>&1 ``

>

``` 2026-02-19T23:41:39.563994Z INFO cuzk_bench: loading C1 output path=/data/32gbench/c1.json === Batch Benchmark === proof type: porep count: 20 concurrency: 15

>

[1/20] COMPLETED — 466.0s (prove=76676 ms, queue=5112 ms) [2/20] COMPLETED — 58.6s (prove=50227 ms, queue=296 ms) [3/20] COMPLETED — 94.1s (prove=54158 ms, queue=743 ms) [4/20] COMPLETED — 129.6s (prove=56897 ms, queue=1191 ms) [5/20] COMPLETED — 167.6s (prove=59... ```

The output is truncated in the conversation log, but the subsequent message [msg 2962] reveals the headline result: 37.1 seconds per proof, beating the Phase 11 baseline of 38.0 seconds.

Why This Message Was Written: The Weight of Validation

This message exists because the assistant had just completed a grueling multi-hour debugging and implementation cycle. The Phase 12 split API was not a simple change. It required modifications across three layers of the software stack: the Rust engine orchestration in engine.rs, the Rust pipeline abstraction in pipeline.rs, the bellperson library's FFI boundary in supraseal.rs, and the C++ CUDA kernel code in groth16_cuda.cu. The preceding messages in the conversation ([msg 2936] through [msg 2960]) document a cascade of compilation errors, trait bound mismatches, use-after-free bugs, and concurrency issues that had to be resolved before a clean build was even possible.

The motivation for running this benchmark at this exact moment is threefold. First, the assistant needed to validate that the split API actually functioned correctly end-to-end—that the gpu_prove_start / finish_pending_proof split path was being exercised, that the background prep_msm_thread was properly handling the b_g2_msm computation without dangling references, and that the Rust finalizer task was correctly processing results. Second, the benchmark was designed to measure whether the architectural change delivered on its promised performance improvement: by moving b_g2_msm off the GPU worker's critical path, the worker should be able to pick up the next partition sooner, increasing throughput. Third, the assistant needed to establish a comparison against the Phase 11 baseline (38.0s/proof) to quantify the improvement and decide whether the complexity of the split API was justified.

The choice of benchmark parameters—--count 20 --concurrency 15—was deliberate. The c=20 (count) provides enough samples to average out variance, while j=15 (concurrency, equivalent to partition_workers=15) matches the Phase 11 baseline configuration. This ensures a fair comparison. The assistant explicitly states this intent: "c=20 j=15 to match the Phase 11 baseline."

How Decisions Were Made in This Message

This message is primarily an execution message rather than a decision-making message, but several implicit decisions are embedded within it.

Decision to benchmark immediately after daemon startup. The assistant started the daemon in message [msg 2959] with nohup and waited only 5 seconds before verifying it was running ([msg 2960]). The benchmark was launched immediately after. This is a deliberate choice to minimize the time between code deployment and validation, reflecting an engineering culture of rapid feedback loops. However, it also means the first proof suffers from a cold-start penalty (466 seconds for the first proof versus ~58 seconds for the second), as the GPU and SRS caches warm up. The assistant later acknowledges this in [msg 2962] by noting "the first proof took 466s (cold start)."

Decision to use the existing config file. The assistant reused /tmp/cuzk-p11-int12.toml from Phase 11, which specifies partition_workers = 10 and gpu_workers_per_device = 2. This is the correct approach for a controlled comparison, but it also means the benchmark is testing the Phase 12 code under Phase 11's configuration parameters. The assistant does not experiment with different parameters in this message, deferring that exploration to later analysis.

Decision to trust the build. After fixing the compilation errors and achieving a clean build ([msg 2955]), the assistant did not run unit tests or integration tests before launching the benchmark. This is a pragmatic risk: the build succeeded, so the next step is to see if the system actually works under load. The assistant implicitly assumes that type-correctness at compile time translates to runtime correctness—an assumption that is frequently violated in systems with complex concurrency and GPU interactions.

Assumptions Made

Several assumptions underpin this benchmark run, some explicit and some implicit.

The config file is still valid. The assistant assumes that the Phase 11 configuration file (/tmp/cuzk-p11-int12.toml) is appropriate for Phase 12. This is reasonable because Phase 12 does not introduce new configuration parameters, but it assumes the hardware environment (GPU memory, CPU cores, system memory) has not changed since Phase 11.

The daemon process will stay alive. The assistant launches the daemon with nohup and immediately proceeds to run the benchmark. There is no monitoring loop to detect daemon crashes. The assistant trusts that the daemon will remain healthy throughout the ~12-minute benchmark run (20 proofs × 37 seconds ≈ 740 seconds, plus queueing overhead).

The benchmark binary is compatible. The assistant built cuzk-bench with --no-default-features ([msg 2956]), which disables default feature flags. The assistant assumes this binary is compatible with the daemon, which was built with default features (including cuda-supraseal). This is a potential source of protocol mismatch, though in practice the bench binary communicates with the daemon over HTTP and does not need the CUDA features itself.

The system has sufficient resources. The assistant does not check available memory, GPU memory, or disk space before starting the benchmark. Given that the Phase 11 benchmark ran successfully on the same hardware, this is a reasonable assumption, but it ignores the possibility that Phase 12's split API might have different memory characteristics (as indeed turns out to be the case in the subsequent chunk analysis, where memory pressure becomes a critical issue).

The split API is actually being exercised. The assistant assumes that because the build succeeded and the daemon started, the #[cfg(feature = "cuda-supraseal")] code path is active and the split API is being used. This is confirmed post-hoc in [msg 2963] by grepping for "GPU prove complete (split)" in the daemon log.

Mistakes and Incorrect Assumptions

While the benchmark ultimately succeeds, several assumptions embedded in this message prove to be incomplete or incorrect.

The cold-start problem is underappreciated. The first proof takes 466 seconds, nearly 8 minutes, compared to ~58 seconds for the second proof. This 8× slowdown is attributed to "cold start" in the subsequent message, but the assistant does not account for this in the benchmark design. A proper benchmark would either discard the first sample or run a warm-up proof before timing. The assistant's reported average of 37.1s/proof includes the cold-start proof, which inflates the average. The subsequent analysis in [msg 2962] does not correct for this—it simply reports the raw average.

The assumption that compilation success implies runtime correctness is nearly violated. In the preceding messages, the assistant fixed a critical use-after-free bug in the C++ CUDA code where the prep_msm_thread captured a dangling reference to the stack-allocated provers array ([msg 2946] context). This bug would have caused undefined behavior at runtime—reading from freed stack memory. The assistant fixed it by copying the provers array into the heap-allocated groth16_pending_proof struct. If this fix had been incorrect, the benchmark would have produced corrupted proofs or crashed. The assistant's decision to proceed directly to benchmarking without a targeted test for this specific concurrency scenario was a gamble that paid off, but it was a gamble nonetheless.

The benchmark does not verify proof correctness. The assistant measures throughput (time per proof) but does not verify that the generated proofs are valid. In the context of Filecoin PoRep, an invalid proof is worse than a slow proof—it wastes resources and may lead to failed sector commitments. The subsequent messages do not show any proof verification step. The assistant implicitly assumes that if the GPU completes without errors and the pipeline processes the results, the proofs are valid. This is a significant gap in the validation methodology.

The concurrency level may be too high. The assistant uses j=15 (concurrency 15) but the config file specifies partition_workers = 10. These are different parameters: partition_workers controls how many synthesis tasks run in parallel, while --concurrency in the bench tool controls how many jobs are submitted concurrently. The relationship between these parameters is not explicitly analyzed in this message, and the subsequent chunk analysis reveals that memory pressure becomes a critical bottleneck when concurrency is too high.

Input Knowledge Required

To fully understand this message, the reader needs knowledge spanning multiple domains.

Filecoin proof generation pipeline. The benchmark produces "porep" (Proof of Replication) proofs for 32 GiB sectors. The C1 output file (/data/32gbench/c1.json) contains pre-computed circuit commitments that serve as input to the Groth16 proving phase. The reader must understand that each proof requires synthesizing a circuit from the C1 output, then performing NTT (Number Theoretic Transform) and MSM (Multi-Scalar Multiplication) on the GPU.

The Phase 12 split API architecture. The split API separates GPU proving into two phases: gpu_prove_start (which performs the GPU kernel work including NTT and the main MSM computations) and finish_pending_proof (which completes the b_g2_msm computation that was offloaded to a background thread). This allows the GPU worker to start processing the next partition without waiting for b_g2_msm to finish.

The cuzk benchmarking framework. The cuzk-bench batch command submits proofs to the cuzk-daemon over HTTP. The --count parameter controls how many proofs to generate, and --concurrency controls how many are in flight simultaneously. The output shows per-proof completion time, prove time (GPU + synthesis), and queue wait time.

The optimization history. Phase 11 achieved 38.0s/proof through memory-bandwidth interventions. Phase 12 aims to improve on this by restructuring the GPU worker's critical path. The benchmark parameters are chosen to match Phase 11's configuration for a fair comparison.

Output Knowledge Created

This message produces several critical pieces of knowledge.

The Phase 12 split API works correctly end-to-end. The benchmark runs to completion without crashes, producing 20 proofs across 200 partition completions (as confirmed in [msg 2963]). The "GPU prove complete (split)" log messages confirm that the new code path is active.

The throughput is 37.1s/proof, a ~2.4% improvement over Phase 11. This is a tangible but modest gain. The improvement is smaller than the architectural change might suggest, indicating that b_g2_msm offloading is not the dominant bottleneck in the pipeline. The assistant's subsequent analysis in the next chunk reveals that memory pressure and PCIe bandwidth contention are more significant constraints.

The cold-start penalty is severe. The first proof takes 466 seconds, approximately 8× longer than steady-state proofs. This has operational implications: if the daemon restarts frequently (due to crashes, deployments, or maintenance), a significant portion of proving time is wasted on warm-up.

The benchmark establishes a baseline for future optimization. With Phase 12's performance characterized, the assistant can now investigate the next bottlenecks. The subsequent chunk analysis does exactly this, identifying memory pressure from the partition semaphore release pattern as the next target for optimization.

The Thinking Process Visible in Reasoning

The assistant's reasoning is visible in the structure and content of this message, even though it is primarily an execution message.

The explicit comparison mindset. The assistant states "to match the Phase 11 baseline" — this reveals a scientific benchmarking methodology. The assistant is not just checking whether the code works; it is measuring whether the change improves performance relative to a known baseline. This is characteristic of disciplined performance engineering.

The choice of benchmark parameters reveals an understanding of statistical validity. Using 20 samples provides enough data to compute meaningful averages and detect outliers. The assistant implicitly understands that a single proof would be insufficient to characterize performance, given the variability introduced by GPU thermal throttling, memory bandwidth contention, and OS scheduling.

The assistant's follow-up in [msg 2962] reveals deeper analytical thinking. After seeing the benchmark results, the assistant immediately checks the daemon log for "GPU prove complete (split)" messages to confirm the new code path is active. It also notes the cold-start issue and computes the average gpu_ms across partitions. This shows that the assistant treats the benchmark not as a pass/fail test but as a diagnostic tool, using the output to understand the system's behavior at multiple levels of granularity.

The absence of proof verification is a notable gap in the reasoning. The assistant does not ask "are the proofs valid?" This may reflect an assumption that GPU computation is deterministic and error-free, or it may be a pragmatic choice to defer verification to a later stage. In either case, it reveals a blind spot in the assistant's reasoning about system correctness.

Conclusion

Message [msg 2961] is a moment of validation in a complex engineering cycle. After hours of debugging compilation errors, fixing concurrency bugs, and restructuring code across three language boundaries, the assistant runs the benchmark that will determine whether the Phase 12 split API is a success or a failure. The result—37.1s/proof, a 2.4% improvement—is a success, but a modest one. The message captures the tension between the ambition of the architectural change and the reality of diminishing returns in a highly optimized system. It also sets the stage for the next round of investigation, where the assistant will discover that memory pressure, not GPU compute, is the true bottleneck limiting throughput.