The Benchmark That Validated Phase 12: A Split GPU Proving API Under Pressure

Introduction

In the high-stakes world of Filecoin proof-of-replication (PoRep) proving, every millisecond counts. The SUPRASEAL_C2 Groth16 proof generation pipeline is a beast of a system—spanning Go orchestration, Rust FFI boundaries, and C++/CUDA GPU kernels—that chews through ~200 GiB of memory to produce a single proof. For engineers optimizing this pipeline, a 2.4% throughput improvement is a hard-won victory, earned through meticulous bug fixing, architectural redesign, and relentless benchmarking.

Message [msg 3042] captures one such victory: the moment the Phase 12 split GPU proving API was put to the test. In this message, the assistant launches a batch benchmark of the newly implemented split API, running 20 proofs with 15 concurrent workers. The output—truncated in the conversation but revealing the first five completion times—represents the culmination of a multi-week optimization journey spanning Phases 9 through 12, encompassing PCIe transfer optimization, a failed two-lock GPU interlock design, memory-bandwidth interventions, and finally the split API that offloads the b_g2_msm computation from the GPU worker's critical path.

What the Message Contains

The message is deceptively simple: a single bash command invoking the cuzk-bench tool with a batch benchmark configuration:

/home/theuser/curio/extern/cuzk/target/release/cuzk-bench batch --type porep --c1 /data/32gbench/c1.json --count 20 --concurrency 15

The output shows the benchmark beginning to produce results. The first proof completes in 79.0 seconds (72.4 seconds of actual proving time plus 299 milliseconds of queue wait). The second takes 120.4 seconds from start (67.3 seconds of proving), the third 161.2 seconds (61.2 seconds proving), the fourth 194.1 seconds (84.6 seconds proving), and the fifth is truncated at 214.3 seconds with a proving time beginning with "6..." The pattern is clear: the system is warming up, with queue times increasing as more proofs enter the pipeline and proving times fluctuating as the GPU worker juggles partitions.

What the truncated output doesn't show—but what the broader context reveals—is the final result: a steady-state throughput of 37.1 seconds per proof, a ~2.4% improvement over the Phase 11 baseline of 38.0 seconds. This is the number that validates the entire Phase 12 effort.

Why This Message Was Written: The Reasoning and Motivation

This benchmark was not a casual test. It was the final validation gate for Phase 12, an optimization phase that had consumed dozens of assistant messages across multiple days of development. To understand why this message exists, one must understand the problem Phase 12 was designed to solve.

The Groth16 proof generation pipeline has a critical path bottleneck: the GPU worker must wait for the b_g2_msm (G2-group multi-scalar multiplication) computation to complete before it can finalize a proof. This computation is performed on the CPU as a "prep" step, but in the original architecture, it blocked the GPU worker from proceeding to the next partition. The insight behind Phase 12 was to split the proving API into a start phase (which launches the GPU work and the b_g2_msm prep in parallel) and a finalize phase (which collects results after both complete). This decoupling allows the GPU worker to return to the pool sooner, increasing throughput.

But implementing this split was fraught with difficulty. The assistant had to fix multiple Rust/C++ FFI compilation errors, add missing struct definitions, correct trait bound mismatches, and—most critically—discover and fix a use-after-free bug in the CUDA code where a background thread captured a dangling reference to a stack-allocated array. The provers array, passed as a function parameter, was being accessed by the prep_msm_thread after the function had returned, reading garbage memory. The fix—copying the provers array into a heap-allocated provers_owned field in the groth16_pending_proof struct—was essential for correctness.

With all bugs fixed and a clean build achieved, the assistant needed to answer one question: does the split API actually improve throughput? This benchmark was designed to answer that question definitively.

How Decisions Were Made

Several design decisions are embedded in this benchmark command. The choice of --count 20 provides enough proofs for the pipeline to reach steady state and produce a statistically meaningful average. A single proof would be dominated by startup overhead; 20 proofs allow the warm-up period (visible in the first few results) to be amortized.

The --concurrency 15 parameter matches the j=15 setting from the daemon configuration, which was established as the optimal concurrency level in earlier phases. This is the number of proof jobs that can be in-flight simultaneously. The assistant chose to benchmark at the proven sweet spot rather than pushing higher concurrency, because earlier tests at pw=12 (partition workers) had already demonstrated memory capacity limits—peaking at 668 GiB RSS on a 755 GiB system, leaving only 87 GiB headroom before OOM.

The choice of --type porep is straightforward: this is the proof type being optimized for Filecoin storage proofs. The C1 input file (/data/32gbench/c1.json) is a pre-computed intermediate from the first phase of proving, representing a 32 GiB sector benchmark.

Notably, the assistant does not include additional diagnostic flags or verbose output. This is a clean, production-like benchmark run. The assumption is that the daemon is already configured with the optimal parameters (gw=2, pw=10, gt=32) and that the benchmark results will speak for themselves.

Assumptions Underlying This Message

This message rests on several critical assumptions:

  1. The use-after-free fix is correct. The assistant had just fixed a subtle concurrency bug where the prep_msm_thread accessed the provers array after the owning function returned. The fix—copying the array into heap-allocated memory and using a provers_safe alias—assumed that all other captured references in the background thread were to heap-allocated or long-lived objects. If this assumption were wrong, the benchmark could produce corrupted results or crash.
  2. The daemon is running with the intended configuration. The assistant had started the daemon with a configuration file at /tmp/cuzk-p12-pw12.toml, but the benchmark was run with the standard configuration. Any mismatch between the daemon's actual configuration and the expected gw=2, pw=10, gt=32, j=15 settings would invalidate the comparison to the Phase 11 baseline.
  3. The system has sufficient memory. With pw=10, the assistant expected peak RSS around 367 GiB (as observed in earlier runs). The 755 GiB system should have ample headroom. However, this assumption was about to be challenged in subsequent messages.
  4. The benchmark tool measures what we think it measures. The prove time reported by cuzk-bench is the time spent in the actual proof generation, excluding queue wait. The assistant assumes this metric is comparable across Phase 11 and Phase 12 runs.

Input Knowledge Required

To fully understand this message, one needs knowledge spanning several domains:

Output Knowledge Created

This message produces several important outputs:

  1. Validation that Phase 12 compiles and runs without crashing. After the compilation fixes and the use-after-free bug fix, the first benchmark run confirms the system is stable at pw=10.
  2. A performance baseline of 37.1s/proof. While the message truncates before showing the final average, subsequent analysis confirms this result. The 2.4% improvement over Phase 11's 38.0s is modest but meaningful—it represents the cumulative effect of the split API optimization.
  3. Evidence that pw=10 is the memory ceiling. The benchmark was run at pw=10 because pw=12 had already been shown to OOM. This message implicitly confirms that the memory capacity bottleneck identified earlier is real and reproducible.
  4. Queue time data. The first five results show queue times increasing from 299ms to 1639ms, indicating the pipeline is filling up. This data is valuable for understanding system behavior under load.

The Thinking Process Visible in the Reasoning

While this message itself contains no explicit reasoning (it is purely a command execution), the reasoning is embedded in the sequence of actions that led to it. The assistant had just completed a multi-step debugging and fixing process:

  1. Identified the use-after-free bug in generate_groth16_proofs_start_c
  2. Added provers_owned field to the groth16_pending_proof struct
  3. Copied the provers array into the heap-allocated copy
  4. Created a provers_safe alias for use inside the background thread
  5. Rebuilt the project successfully
  6. Killed the old daemon and started a fresh one
  7. Launched an RSS monitor to track memory usage
  8. Then ran the benchmark The decision to run the benchmark before investigating the memory pressure issue further reveals a prioritization: first confirm the split API works and measure its throughput, then tackle the memory capacity problem. This is a pragmatic engineering approach—establish the baseline before optimizing further. The choice of pw=10 (rather than attempting pw=12 again) shows that the assistant learned from the earlier OOM failure. Rather than repeating the same experiment, the assistant accepted the memory ceiling and benchmarked at the proven-safe configuration.

Mistakes and Discoveries

The most significant discovery hidden behind this message is that the use-after-free fix alone did not resolve the OOM at pw=12. In the very next message ([msg 3043]), the assistant checks the RSS trace and finds a peak of 668 GiB—essentially identical to the pre-fix run. This reveals that the memory pressure was not caused by the dangling pointer bug (which would cause corruption or crashes, not memory accumulation), but by a genuine capacity issue: too many synthesized partitions holding their full ~16 GiB datasets while waiting for the single-slot GPU channel.

This discovery would lead to the next phase of investigation (chunk 1 of segment 30), where the assistant builds a global buffer tracker with atomic counters, discovers that the partition semaphore releases too early, and iterates through two different fixes—first holding the semaphore permit until GPU delivery (which fixes memory but regresses throughput to 39.9s), then increasing channel capacity from 1 to partition_workers (which balances both concerns).

The mistake, if any, was the assumption that the use-after-free fix might also help with memory pressure. The assistant speculated that "perhaps the UB was causing memory corruption that prevented proper dealloc" ([msg 3035]). This turned out to be incorrect—the UB was a correctness bug, not a memory leak. But exploring this hypothesis was a reasonable diagnostic step, and the benchmark provided the data needed to rule it out.

Significance in the Broader Project

Message [msg 3042] sits at a pivotal moment in the optimization journey. It represents the successful delivery of Phase 12—a working split API that offloads b_g2_msm from the GPU critical path—while simultaneously revealing the next frontier: memory pressure management. The 37.1s/proof result is not the end of the road but a waypoint. The memory capacity bottleneck identified here would drive the next optimization cycle, focusing on reducing per-partition memory footprint and improving pipeline buffering.

In the larger narrative of the SUPRASEAL_C2 optimization project, this message is where theory meets practice. The elegant architectural idea of the split API is validated by real hardware, and the messy reality of memory constraints asserts itself. It is a reminder that in high-performance computing, the most carefully designed optimization is only as good as the system's ability to sustain it.