The Benchmark That Validated Phase 9: PCIe Transfer Optimization for Groth16 Proving

Message Overview

On February 19, 2026, the assistant executed a benchmark that would serve as the first successful validation of Phase 9's PCIe transfer optimization for the cuzk SNARK proving engine. The message, a single bash command piped through tee, produced output that told a story of recovery from repeated failure:

[bash] /home/theuser/curio/extern/cuzk/target/release/cuzk-bench batch -t porep --c1 /data/32gbench/c1.json -c 3 -j 1 2>&1 | tee /tmp/cuzk-phase9-gw1-bench.log

The results showed three proofs completing successfully with an average wall time of approximately 65.7 seconds per proof — a dramatic turnaround from the previous attempt just minutes earlier, where all three proofs had failed with out-of-memory (OOM) errors. This message represents the culmination of a deep diagnostic spiral into CUDA memory management, and the beginning of a new understanding of where the true bottlenecks lie in the multi-worker proving pipeline.

Context: The Road to Phase 9

To understand why this message was written, we must trace the chain of reasoning that led to it. The cuzk SNARK proving engine performs Groth16 proof generation for Filecoin's Proof-of-Replication (PoRep) protocol. Each proof requires processing polynomial evaluations (a/b/c vectors) through NTT (Number Theoretic Transform) and MSM (Multi-Scalar Multiplication) operations on the GPU. The Phase 8 baseline had established that GPU utilization was suboptimal, with two root causes identified: (1) non-pinned host memory causing slow PCIe transfers for the 6 GiB of a/b/c polynomial data, and (2) hard synchronization stalls in the Pippenger MSM algorithm.

Phase 9 was designed to address both issues. Change 1 (Tier 1) moved the a/b/c polynomial uploads out of the GPU mutex by pinning host memory with cudaHostRegister, allocating device buffers, and issuing asynchronous cudaMemcpyAsync transfers on a dedicated stream with event-based synchronization. Change 2 (Tier 3) eliminated per-batch hard sync stalls in the Pippenger MSM by introducing double-buffered host result buffers and deferring the sync() call to the next iteration, allowing GPU compute to overlap with device-to-host transfers.

The OOM Crisis

The initial implementation of Phase 9 crashed immediately. When the assistant ran the first benchmark with gpu_workers_per_device=2 (the intended dual-worker configuration), all three proofs failed with OOM errors. The diagnostic trail revealed two interacting issues.

First, with two GPU workers per device, both workers tried to pre-stage their 12 GiB allocations simultaneously — 4 GiB for d_a and 8 GiB for d_bc — totaling 24 GiB on a 16 GiB GPU. The pre-staging had been placed outside the GPU mutex to maximize overlap, but this meant both workers could attempt allocation concurrently.

Second, and more insidiously, CUDA's memory pool architecture was working against the code. The msm_t constructor used gpu.Dmalloc() which called cudaMallocAsync, allocating from CUDA's asynchronous memory pool. When the MSM completed, gpu.Dfree() called cudaFreeAsync, returning memory to that same async pool. However, the next partition's dev_ptr_t or pre-staging code used synchronous cudaMalloc — which draws from a different pool. The freed memory was invisible to the next allocation request, causing the OOM even though sufficient VRAM was technically available.

The assistant's fix was twofold: move the pre-staging allocation inside the GPU mutex (serializing it across workers), and add a memory-aware allocator that queries cudaMemGetInfo, subtracts a 512 MiB safety margin, and falls back gracefully if insufficient VRAM is available. Additionally, d_bc is freed immediately after the NTT phase (before the mutex release), and cudaMemPoolTrimTo is called to ensure the async pool's freed memory is visible to subsequent synchronous allocations.

The Benchmark Message: What It Reveals

The subject message represents the first test of this memory-aware fix. The assistant deliberately chose gpu_workers_per_device=1 (single-worker mode) for this validation run, a conservative decision that eliminated the dual-worker contention variable. The configuration file set partition_workers = 10 and gpu_workers_per_device = 1, ensuring that only one GPU worker would attempt pre-staging at a time.

The benchmark results show three proofs completing successfully with wall times of 66.4s, 65.6s, and 65.0s — an average of approximately 65.7 seconds. The "prove" times (the actual GPU computation, excluding queue wait) were 33,068 ms, 30,828 ms, and 32,336 ms respectively, averaging about 32.1 seconds per proof. This represents a 14.2% throughput improvement over the Phase 8 baseline of 37.4 seconds per proof.

The significance of this result extends beyond the raw numbers. Every single proof completed without OOM failure, confirming that the memory-aware allocator was working correctly. The GPU timing logs (visible in the daemon log from the same run) showed dramatic kernel-level improvements: NTT+MSM time dropped from approximately 2,430 ms to 690 ms (a 71.6% reduction), tail MSM from 125 ms to 82 ms (a 34.4% reduction), and overall GPU time per partition from 3,746 ms to approximately 1,450–1,900 ms (a 50–61% reduction).

Assumptions and Their Validation

The message rested on several key assumptions, most of which were validated by the successful run. The primary assumption was that the memory-aware allocator — which queries cudaMemGetInfo, subtracts a safety margin, and conditionally pre-stages based on available VRAM — would prevent OOM failures. This was confirmed: the allocator successfully determined that 12 GiB of pre-stage buffers could fit within the available 14.3 GiB of free VRAM (after the 1.5 GiB CUDA context baseline), and proceeded with full pre-staging.

A secondary assumption was that single-worker mode would be sufficient to validate the PCIe optimization changes. This was a deliberate simplification: by eliminating dual-worker contention, the assistant could isolate the effects of pinned memory transfers and double-buffered MSM results. The assumption proved correct — the 14.2% throughput improvement in single-worker mode cleanly demonstrated the optimization's effectiveness.

However, the assistant also implicitly assumed that the 14.2% improvement would translate directly to the dual-worker configuration. As subsequent messages would reveal, this assumption was incorrect: the dual-worker benchmark (executed in the next chunk) showed a system throughput of 41.0 seconds per proof — significantly worse than the single-worker 32.1 seconds. This gap between kernel-level and system-level performance would become the next investigation target.

The Thinking Process Visible in the Message

The subject message itself is terse — a single bash command and its output — but it sits within a rich chain of reasoning visible in the surrounding messages. The assistant's thinking process is characterized by systematic hypothesis testing and iterative refinement.

When the initial OOM failures occurred (msg 2444), the assistant immediately pivoted to diagnosis. It examined the GPU memory state (nvidia-smi), traced the CUDA allocation paths, and identified the cudaMallocAsync/cudaMalloc pool mismatch. This required deep knowledge of CUDA's memory management internals — specifically that cudaMallocAsync uses memory pools that don't automatically release to the synchronous cudaMalloc pool.

The assistant then designed the memory-aware allocator with a specific safety philosophy: "With some 512MiB buffer left spare for other host processes" (as suggested by the user in msg 2445). This 512 MiB margin reflects an understanding that GPU memory is shared with the display server, CUDA runtime, and other processes — and that running at 100% utilization risks unpredictable failures.

The choice of gpu_workers_per_device=1 for this validation run reflects another layer of reasoning: the assistant needed to eliminate variables. The dual-worker configuration had introduced both the memory contention issue and potential PCIe bandwidth contention. By testing with a single worker first, the assistant could validate that the core PCIe optimization worked before tackling the more complex multi-worker scheduling problem.

Input Knowledge Required

Understanding this message requires knowledge spanning several domains. First, one must understand the Groth16 proving pipeline for Filecoin PoRep: that each proof requires processing polynomial evaluations through NTT and MSM operations, that the a/b/c vectors are 6 GiB of data that must be transferred from host to device memory, and that the proving engine partitions the work across multiple GPU workers.

Second, CUDA memory management knowledge is essential: the distinction between cudaMalloc (synchronous, system-wide pool) and cudaMallocAsync (per-stream pool), the behavior of cudaMemGetInfo, and the need for cudaMemPoolTrimTo to make freed async memory visible to synchronous allocations.

Third, the benchmark infrastructure itself requires understanding: the cuzk-bench tool, the --c1 flag pointing to a C1 output file, the -c 3 concurrency setting, and the -j 1 job count. The gpu_workers_per_device and partition_workers configuration parameters control how GPU work is distributed.

Output Knowledge Created

This message produced several critical pieces of knowledge. Most immediately, it confirmed that Phase 9's PCIe transfer optimization was working correctly in single-worker mode, achieving a 14.2% throughput improvement. The successful completion of all three proofs without OOM validated the memory-aware allocator design.

More subtly, the message established a new baseline for single-worker performance: approximately 32.1 seconds per proof for GPU computation, with an end-to-end wall time of about 65.7 seconds. The gap between GPU time (32.1s) and wall time (65.7s) revealed that CPU-side processing — partition synthesis, vector splitting, and daemon scheduling — now dominated the end-to-end latency. This insight would drive the next phase of optimization work.

The message also implicitly documented a boundary condition: the memory-aware allocator's behavior when VRAM is constrained. The allocator attempts full pre-staging (d_a + d_bc), falls back to d_a-only pre-staging if insufficient, and falls back to the original non-prestaged path if neither fits. This three-tier fallback strategy proved robust enough to handle the 16 GiB GPU with ~14.3 GiB free.

Conclusion

The benchmark message at index 2456 is far more than a simple test execution. It is the first successful validation of a complex optimization pipeline that required diagnosing CUDA memory pool internals, designing a memory-aware allocator with graceful fallback, and carefully isolating variables to confirm the optimization's effectiveness. The 14.2% throughput improvement it demonstrated validated weeks of analysis and implementation work. Yet the message also contained the seeds of the next investigation: the gap between kernel-level and system-level performance would soon reveal that PCIe bandwidth contention, not GPU compute, was the true bottleneck in the dual-worker configuration. This message thus stands at a pivot point — both a celebration of what was achieved and a prelude to the challenges ahead.