The Final Mile: How Precise Instrumentation and Systematic Design Resolved GPU Underutilization in the CuZK Proving Pipeline

Introduction

In the high-stakes world of GPU-accelerated zero-knowledge proof generation, every percentage point of utilization translates directly into throughput, cost, and competitive advantage. When the CuZK proving daemon—a sophisticated CUDA-accelerated proof generator for Filecoin—was discovered to be running at only 50% GPU utilization, the team embarked on a multi-session investigation that would span dozens of messages, hundreds of lines of instrumentation, and deep dives into Rust async runtimes, C++ mutex internals, and CUDA memory management. This article examines the final, decisive chunk of that investigation: the moment when raw timing data crystallized into a clear diagnosis, the design space for a zero-copy pinned memory pool was systematically explored, and the core components of the solution were implemented and validated.

This chunk represents the culmination of a debugging journey that began with mysterious idle gaps in GPU activity and ended with a concrete architectural blueprint for eliminating the Host-to-Device (H2D) transfer bottleneck. It is a story of disciplined investigation, collaborative problem-solving, and the power of precise instrumentation to reveal hidden performance killers.

The Mystery: A GPU That Should Be Busy But Isn't

To appreciate the significance of this chunk, one must understand the system under investigation. The CuZK proving daemon processes zero-knowledge proofs through a pipeline: CPU-based synthesis (constructing arithmetic circuit witnesses), GPU proving (executing number-theoretic transforms and multi-scalar multiplications), and finalization (serializing proofs and releasing memory). The system employs two GPU workers specifically designed to interleave PCIe transfers with GPU compute—while Worker A holds the GPU mutex and runs CUDA kernels, Worker B is supposed to perform CPU-side preprocessing, so that when Worker A releases the mutex, Worker B can immediately begin its GPU work. In theory, this should keep the GPU nearly continuously busy.

Yet the metrics told a different story. GPU utilization hovered around 50%, with the GPU bursting to 75–100% for roughly 1–2 seconds, then idling for 2–8 seconds. The prove_start_ms metric—measuring the total wall time of the C++ gpu_prove_start function—ranged from 4.2 seconds to over 16 seconds per partition, while the user confirmed that actual GPU compute was only 1.5–2 seconds per partition. Something was consuming 2–14 seconds of overhead per partition, and the GPU was sitting idle during most of it.

The Investigation: Systematic Elimination of Suspects

The preceding segments had already eliminated several plausible suspects. Precise Rust-side instrumentation using GPU_TIMING and FIN_TIMING log markers had shown that tracker lock contention was zero—the hot path through the Rust GPU worker loop contributed essentially no overhead. The malloc_trim calls that the team had suspected of causing delays under the tracker lock were measured and found to take 32–271ms in the finalizer, but this was off the critical path for GPU utilization. The Rust-side orchestration was not the problem.

The investigation then moved into the C++ codebase. By reading the CUDA kernel implementation in groth16_cuda.cu, the assistant reconstructed the internal phases of generate_groth16_proofs_start_c. The function has a split-phase design: a prep_msm_thread runs CPU preprocessing (classifying scalars, building bitmasks, populating tail MSM arrays) in parallel with a GPU thread that handles NTT and MSM operations. The two threads synchronize via a barrier, and a GPU mutex is held throughout the entire GPU thread's lifecycle.

The critical insight came from analyzing the CUZK_TIMING instrumentation that was already present in the C++ code. In message 3028, the assistant ran a remote SSH command to grep these logs from the proving machine:

CUZK_TIMING: gpu_tid=0 ntt_msm_h_ms=8860
CUZK_TIMING: gpu_tid=0 batch_add_ms=403
CUZK_TIMING: gpu_tid=0 tail_msm_ms=194 gpu_total_ms=9458
CUZK_TIMING: prep_msm_ms=4178

The data was revelatory. The ntt_msm_h_ms metric—measuring the time for NTT setup and MSM on the h polynomial, which includes H2D transfers—varied wildly from 2,699 ms to 8,860 ms. Meanwhile, batch_add_ms was a stable ~400 ms, tail_msm_ms was ~197 ms, and prep_msm_ms (CPU preprocessing) ranged from 1,488 ms to 4,178 ms. The actual GPU compute (batch_add + tail_msm + the kernel portion of ntt_msm_h) was roughly 1.2 seconds, consistent with the user's claim of 1.5–2 seconds per partition.

The smoking gun was the variation in ntt_msm_h_ms. A 3x swing (2.7s to 8.9s) pointed directly to memory bandwidth contention. When multiple synthesis threads were running simultaneously, they competed for host memory bandwidth, which slowed the H2D PCIe transfers that occurred inside ntt_msm_h.

The Confirmation: nvtop Reveals the PCIe Bandwidth Story

The user's observation at message 3041 provided the critical corroborating evidence: "during gaps nvtop rx (to device) is only 1-4GB/s, it bursts to 50GB/s during compute activity." The assistant immediately recognized the implication. The 50 GB/s bursts corresponded to MSM operations reading SRS points from cudaHostAlloc'd pinned memory, which achieves near-PCIe-Gen5 line rate. The 1-4 GB/s during gaps was the H2D copy of a/b/c vectors from regular heap memory, where the CUDA driver must stage through a small pinned bounce buffer, throttling throughput by 10–50×.

This was the root cause. The a/b/c synthesis vectors—the A, B, and C polynomial evaluations that form the backbone of a Groth16 proof—were standard heap allocations. When CUDA performed cudaMemcpy from unpinned host memory, the driver had to stage the data through a small pinned bounce buffer, achieving only 1–4 GB/s. The SRS points, being already pinned via cudaHostAlloc, enjoyed direct DMA access at the full PCIe Gen5 x16 line rate of approximately 50 GB/s.

The asymmetry was stark and the solution clear: the a/b/c vectors needed to be in pinned memory.

The Pivot: From Diagnosis to Design

Message 3051 marks the decisive turning point. The user wrote: "Investigate pinning the vectors, presumably by extending memorymanager with dynamic pinned memory manager (pinned mem alloc is expensive and slow by itself so we probbaly need a pool which we reuse and only free when e.g. we need memory for srs/pce.. which is also pinned?)"

This single message crystallizes the root cause, proposes the architectural solution, and anticipates the key engineering challenge—all in a few sentences. It is the moment the team pivoted from diagnosis to design.

The assistant's response in message 3052 was a structured todowrite call that decomposed the high-level directive into concrete research tasks: investigate current pinned memory usage and costs, understand a/b/c vector sizes and lifecycle, check how synthesis allocates the vectors, and design the pinned memory pool. This systematic decomposition reflects sound engineering practice: design decisions should be informed by data, not made in a vacuum.

The Design Space Exploration

Message 3062 contains the most extensive reasoning in the chunk—a deep design exploration that weighs multiple approaches against each other. The assistant identified four distinct options:

Option A: Custom Allocator for Vec. Create a custom Rust allocator that redirects Vec<Fr> allocations to a pinned memory pool. This would require zero changes to synthesis code but runs into Rust's allocator limitations—the GlobalAlloc trait operates at the level of Layout, not at the level of pre-allocated pools with specific lifetimes. Moreover, Vec can reallocate, which would break the pinned memory contract. This option was quickly discarded.

Option B: Post-Synthesis Registration. After synthesis completes, register the existing Vec data as pinned memory using cudaHostRegister, then unregister after the GPU transfer. The assistant's research had revealed that cudaHostRegister is expensive—potentially costing as much as the transfer itself. For 2+ GiB buffers, this registration cost could add seconds to the pipeline, effectively doubling the H2D transfer time rather than eliminating it.

Option C: Staged Copy Through Pinned Buffers. Maintain a pool of pre-allocated pinned buffers and copy the a/b/c vectors into them before calling prove_start. The assistant calculated that at DDR5 speeds, copying 8 GiB of data takes approximately 80 milliseconds—a trivial cost compared to the current 2–9 second overhead. The key insight was that the memcpy could be performed before acquiring the GPU mutex, allowing it to overlap with other workers' GPU compute. This approach avoids modifying bellperson's ProvingAssignment and keeps all changes within cuzk-core.

Option D: Direct Synthesis into Pinned Memory. Synthesize directly into pinned buffers from the pool, eliminating the copy entirely. This requires knowing the buffer sizes upfront during ProvingAssignment::new_with_capacity. The assistant noted that the capacity hint system—already implemented in the codebase—provides exact sizes after the first synthesis run for a given circuit type. This means the pinned buffers can be pre-allocated to exactly the right size, and the synthesis can push() into them directly.

The assistant leaned toward Option D as the primary approach, but the reasoning revealed a deep appreciation for the complications. The fundamental challenge was ownership: when a Vec is constructed from a pre-allocated buffer using Vec::from_raw_parts, Rust's standard Drop implementation will attempt to deallocate the memory through the global allocator. Since the memory was allocated via cudaHostAlloc, this would cause undefined behavior. The assistant identified three possible solutions: ManuallyDrop wrappers, a custom PinnedVec type with a safe Drop implementation, or the staged copy approach (Option C) that sidesteps the ownership problem entirely.

The Constraint That Was Holding Things Back

Throughout the design exploration, one constraint loomed large: pinning too much system memory risks degrading overall system performance. The conventional wisdom in CUDA programming is that pinning more than 30–50% of available RAM causes OS paging issues, as pinned pages cannot be swapped out. The SRS already consumed approximately 44 GiB of pinned memory. Adding 8–13 GiB per concurrent partition for pinned a/b/c buffers would push the total pinned allocation dangerously close to that threshold. This constraint forced the design toward conservative strategies: smaller pools, careful eviction policies, and complex budget tracking.

Then came the user's clarification in message 3063: "Note most system memory is used just for this workload, no other workloads using system RAM."

These seven words dismantled the core assumption that had been shaping the assistant's architectural reasoning. The machine was a dedicated proving node with 755 GiB of RAM, provisioned specifically to serve the CuZK pipeline. No other workloads competed for memory. The OS paging concern that had been constraining the design simply did not apply.

The assistant's response in message 3064 demonstrated immediate recognition of this shift: "Good — that simplifies things. No need to worry about pinning >50% of RAM starving other processes. We can pin aggressively." The word "aggressively" signaled a departure from the cautious, multi-option evaluation that characterized the earlier design phase. The constraint that had forced consideration of complex eviction policies, pool shrinking mechanisms, and budget trade-offs was gone.

Tracing the Injection Point

With the strategic direction confirmed, the assistant moved to trace the exact code path where pinned buffers would be injected. Message 3064 read the prove_start function signature in bellperson/src/groth16/prover/supraseal.rs, revealing the critical interface:

pub fn prove_start<E, P: ParameterSource<E>>(
    provers: Vec<ProvingAssignment<E::Fr>>,
    input_assignments: Vec<Arc<Vec<E::Fr>>>,
    aux_assignments: Vec<Arc<Vec<E::Fr>>>,
    params: P,
    r_s: Vec<E::Fr>,
    s_s: Vec<E::Fr>,
    gpu_mtx: GpuMutexPtr,
    gpu_index: i32,
) -> Result<PendingProofHandle<E>, SynthesisError>

The key parameter is provers: Vec<ProvingAssignment<E::Fr>>. Each ProvingAssignment contains the a, b, and c vectors—the very data that needed to reside in pinned memory. The prove_start function is the gateway to the C++ gpu_prove_start routine, which extracts raw pointers from these vectors and passes them to the CUDA kernels. If pinned buffers could be injected at this boundary—either by having ProvingAssignment use pinned backing directly, or by copying into pinned staging buffers before calling this function—the H2D bottleneck would be eliminated.

The assistant then traced the data flow further upstream, reading the SynthesizedProof struct in messages 3066–3068. This struct is the bridge between synthesis and GPU proving, containing provers: Vec<ProvingAssignment<Fr>> along with input and auxiliary witness vectors. Understanding its structure was essential for determining where and how to inject pinned memory buffers into the pipeline.

The Implementation: PinnedPool and PinnedBacking

The design that emerged from this investigation was a PinnedPool struct integrated with the MemoryBudget system, backed by cudaHostAlloc'd buffers managed through a free list. The PinnedPool would pre-allocate buffers sized for each circuit type (SnapDeals, PoRep) and reuse them across partitions, avoiding the per-allocation overhead of cudaHostAlloc.

The core innovation was the PinnedBacking struct, which safely holds and releases cudaHostAlloc buffers. Rather than modifying bellperson's ProvingAssignment to use a custom PinnedVec type (which would ripple through the entire synthesis code), the assistant designed a mechanism where the ProvingAssignment could optionally hold pinned backing that would be released after the GPU transfer completed. This was achieved by extending ProvingAssignment with a pinned_backing field and a release_abc method that returns the pinned buffers to the pool.

The implementation was validated with cargo check, confirming that the new types compiled cleanly with the existing codebase. The PinnedPool struct, the PinnedBacking struct, and the modifications to bellperson's ProvingAssignment were all in place, positioning the team to wire the pool into the engine and complete the integration.

The Knowledge Created

This chunk produced several forms of knowledge that transformed the investigation:

A confirmed root cause. The H2D transfer of unpinned a/b/c vectors was definitively identified as the primary bottleneck causing GPU underutilization. The timing data showed ntt_msm_h varying 3x (2.7s to 8.9s) while actual GPU compute remained a stable ~1.2s. The nvtop observations confirmed the mechanism: RX bandwidth dropped to 1–4 GB/s during gaps (heap allocations) and burst to 50 GB/s during compute (pinned SRS).

A solution architecture. The zero-copy pinned memory pool design, integrated with the MemoryBudget system, provided a clear path forward. The pool would pre-allocate cudaHostAlloc'd buffers, reuse them across partitions, and safely release them when higher-priority allocations (SRS, PCE) needed memory.

A safety analysis. The assistant identified the undefined behavior risk of using Vec::from_raw_parts with cudaHostAlloc'd memory and proposed mitigation strategies using ManuallyDrop and custom Drop implementations.

A validated implementation. The core components (PinnedPool, PinnedBacking, release_abc) were implemented and compiled successfully with cargo check, providing confidence that the design was sound.

The Collaborative Pattern

Throughout this chunk, a distinctive pattern of human-AI collaboration emerges. The AI assistant excels at systematic reasoning: tracing code paths, weighing design alternatives, identifying safety risks, and producing implementation code. The user excels at providing domain knowledge that the AI cannot infer from the codebase alone: the system's workload isolation (message 3063), the two-worker interleaving design intent (message 3009), and the actual GPU compute time per partition (message 3007).

This division of labor is effective because each party contributes what it does best. The AI's strength is navigating a complex design space once the constraints are known; the user's strength is knowing what those constraints actually are. The seven-word clarification that "most system memory is used just for this workload" is a perfect example: the AI could not have inferred this from the code, but once told, it could immediately adjust its entire design approach.

Conclusion

This chunk represents the culmination of a multi-session investigation into GPU underutilization in the CuZK proving daemon. The team moved from mystery to diagnosis to design to implementation through a process of systematic instrumentation, hypothesis testing, and collaborative reasoning. The root cause—the H2D transfer bottleneck caused by heap-allocated synthesis vectors—was identified through precise timing data and confirmed by nvtop observations. The solution—a zero-copy pinned memory pool integrated with the MemoryBudget system—was designed through careful exploration of multiple approaches and implemented with attention to Rust's safety guarantees.

The story of this chunk is a testament to the power of disciplined engineering investigation. When faced with a complex performance problem spanning Rust async runtimes, C++ mutex internals, and CUDA memory management, the team did not guess or speculate. They instrumented precisely, measured carefully, and let the data guide them to the root cause. They explored the design space systematically, weighing trade-offs against constraints. And when a critical constraint was removed by user domain knowledge, they pivoted instantly to a more aggressive and effective solution.

The GPU utilization that had been stuck at 50% was about to be unlocked. The zero-copy pinned memory pool would collapse the H2D transfer from seconds to milliseconds, finally achieving the near-100% GPU utilization that the two-worker interleaving design was meant to deliver.