Zero-Copy Pinned Memory: Resolving GPU Underutilization in the CuZK Proving Pipeline

Introduction

In the high-stakes world of GPU-accelerated zero-knowledge proof generation, every percentage point of GPU utilization translates directly into throughput, cost, and competitive advantage. When the CuZK proving daemon—a sophisticated CUDA-accelerated proof generator for Filecoin—was discovered to be running at only 50% GPU utilization, the team embarked on a multi-session investigation that would span dozens of messages, hundreds of lines of instrumentation, and deep dives into Rust async runtimes, C++ mutex internals, and CUDA memory management.

Segment 22 represents the culmination of that investigation: the moment when raw timing data crystallized into a clear diagnosis, the design space for a zero-copy pinned memory pool was systematically explored, and the core components of the solution were implemented and validated with a successful cargo check. This segment is a story of disciplined investigation, collaborative problem-solving, and the power of precise instrumentation to reveal hidden performance killers.

The segment is documented across two chunk articles: [1] covers the investigative phase—the precise instrumentation that identified the Host-to-Device (H2D) transfer bottleneck as the root cause of GPU underutilization—while [2] covers the implementation phase—the design and construction of a PinnedPool struct, the PinnedBacking mechanism for safely holding CUDA-allocated memory in Rust vectors, and the modifications to bellperson's ProvingAssignment to support direct synthesis into DMA-able buffers. Together, they tell a complete story of moving from mystery to diagnosis to solution.

The Mystery: A GPU That Should Be Busy But Isn't

To appreciate the significance of this segment, one must understand the system under investigation. The CuZK proving daemon processes zero-knowledge proofs through a pipeline: CPU-based synthesis (constructing arithmetic circuit witnesses), GPU proving (executing number-theoretic transforms and multi-scalar multiplications), and finalization (serializing proofs and releasing memory). The system employs two GPU workers specifically designed to interleave PCIe transfers with GPU compute—while Worker A holds the GPU mutex and runs CUDA kernels, Worker B is supposed to perform CPU-side preprocessing, so that when Worker A releases the mutex, Worker B can immediately begin its GPU work. In theory, this should keep the GPU nearly continuously busy.

Yet the metrics told a different story. GPU utilization hovered around 50%, with the GPU bursting to 75–100% for roughly 1–2 seconds, then idling for 2–8 seconds. The prove_start_ms metric—measuring the total wall time of the C++ gpu_prove_start function—ranged from 4.2 seconds to over 16 seconds per partition, while the user confirmed that actual GPU compute was only 1.5–2 seconds per partition. Something was consuming 2–14 seconds of overhead per partition, and the GPU was sitting idle during most of it.

The preceding segments had already eliminated several plausible suspects. Precise Rust-side instrumentation using GPU_TIMING and FIN_TIMING log markers had shown that tracker lock contention was zero—the hot path through the Rust GPU worker loop contributed essentially no overhead. The malloc_trim calls that the team had suspected of causing delays under the tracker lock were measured and found to take 32–271ms in the finalizer, but this was off the critical path for GPU utilization. The Rust-side orchestration was not the problem.

The Investigation: Systematic Elimination of Suspects

The investigation then moved into the C++ codebase. By reading the CUDA kernel implementation in groth16_cuda.cu, the assistant reconstructed the internal phases of generate_groth16_proofs_start_c. The function has a split-phase design: a prep_msm_thread runs CPU preprocessing (classifying scalars, building bitmasks, populating tail MSM arrays) in parallel with a GPU thread that handles NTT and MSM operations. The two threads synchronize via a barrier, and a GPU mutex is held throughout the entire GPU thread's lifecycle.

The critical insight came from analyzing the CUZK_TIMING instrumentation that was already present in the C++ code. The assistant ran a remote SSH command to grep these logs from the proving machine, revealing a pattern that would unlock the entire investigation:

CUZK_TIMING: gpu_tid=0 ntt_msm_h_ms=8860
CUZK_TIMING: gpu_tid=0 batch_add_ms=403
CUZK_TIMING: gpu_tid=0 tail_msm_ms=194 gpu_total_ms=9458
CUZK_TIMING: prep_msm_ms=4178

The data was revelatory. The ntt_msm_h_ms metric—measuring the time for NTT setup and MSM on the h polynomial, which includes H2D transfers—varied wildly from 2,699 ms to 8,860 ms. Meanwhile, batch_add_ms was a stable ~400 ms, tail_msm_ms was ~197 ms, and prep_msm_ms (CPU preprocessing) ranged from 1,488 ms to 4,178 ms. The actual GPU compute (batch_add + tail_msm + the kernel portion of ntt_msm_h) was roughly 1.2 seconds, consistent with the user's claim of 1.5–2 seconds per partition.

The smoking gun was the variation in ntt_msm_h_ms. A 3x swing (2.7s to 8.9s) pointed directly to memory bandwidth contention. When multiple synthesis threads were running simultaneously, they competed for host memory bandwidth, which slowed the H2D PCIe transfers that occurred inside ntt_msm_h.

The Confirmation: nvtop Reveals the PCIe Bandwidth Story

The user's observation provided the critical corroborating evidence: during gaps, nvtop RX (receive to device) was only 1–4 GB/s, but it burst to 50 GB/s during compute activity. The assistant immediately recognized the implication. The 50 GB/s bursts corresponded to MSM operations reading SRS points from cudaHostAlloc'd pinned memory, which achieves near-PCIe-Gen5 line rate. The 1–4 GB/s during gaps was the H2D copy of a/b/c vectors from regular heap memory, where the CUDA driver must stage through a small pinned bounce buffer, throttling throughput by 10–50×.

This was the root cause. The a/b/c synthesis vectors—the A, B, and C polynomial evaluations that form the backbone of a Groth16 proof—were standard heap allocations. When CUDA performed cudaMemcpy from unpinned host memory, the driver had to stage the data through a small pinned bounce buffer, achieving only 1–4 GB/s. The SRS points, being already pinned via cudaHostAlloc, enjoyed direct DMA access at the full PCIe Gen5 x16 line rate of approximately 50 GB/s.

The asymmetry was stark and the solution clear: the a/b/c vectors needed to be in pinned memory.

The Pivot: From Diagnosis to Design

Message 3051 marks the decisive turning point in the segment. The user wrote: "Investigate pinning the vectors, presumably by extending memorymanager with dynamic pinned memory manager (pinned mem alloc is expensive and slow by itself so we probbaly need a pool which we reuse and only free when e.g. we need memory for srs/pce.. which is also pinned?)"

This single message crystallizes the root cause, proposes the architectural solution, and anticipates the key engineering challenge—all in a few sentences. It is the moment the team pivoted from diagnosis to design. The assistant's response was a structured decomposition of the high-level directive into concrete research tasks: investigate current pinned memory usage and costs, understand a/b/c vector sizes and lifecycle, check how synthesis allocates the vectors, and design the pinned memory pool. This systematic decomposition reflects sound engineering practice: design decisions should be informed by data, not made in a vacuum.

The Design Space Exploration

The assistant's extensive reasoning in message 3062 contains a deep design exploration that weighs multiple approaches against each other. Four distinct options were identified and evaluated:

Option A: Custom Allocator for Vec. Create a custom Rust allocator that redirects Vec<Fr> allocations to a pinned memory pool. This would require zero changes to synthesis code but runs into Rust's allocator limitations—the GlobalAlloc trait operates at the level of Layout, not at the level of pre-allocated pools with specific lifetimes. Moreover, Vec can reallocate, which would break the pinned memory contract. This option was quickly discarded.

Option B: Post-Synthesis Registration. After synthesis completes, register the existing Vec data as pinned memory using cudaHostRegister, then unregister after the GPU transfer. The assistant's research had revealed that cudaHostRegister is expensive—potentially costing as much as the transfer itself. For 2+ GiB buffers, this registration cost could add seconds to the pipeline, effectively doubling the H2D transfer time rather than eliminating it.

Option C: Staged Copy Through Pinned Buffers. Maintain a pool of pre-allocated pinned buffers and copy the a/b/c vectors into them before calling prove_start. The assistant calculated that at DDR5 speeds, copying 8 GiB of data takes approximately 80 milliseconds—a trivial cost compared to the current 2–9 second overhead. The key insight was that the memcpy could be performed before acquiring the GPU mutex, allowing it to overlap with other workers' GPU compute. This approach avoids modifying bellperson's ProvingAssignment and keeps all changes within cuzk-core.

Option D: Direct Synthesis into Pinned Memory. Synthesize directly into pinned buffers from the pool, eliminating the copy entirely. This requires knowing the buffer sizes upfront during ProvingAssignment::new_with_capacity. The assistant noted that the capacity hint system—already implemented in the codebase—provides exact sizes after the first synthesis run for a given circuit type. This means the pinned buffers can be pre-allocated to exactly the right size, and the synthesis can push() into them directly.

The assistant leaned toward Option D as the primary approach, but the reasoning revealed a deep appreciation for the complications. The fundamental challenge was ownership: when a Vec is constructed from a pre-allocated buffer using Vec::from_raw_parts, Rust's standard Drop implementation will attempt to deallocate the memory through the global allocator. Since the memory was allocated via cudaHostAlloc, this would cause undefined behavior. The assistant identified three possible solutions: ManuallyDrop wrappers, a custom PinnedVec type with a safe Drop implementation, or the staged copy approach (Option C) that sidesteps the ownership problem entirely.

The Constraint That Was Holding Things Back

Throughout the design exploration, one constraint loomed large: pinning too much system memory risks degrading overall system performance. The conventional wisdom in CUDA programming is that pinning more than 30–50% of available RAM causes OS paging issues, as pinned pages cannot be swapped out. The SRS already consumed approximately 44 GiB of pinned memory. Adding 8–13 GiB per concurrent partition for pinned a/b/c buffers would push the total pinned allocation dangerously close to that threshold. This constraint forced the design toward conservative strategies: smaller pools, careful eviction policies, and complex budget tracking.

Then came the user's clarification in message 3063: "Note most system memory is used just for this workload, no other workloads using system RAM."

These seven words dismantled the core assumption that had been shaping the assistant's architectural reasoning. The machine was a dedicated proving node with 755 GiB of RAM, provisioned specifically to serve the CuZK pipeline. No other workloads competed for memory. The OS paging concern that had been constraining the design simply did not apply.

The assistant's response in message 3064 demonstrated immediate recognition of this shift: "Good — that simplifies things. No need to worry about pinning >50% of RAM starving other processes. We can pin aggressively." The word "aggressively" signaled a departure from the cautious, multi-option evaluation that characterized the earlier design phase. The constraint that had forced consideration of complex eviction policies, pool shrinking mechanisms, and budget trade-offs was gone.## Tracing the Injection Point

With the strategic direction confirmed, the assistant moved to trace the exact code path where pinned buffers would be injected. Message 3064 read the prove_start function signature in bellperson/src/groth16/prover/supraseal.rs, revealing the critical interface:

pub fn prove_start<E, P: ParameterSource<E>>(
    provers: Vec<ProvingAssignment<E::Fr>>,
    input_assignments: Vec<Arc<Vec<E::Fr>>>,
    aux_assignments: Vec<Arc<Vec<E::Fr>>>,
    params: P,
    r_s: Vec<E::Fr>,
    s_s: Vec<E::Fr>,
    gpu_mtx: GpuMutexPtr,
    gpu_index: i32,
) -> Result<PendingProofHandle<E>, SynthesisError>

The key parameter is provers: Vec&lt;ProvingAssignment&lt;E::Fr&gt;&gt;. Each ProvingAssignment contains the a, b, and c vectors—the very data that needed to reside in pinned memory. The prove_start function is the gateway to the C++ gpu_prove_start routine, which extracts raw pointers from these vectors and passes them to the CUDA kernels. If pinned buffers could be injected at this boundary—either by having ProvingAssignment use pinned backing directly, or by copying into pinned staging buffers before calling this function—the H2D bottleneck would be eliminated.

The assistant then traced the data flow further upstream, reading the SynthesizedProof struct. This struct is the bridge between synthesis and GPU proving, containing provers: Vec&lt;ProvingAssignment&lt;Fr&gt;&gt; along with input and auxiliary witness vectors. Understanding its structure was essential for determining where and how to inject pinned memory buffers into the pipeline.

The Build System Reconnaissance

Before writing any code, the assistant needed to understand the existing CUDA FFI infrastructure. A grep for cudaHostAlloc|cudaFreeHost|cudaMallocHost|cuda_host returned no results, confirming that no pinned memory allocation bindings existed anywhere in the codebase. This meant the assistant would have to build from scratch.

The next step was understanding how CUDA code was compiled. A glob for **/build.rs found a single result at /tmp/czk/extern/supraseal-c2/build.rs. Reading that file revealed that the crate uses sppark::build::ccmd() to compile a single CUDA source file, groth16_cuda.cu. This confirmed that CUDA was already integrated into the build system and that new CUDA API bindings could be added following the same pattern.

The assistant then read cuzk-core/Cargo.toml to verify the dependency chain. The feature flag cuda-supraseal being the default confirmed that CUDA support was enabled by default. The presence of libc = &#34;0.2&#34; indicated that raw C FFI was already in use, making CUDA FFI additions architecturally consistent. The ec-gpu-gen dependency confirmed that the scalar field types (Fr) are Copy types with no destructors—a critical safety invariant for the ManuallyDrop approach.

With this reconnaissance complete, the assistant declared "OK, now I have enough context" and laid out the concrete architecture:

cuzk-core/src/pinned_pool.rs  — PinnedPool (cudaHostAlloc/Free, budget-integrated)
bellperson/src/groth16/prover/mod.rs — PinnedBacking + release_abc on ProvingAssignment
bellperson/src/groth16/prover/supraseal.rs — prove_start uses release_abc
cuzk-core/src/pipeline.rs — synthesis checks out pinned buffers, passes to ProvingAssignment
cuzk-core/src/engine.rs — creates pool, evictor integration

Implementing the PinnedPool

The PinnedPool struct was written to cuzk-core/src/pinned_pool.rs. It manages cudaHostAlloc'd buffers with a free list tracking (pointer, size) pairs. It integrates with the existing MemoryBudget system via try_acquire: when allocating a new buffer, the pool acquires budget upfront; when freeing, it releases the budget. The pool uses exact-size allocation rather than pre-sized pools for each circuit type, naturally handling mixed workloads (SnapDeals at ~2.59 GiB per buffer, PoRep at ~4.17 GiB).

However, the initial implementation contained a subtle API mismatch. The try_acquire method on MemoryBudget uses an unusual Rust pattern: pub fn try_acquire(self: &amp;Arc&lt;Self&gt;, amount: u64) -&gt; Option&lt;MemoryReservation&gt;. The assistant had assumed a conventional signature and written code that didn't match. Reading the actual API revealed the correct pattern, and the assistant applied a fix. This moment of self-correction—catching the mistake before compilation—demonstrates the value of verifying API signatures against source code rather than relying on assumptions.

The read also revealed the Two-Phase Release contract for a/b/c memory: after prove_start, the vectors are freed and ~12.5 GiB is returned to the budget; after prove_finish, the remaining ~1.1 GiB is released. This documentation confirmed that the pool's buffer checkout/checkin lifecycle could be mapped onto the existing budget protocol without conflict.

The Module Registration

With the PinnedPool implemented, the assistant registered it in cuzk-core/src/lib.rs. This single-line edit—pub mod pinned_pool;—marked the transition from design to implementation. Before this edit, pinned_pool.rs was a standalone file, invisible to the compiler's module resolution. After this edit, the PinnedPool type became part of the crate's public interface, accessible to engine.rs, pipeline.rs, and any other module that needs to allocate pinned memory.

The Critical Bridge: Modifying ProvingAssignment

The most delicate part of the implementation was modifying bellperson's ProvingAssignment to safely hold and release pinned memory buffers. This required bridging two separate codebases—cuzk-core (where the pool lives) and bellperson (where the vectors are synthesized)—without introducing undefined behavior or breaking the public API.

The assistant read the ProvingAssignment struct definition, confirming that a, b, and c are declared as plain Vec&lt;Scalar&gt;. This was the "before" snapshot that the entire pinned memory effort sought to transform. The struct is generic over Scalar: PrimeField, which meant the pinned backing mechanism must be generic as well.

The PinnedBacking struct was added to hold the raw pinned buffer pointers and a return callback (Box&lt;dyn FnOnce(Vec&lt;*mut u8&gt;)&gt;). The ProvingAssignment struct was extended with a pinned_backing: Option&lt;PinnedBacking&gt; field. This design keeps bellperson CUDA-free while allowing cuzk-core to provide the pool integration through the callback mechanism.

The assistant then added three methods to ProvingAssignment:

  1. new_with_pinned(): An alternative constructor that accepts pre-allocated pinned buffers and constructs Vec&lt;Scalar&gt; instances via Vec::from_raw_parts(pinned_ptr, 0, capacity). This allows synthesis to push directly into DMA-able memory.
  2. release_abc(): Uses std::mem::take to replace each Vec with an empty one (whose drop is a no-op), then invokes the return callback to recycle the buffers. This prevents the global allocator from attempting to free pinned memory.
  3. Custom Drop implementation: A safety net that calls release_abc() if the pinned backing is still present when the struct is dropped. This ensures that even if release_abc() was never called explicitly (e.g., due to an error path), the pinned buffers are properly returned to the pool rather than being deallocated through the wrong allocator. The assistant also added pinned_backing: None to all existing constructors, including the ConstraintSystem::new() method. This ensured that existing code paths (native proving, non-CUDA builds) continue to work without modification.

The Final Stitch: Wiring release_abc into prove_start

The final integration step was updating the prove_start function in bellperson/src/groth16/prover/supraseal.rs to call release_abc() after extracting pointers. The prove_start function is the Rust-side entry point that invokes the C++ GPU proving code. It extracts raw pointers from the a, b, and c vectors and passes them to the C++ layer for GPU computation.

After the pointers have been extracted and handed off to the GPU, the Rust-side vectors are no longer needed by the proving logic. However, they cannot simply be dropped: because their backing memory was allocated via cudaHostAlloc through the PinnedPool, a normal Vec::drop would attempt to free the memory through the global allocator, causing undefined behavior. The release_abc() method must be called instead, which safely returns the buffers to the pool for reuse.

This edit is the critical handshake between the new pinned-memory infrastructure and the existing GPU proving pathway. Without it, the entire zero-copy architecture would be incomplete—the buffers would never be returned to the pool, and the pool would quickly exhaust its capacity, or worse, the program would crash on the next deallocation.

Validation: The Checkpoint

With all the core components implemented, the assistant ran cargo check --features cuda-supraseal to validate the changes. The output revealed only a pre-existing visibility error in engine.rs concerning JobTracker and process_partition_result—entirely unrelated to the newly written pinned memory code. The fact that no new errors appeared from the PinnedPool implementation in cuzk-core, the PinnedBacking modifications in bellperson, or the prove_start changes in supraseal.rs was the quiet confirmation that the core architectural changes compile cleanly.

This checkpoint validated several critical assumptions:

The Architecture in Retrospect

Looking back at the implementation, several design decisions stand out as particularly elegant:

The ManuallyDrop pattern: By keeping a/b/c as Vec&lt;Scalar&gt; and adding an optional PinnedBacking field, the assistant avoided changing field types across the entire codebase. The release_abc() method uses std::mem::take to replace each Vec with an empty one, then ManuallyDrop::take to extract and forget the original contents. This is a minimal-change strategy that reduces the risk of introducing bugs in unrelated code paths.

The callback-based abstraction: The PinnedBacking struct stores a Box&lt;dyn FnOnce(Vec&lt;*mut u8&gt;)&gt; callback rather than a direct pool reference. This keeps bellperson CUDA-free while allowing cuzk-core to provide the pool integration. The callback is registered at startup by the higher-level engine layer, which has access to CUDA and the PinnedPool.

The Drop safety net: The custom Drop implementation on ProvingAssignment transforms a fragile manual protocol into a robust RAII pattern. If release_abc() was never called explicitly (e.g., due to an error path, a refactor, or a logic bug), the Drop impl ensures the pinned buffers are returned to the pool rather than being deallocated through the wrong allocator.

The budget integration: The pool holds permanent reservations for allocated buffers, releasing them only when buffers are freed through shrinkage or eviction. Checkout and checkin of buffers from the pool do not touch the budget at all—the reservation is already held by the pool. This cleanly separates buffer lifecycle from budget accounting.

Assumptions and Risks

The implementation makes several assumptions worth examining:

That Fr (the scalar field element) is Copy with no destructor: The ManuallyDrop::take approach "forgets" the elements stored in the Vec. For scalar field elements like Fr, which are plain data with no destructors, this is safe. But if the generic Scalar type were ever instantiated with a type that has drop glue, this would leak resources. The assistant explicitly notes this assumption in the reasoning.

That the callback mechanism is reliable: The PinnedBacking struct stores a callback that returns buffers to the pool. If the callback is never called (e.g., due to a panic during synthesis that bypasses the Drop impl), the buffer leaks. The Drop safety net mitigates this for normal control flow, but double-panics or abort-on-panic configurations could still cause leaks.

That Vec::from_raw_parts with a CUDA-allocated pointer is valid: The Rust specification requires that pointers passed to Vec::from_raw_parts come from the correct allocator. CUDA's cudaHostAlloc returns host-pinned memory that is also valid for CPU access, so reading and writing through the Vec's pointer is safe. But the allocator mismatch is precisely why the ManuallyDrop trick is needed—the Vec must never be allowed to call its own dealloc.

That the pool's free list is efficient under contention: The pool uses a Mutex&lt;Vec&lt;...&gt;&gt; free list. Under heavy concurrent synthesis (20+ threads), contention on this mutex could become a bottleneck. The pool's design minimizes checkout/checkin frequency (once per partition, not per element), which mitigates this concern.

The Road Ahead

With the core pinned memory integration compiling cleanly, the team is positioned to complete the remaining integration work. The PinnedPool must be created at engine startup, integrated with the evictor to handle memory pressure, and passed through the synthesis dispatch so that every partition's a/b/c vectors are allocated from pinned memory. The pre-existing JobTracker visibility error must also be fixed, but it is a straightforward visibility change unrelated to the pinned memory work.

The zero-copy pipeline promises to collapse the H2D transfer from seconds to milliseconds, dramatically improving GPU utilization by allowing the NTT setup to overlap cleanly with the compute phases of other partitions. When fully integrated and deployed, this optimization could double the proving throughput of the entire cuzk system—a transformative improvement born from careful instrumentation, rigorous design reasoning, and meticulous implementation.

The Collaborative Pattern

Throughout this segment, a distinctive pattern of human-AI collaboration emerges. The AI assistant excels at systematic reasoning: tracing code paths, weighing design alternatives, identifying safety risks, and producing implementation code. The user excels at providing domain knowledge that the AI cannot infer from the codebase alone: the system's workload isolation, the two-worker interleaving design intent, and the actual GPU compute time per partition.

This division of labor is effective because each party contributes what it does best. The AI's strength is navigating a complex design space once the constraints are known; the user's strength is knowing what those constraints actually are. The seven-word clarification that "most system memory is used just for this workload" is a perfect example: the AI could not have inferred this from the code, but once told, it could immediately adjust its entire design approach.

Conclusion

Segment 22 represents the culmination of a multi-session investigation into GPU underutilization in the CuZK proving daemon. The team moved from mystery to diagnosis to design to implementation through a process of systematic instrumentation, hypothesis testing, and collaborative reasoning. The root cause—the H2D transfer bottleneck caused by heap-allocated synthesis vectors—was identified through precise timing data and confirmed by nvtop observations. The solution—a zero-copy pinned memory pool integrated with the MemoryBudget system—was designed through careful exploration of multiple approaches and implemented with attention to Rust's safety guarantees.

The story of this segment is a testament to the power of disciplined engineering investigation. When faced with a complex performance problem spanning Rust async runtimes, C++ mutex internals, and CUDA memory management, the team did not guess or speculate. They instrumented precisely, measured carefully, and let the data guide them to the root cause. They explored the design space systematically, weighing trade-offs against constraints. And when a critical constraint was removed by user domain knowledge, they pivoted instantly to a more aggressive and effective solution.

The GPU utilization that had been stuck at 50% was about to be unlocked. The zero-copy pinned memory pool would collapse the H2D transfer from seconds to milliseconds, finally achieving the near-100% GPU utilization that the two-worker interleaving design was meant to deliver.

References

[1] "The Final Mile: How Precise Instrumentation and Systematic Design Resolved GPU Underutilization in the CuZK Proving Pipeline" — Chunk article covering the investigative phase of Segment 22.

[2] "Building the Zero-Copy Bridge: Implementing a Pinned Memory Pool for GPU Proving" — Chunk article covering the implementation phase of Segment 22.