The Pinned Memory Pool Decision: A Pivot Point in GPU Utilization Optimization

Introduction

In the course of optimizing a GPU-accelerated zero-knowledge proving pipeline, a single message crystallized weeks of investigation, instrumentation, and analysis into a concrete design choice. Message [msg 3069] is the moment where the assistant, having traced GPU underutilization to a Host-to-Device (H2D) memory transfer bottleneck, presents a formal design for a pinned memory pool and asks the user to choose between two implementation strategies. This message is not merely a status update—it is a decision node that determines the architecture of the entire memory management subsystem for the proving engine.

To understand the significance of this message, one must appreciate the investigative journey that preceded it. The team had been battling a persistent ~50% GPU utilization problem in the cuzk proving daemon. After ruling out initial suspects like tracker lock contention and malloc_trim overhead through precise Rust-side instrumentation (the GPU_TIMING and FIN_TIMING logging infrastructure), attention shifted to the C++ gpu_prove_start function. By adding timing instrumentation around GPU mutex acquisition, barrier waits, and the ntt_msm_h phase, the team identified the true bottleneck: the Host-to-Device transfer of the a/b/c synthesis vectors inside execute_ntts_single. This transfer was running at a mere 1-4 GB/s instead of the PCIe Gen5 x16 line rate of ~50 GB/s.

The Problem Statement

The message opens with a concise problem statement that distills this complex investigation into a single sentence:

execute_ntts_single does cudaMemcpyAsync from unpinned Vec<Fr> (a/b/c) → device. CUDA stages through a tiny pinned bounce buffer at 1-4 GB/s. With pinned source memory, it would go at ~50 GB/s (PCIe Gen5 line rate).

This is a textbook CUDA performance pitfall. When host memory is allocated through standard malloc (or Rust's global allocator, which ultimately calls malloc), the pages are not pinned—they are swappable and not directly accessible by the GPU's DMA engine. CUDA's runtime must stage the transfer through a small internal pinned bounce buffer, typically 1-4 MB in size. For transfers of several gigabytes, this bounce buffering creates a severe bottleneck. The transfer proceeds in small chunks, each requiring a copy into the bounce buffer and then a DMA to the device, resulting in the observed 1-4 GB/s throughput.

The fix is conceptually simple: ensure the source memory is pinned (page-locked) so the GPU can DMA directly from it at full PCIe bandwidth. But the implementation is fraught with engineering challenges, as the message goes on to explore.

Quantifying the Problem

The message provides precise size figures that ground the design in real numbers:

The Two Approaches

The core of the message is the presentation of two architectural approaches, framed as a question to the user. This is a critical design decision that will shape the codebase for the foreseeable future.

Option A: Staging Memcpy

Option A is the simpler, less invasive approach. It involves:

  1. Creating a pool of pre-allocated pinned buffers (allocated once via cudaHostAlloc)
  2. After synthesis completes, copying the a/b/c vectors from their standard heap allocations into these pinned buffers (a memcpy at DDR5 speeds, ~80ms for 8 GiB)
  3. Passing the pinned buffer pointers to the C++ execute_ntts_single function
  4. Returning the buffers to the pool after the GPU transfer completes The key advantage is minimal code change: bellperson's ProvingAssignment struct remains untouched, and the synthesis code continues to use standard Vec<Fr> allocations. The pinned pool logic lives entirely in cuzk-core's gpu_prove_start function, which already orchestrates the transition from synthesis to GPU proving. The disadvantage is the ~80ms memcpy overhead per partition. While small relative to the current 2-8 second transfer time, it is not zero. More subtly, this memcpy consumes host memory bandwidth at a time when other threads may be performing synthesis (which also heavily uses memory bandwidth). On a system with many cores, this contention could reduce the benefit.

Option B: Zero-Copy Direct Synthesis

Option B is the more ambitious approach: synthesize directly into pinned memory, eliminating the staging copy entirely. This requires:

  1. Creating a PinnedVec<T> type that allocates from a pinned memory pool instead of the global heap
  2. Modifying bellperson's ProvingAssignment to use PinnedVec<Fr> instead of Vec<Fr> for the a/b/c fields
  3. Pre-allocating pinned buffers of the correct size before synthesis begins (using the capacity hint system that caches vector sizes after the first synthesis run)
  4. After GPU transfer completes, returning the buffers to the pool instead of freeing them The advantage is true zero-copy: the data is DMA-able from the moment it is written during synthesis. The H2D transfer becomes a true direct memory access, limited only by PCIe bandwidth. The disadvantages are significant: it requires modifying bellperson's core ProvingAssignment struct, which is a dependency maintained by a separate team (Supranational). The PinnedVec type must carefully manage ownership and cleanup to avoid undefined behavior when Rust's Vec::drop tries to free memory through the global allocator that was actually allocated via cudaHostAlloc. The synthesis code, which calls push() on these vectors, must continue to work correctly with the new backing store.

The Reasoning Behind the Choice

The message does not explicitly recommend one option—it presents both and asks the user to decide. However, the assistant's earlier reasoning in [msg 3062] reveals a clear trajectory. The assistant systematically explored and rejected several intermediate approaches before arriving at these two viable candidates.

The first approach considered was cudaHostRegister, which pins an existing heap allocation in place. This was rejected because the registration cost can equal the transfer cost itself—effectively doubling the time for the operation. It is not suitable for per-partition use.

The second approach was a custom allocator that redirects Vec<Fr> allocations to pinned memory. This was rejected due to Rust's allocator limitations: the global allocator cannot be easily swapped per-allocation, and a per-type allocator would require significant changes to how Vec interacts with the allocator API.

The third approach was a PinnedVec wrapper that manages its own pinned buffer and supports push() operations. This evolved into Option B.

The fourth approach was the staging memcpy, which became Option A.

The assistant's reasoning reveals a deep understanding of the tradeoffs involved. The ~80ms memcpy overhead of Option A is weighed against the complexity and invasiveness of Option B. The assistant notes that the memcpy can be performed before acquiring the GPU mutex, allowing it to overlap with other partitions' GPU compute—effectively hiding the cost in some concurrency scenarios.

Assumptions and Context

Several assumptions underpin this message:

The system has abundant RAM. The user explicitly confirmed in [msg 3063] that "most system memory is used just for this workload, no other workloads using system RAM." This allows aggressive pinning without fear of starving other processes. On a system with 755 GiB of RAM, pinning an additional 15-25 GiB for the pool is feasible.

The capacity hint system works reliably. The assistant assumes that after the first synthesis run for a given circuit type, the exact vector sizes are known and cached. This is critical for Option B, which requires pre-allocating pinned buffers of the correct size before synthesis begins. If the hint system is wrong, the pinned buffer could be too small, requiring fallback logic.

The C++ code can consume pinned pointers without modification. The assistant assumes that execute_ntts_single can accept arbitrary pointers and will benefit from pinned memory transparently. This is correct for CUDA: cudaMemcpyAsync automatically uses DMA for pinned source memory. No changes to the CUDA kernel code are required.

The pool can be integrated with the existing MemoryBudget system. The assistant's earlier reasoning emphasizes that pinned memory should be tracked as part of the overall memory budget. The PinnedPool must acquire reservations from the budget before allocating, and must be shrinkable when higher-priority consumers (SRS, PCE) need memory.

What the Message Creates

This message creates several things:

  1. A formal design document for the pinned memory pool, with problem statement, size analysis, and two architectural approaches.
  2. A decision point that forces the user to choose between simplicity (Option A) and performance (Option B). The user's answer—choosing Option B—sets the direction for the implementation that follows in [chunk 22.1].
  3. Shared understanding between the assistant and the user about the nature of the problem and the solution space. After this message, both parties are aligned on what needs to be built and why.
  4. A boundary between investigation and implementation. This message marks the end of the diagnostic phase and the beginning of the construction phase. The root cause is identified, quantified, and a solution is designed. The next messages will implement that solution.

The User's Response

The user's answer (embedded in the message as a continuation) chooses Option B: "Zero-copy, but requires PinnedVec type in bellperson and changes to ProvingAssignment struct." This is the more ambitious choice, reflecting a willingness to accept short-term complexity for long-term performance. The user recognizes that the ~80ms memcpy overhead of Option A, while small, would compound across thousands of partitions in production, and that the architectural purity of zero-copy is worth the implementation cost.

This decision cascades into the implementation that follows in [chunk 22.1], where the assistant implements the PinnedPool struct, the PinnedBacking mechanism in ProvingAssignment, and the release_abc method that safely returns pinned buffers to the pool without triggering undefined behavior.

Conclusion

Message [msg 3069] is a classic example of a well-structured engineering decision document. It presents a clearly defined problem, quantifies the impact, enumerates the options with their tradeoffs, and asks for a decision. The message is the culmination of an extensive investigative process that involved instrumenting C++ code, analyzing nvtop bandwidth data, tracing call chains through multiple abstraction layers, and reasoning about CUDA memory management semantics.

The choice of Option B—direct synthesis into pinned memory—represents a bet on architectural cleanliness over implementation expedience. It requires modifying a core dependency (bellperson), designing a safe ownership model for pinned memory in Rust, and integrating with an existing memory budget system. But if successful, it eliminates the H2D bottleneck entirely, collapsing the transfer from seconds to milliseconds and dramatically improving GPU utilization.

This message is also notable for what it reveals about the assistant's thinking process. The assistant does not simply present two options; it has already explored and rejected multiple intermediate approaches (cudaHostRegister, custom allocator, PinnedVec wrapper) and arrived at the two that represent the Pareto frontier of the design space. The reasoning in [msg 3062] shows a systematic exploration of alternatives, each rejected for specific technical reasons. This is the hallmark of mature engineering: not just finding a solution, but understanding why other solutions are worse.

The pinned memory pool that emerges from this decision becomes a foundational component of the proving engine's memory architecture, alongside the SRS and PCE memory regions. It is a testament to the value of precise instrumentation, systematic debugging, and thoughtful design before diving into implementation.