The Pivot to Pinned Memory: How a Research Task Unlocked the Path to Zero-Copy GPU Proving

Introduction

In the high-stakes world of zero-knowledge proof generation, every millisecond counts. When the cuzk proving daemon was found to be utilizing its NVIDIA GPU at only ~50% capacity, the team embarked on a systematic investigation that peeled back layer after layer of complexity. Message [msg 3053] marks a pivotal moment in that investigation: the decision to dispatch a comprehensive codebase research task to understand how pinned memory is currently used, laying the groundwork for a zero-copy solution that would eliminate the dominant bottleneck in the proving pipeline.

This message is not a code change, nor is it a flashy breakthrough. It is a research dispatch — a deliberate, methodical information-gathering step that exemplifies the disciplined approach required to solve performance problems in complex GPU-accelerated systems. To understand its significance, we must trace the investigative journey that led to this point and examine the reasoning that made this research task essential.

The Investigation That Preceded It

The story begins with a troubling observation: the cuzk proving daemon's GPU utilization showed a bursty pattern of ~1-2 seconds of intense compute followed by 2-8 seconds of near-idle gaps. The team had already ruled out several initial suspects — tracker lock contention, malloc_trim overhead, and C++ mutex contention — through precise Rust-side instrumentation (GPU_TIMING and FIN_TIMING logging).

The decisive clue came from [msg 3041], where the user observed that nvtop showed PCIe RX bandwidth dropping to 1-4 GB/s during the idle gaps but bursting to 50 GB/s during compute phases. This was the key insight: the 50 GB/s bursts corresponded to MSM (multi-scalar multiplication) operations reading SRS (Structured Reference String) points from CUDA-pinned host memory via direct PCIe DMA at near line rate. The 1-4 GB/s trickle during gaps corresponded to the Host-to-Device (H2D) transfer of the a/b/c synthesis vectors — regular heap-allocated Rust Vec<Fr> objects that CUDA must stage through a small pinned bounce buffer, throttling throughput by 10-50x.

The assistant confirmed this hypothesis by adding granular timing instrumentation to the C++ gpu_prove_start function in [msg 3033]-[msg 3036], separating mutex wait time, barrier wait time, and the ntt_kernels phase. The resulting data was definitive:

| Component | Time Range | Nature | |-----------|-----------|--------| | ntt_kernels (H2D copy) | 287–5617ms | The bottleneck — varies 4x with memory contention | | msm_invoke | ~631ms | Stable GPU compute | | batch_add | ~401ms | Stable GPU compute | | tail_msm | ~197ms | Stable GPU compute | | coset_intt_sync | ~113ms | Stable GPU compute |

The actual GPU compute was a stable ~1.2 seconds per partition. Everything else was the GPU holding the mutex while waiting for data to arrive over PCIe at a fraction of the available bandwidth. The root cause was clear: the a/b/c vectors were allocated as standard heap memory, while the SRS points enjoyed the performance benefits of cudaHostAlloc-pinned memory.

The User's Suggestion and the Assistant's Response

In [msg 3051], the user proposed investigating pinned memory as a solution, noting that "pinned mem alloc is expensive and slow by itself so we probbaly need a pool which we reuse." The assistant responded in [msg 3052] by creating a structured todo list with research items:

1. Investigate current pinned memory usage (SRS, PCE) and cudaHostRegister/cudaHostAlloc costs
2. Design pinned memory pool for a/b/c synthesis output vectors
3. Understand a/b/c vector sizes and lifecycle (alloc in synthesis, consumed in prove_start)
4. Check how...

Message [msg 3053]: The Research Dispatch

Message [msg 3053] is the assistant's execution of the first todo item. It dispatches a task tool call — spawning a subagent session to perform a comprehensive codebase investigation. The task prompt asks the subagent to:

Research the cuzk codebase to understand: 1. Current pinned memory usage: How is SRS pinned memory allocated? Search for cudaHostAlloc, cudaHostRegister, cudaMallocHost in the entire /tmp/czk/extern directory tree. Also check ...

The task returns a detailed report cataloging every pinned memory allocation in the codebase. The key findings include:

• SRS allocation: The SRS (Structured Reference String) points are allocated via cudaHostAlloc(&pinned, total, cudaHostAllocPortable) in the supraseal C++ layer. This is the reason MSM operations achieve 50 GB/s PCIe throughput — the SRS data is directly DMA-able from host memory.
• PCE (Pre-Compiled Circuit Evaluator): Similarly uses pinned memory for its evaluation data.
• a/b/c vectors: The synthesis output vectors that feed into prove_start are standard Vec<Fr> heap allocations — no pinning at all. This is the root cause of the 1-4 GB/s transfer bottleneck.
• No existing pool: There is no reusable pinned memory pool in the codebase. Each pinned allocation is done on-demand and freed individually. The report also maps the lifecycle of the a/b/c vectors: they are allocated during synthesis (in the Rust bellperson constraint system), passed to prove_start, where C++ code extracts their pointers and copies them to the GPU via cudaMemcpy. The copy is the bottleneck.## Why This Message Was Written: The Reasoning and Context The decision to dispatch a research task rather than immediately implementing a solution reflects a critical engineering discipline: understand before you fix. The team had already identified the bottleneck — unpinned a/b/c vectors causing slow H2D transfer — but the solution space had multiple dimensions that needed exploration before committing to an implementation. The key questions that motivated this research were: 1. How does the existing pinned memory infrastructure work? The SRS already used pinned memory successfully. Understanding its allocation pattern would inform the design of a similar mechanism for synthesis vectors. 2. What are the costs? The user had correctly noted that cudaHostAlloc is expensive — it allocates from a limited pool of pinned memory and involves a system call. A pool-based approach was anticipated, but the exact sizes, lifetimes, and allocation patterns needed to be understood first. 3. What are the integration points? The a/b/c vectors flow from Rust synthesis code through the bellperson constraint system, across the Rust/C++ boundary, and into CUDA kernels. Any change to their allocation strategy would need to touch multiple layers of the stack. 4. What constraints exist? The memory budget system, the evictor, and the concurrent synthesis workers all impose constraints on how much pinned memory can be allocated and when it can be freed. The assistant's reasoning, visible in the thinking blocks of preceding messages, shows a careful weighing of options. In [msg 3042], the assistant considered two approaches: pre-allocating pinned buffers for synthesis outputs versus using cudaHostRegister on existing heap allocations. The former was preferred because cudaHostRegister on 2+ GiB allocations is expensive and risky, and because under heavy memory pressure from concurrent synthesis threads, even a staged memcpy would be severely bottlenecked by contested host memory bandwidth. The research task was designed to gather the data needed to make an informed architectural decision. Without this information, any implementation risked being either incomplete (missing integration points) or inefficient (using the wrong allocation strategy).

Input Knowledge Required

To understand this message, one needs:

1. CUDA memory model knowledge: Understanding the difference between pageable (heap) memory and pinned (page-locked) memory, and why cudaMemcpy is faster from pinned memory (direct DMA vs. staged through a bounce buffer). The PCIe Gen5 x16 line rate of ~50 GB/s provides the theoretical ceiling.
2. The cuzk proving pipeline architecture: Knowledge that synthesis produces a/b/c vectors (the assignment), which are then consumed by prove_start to run NTTs and MSMs on the GPU. The SRS points are already pinned, explaining the 50 GB/s bursts during MSM phases.
3. The investigation history: The message builds directly on the timing instrumentation added in the preceding messages ([msg 3033]-[msg 3036]) and the nvtop observation ([msg 3041]). Without that context, the research task appears disconnected from the problem.
4. The memory budget system: The existing MemoryBudget infrastructure that manages GPU and host memory limits, with which any pinned pool must integrate.
5. The bellperson constraint system: Understanding how ProvingAssignment stores the a/b/c vectors and how they are passed to the C++ proving layer.

Output Knowledge Created

The research task produced a comprehensive report that:

1. Cataloged every pinned memory allocation in the codebase, creating a complete map of the existing infrastructure.
2. Identified the exact allocation pattern for SRS (via cudaHostAlloc with cudaHostAllocPortable flag), providing a template for the synthesis vector solution.
3. Documented the a/b/c vector lifecycle, from synthesis allocation through prove_start consumption, identifying every touch point that would need modification.
4. Revealed the absence of a reusable pool, confirming the need for a new PinnedPool component.
5. Quantified the size ranges for the a/b/c vectors, enabling precise pool sizing decisions. This knowledge directly enabled the implementation that followed in chunk 1 of segment 22 ([msg 3070]-[msg 3095]), where the assistant implemented the PinnedPool struct, extended ProvingAssignment with PinnedBacking, and modified prove_start to release vectors back to the pool.

Assumptions and Potential Pitfalls

The research task operated under several assumptions:

1. That pinning the a/b/c vectors would eliminate the bottleneck: This was well-supported by the evidence — the SRS points already achieved 50 GB/s throughput with pinned memory, and the a/b/c vectors were the only unpinned data in the critical path. However, it assumed that the bottleneck was purely on the H2D transfer side and not in the kernel launch overhead or synchronization patterns.
2. That a pool-based approach is feasible: The assumption that pinned memory can be recycled efficiently depends on the allocation sizes being predictable and the pool not fragmenting. If synthesis produces wildly varying partition sizes, the pool might need to over-allocate or suffer from fragmentation.
3. That the Rust/C++ boundary can be cleanly modified: The a/b/c vectors are currently Rust Vecs. Replacing their backing memory with CUDA-pinned buffers requires careful lifetime management to avoid undefined behavior when Rust tries to free memory it doesn't own.
4. That the memory budget can accommodate pinned allocations: Pinned memory is a limited resource on CUDA-capable systems. The pool must integrate with the existing MemoryBudget to avoid starving other components (SRS, PCE) of pinned memory. One potential mistake in the reasoning chain is the assumption that the H2D transfer is the only significant bottleneck. The timing data showed ntt_kernels varying from 287ms to 5617ms, but even at 287ms (the fastest observed), the H2D transfer still represents ~20% of the total partition time. Eliminating it entirely would yield significant gains, but the barrier wait times (0-1105ms) and mutex wait times (1149-6477ms) represent additional optimization opportunities that the pinned memory solution does not directly address.

The Thinking Process Visible in the Message

The message itself is a task dispatch, so the thinking is implicit in its design. The subagent prompt is structured to answer specific questions that the assistant identified as critical gaps in its understanding:

• "Search for cudaHostAlloc, cudaHostRegister, cudaMallocHost" — This is a reconnaissance query, mapping the existing infrastructure before designing new components.
• "Also check..." — The ellipsis hints at the breadth of the investigation, covering not just the allocation calls but also the surrounding context: buffer sizes, lifetimes, and integration with the memory budget. The task_id (ses_3177d355affeRB8D66cfBh6ACY) and the note about resuming indicate that the assistant anticipated this task might need to be continued or referenced later, reflecting a systematic approach to information management. The report structure in the result — a table cataloging every call site with file, line, and purpose — shows that the subagent executed the research thoroughly, producing actionable intelligence rather than a vague summary.

Conclusion

Message [msg 3053] is a masterclass in disciplined performance debugging. Faced with a clear bottleneck and a promising solution direction, the assistant chose not to rush into implementation but to first gather comprehensive data about the existing infrastructure. This research task transformed an informed hypothesis — "the a/b/c vectors should be pinned" — into a concrete architectural plan backed by a complete map of the codebase's memory management patterns.

The knowledge produced by this single message directly enabled the implementation of the PinnedPool, PinnedBacking, and the modifications to bellperson's ProvingAssignment that followed. It exemplifies the principle that in complex systems, the most valuable output is often not code but understanding — and that the fastest path to a correct implementation is sometimes the slow, methodical path of research first.