The Zero-Copy Pivot: A Single Grep That Defined a CUDA Memory Architecture

The Message

[assistant] [grep] cudaHostAlloc|cudaFreeHost|cudaMallocHost|cuda_host
No files found

This is the complete content of message index 3071 in the opencode session. On its surface, it is unremarkable: a developer searches for existing uses of CUDA's host-pinned memory allocation functions and finds none. But in the context of the broader investigation—a multi-session deep dive into GPU underutilization in a zero-knowledge proof proving pipeline—this single grep represents a critical turning point. It is the moment when a theoretical solution collides with the reality of the codebase, and the architecture for one of the most impactful performance optimizations in the session is born.

The Problem That Led Here

To understand why this message was written, one must understand the crisis that preceded it. The team had been investigating a persistent and puzzling performance problem: the GPU in their proving pipeline was running at only ~50% utilization. Through meticulous instrumentation—adding precise timing probes to the GPU worker loop and finalizer—they had ruled out the initial suspects: tracker lock contention, malloc_trim overhead, and C++ mutex contention. The real culprit, identified in the preceding investigative chunk, was the Host-to-Device (H2D) transfer of the a/b/c synthesis vectors inside the execute_ntts_single function.

The numbers told a stark story. The a/b/c vectors—each holding field elements for the proving assignment—were massive: approximately 2.59 GiB per vector for SnapDeals proofs (totaling 7.8 GiB per partition) and 4.17 GiB per vector for PoRep proofs (totaling 12.5 GiB). These vectors were being transferred from host memory to GPU memory at a paltry 1–4 GB/s, instead of the PCIe Gen5 x16 theoretical line rate of ~50 GB/s. The bottleneck was not the bus itself, but the source of the transfer: the vectors were allocated as standard heap memory via malloc, which meant CUDA could not use direct memory access (DMA) for the transfer. Instead, the CUDA driver was forced to stage the transfer through a small pinned bounce buffer, serializing the operation and crushing throughput.

The root cause was clear: the a/b/c vectors needed to be allocated from pinned (page-locked) host memory using cudaHostAlloc, which would allow the GPU to DMA directly from the source buffer at full PCIe bandwidth. This was the zero-copy approach: eliminate the staging copy entirely by ensuring the data lived in DMA-able memory from the moment it was synthesized.

The Decision: Option B

The user and assistant had just concluded a design discussion weighing two approaches. Option A was a simpler staging strategy: allocate pinned buffers in gpu_prove_start, memcpy the a/b/c vectors into them (adding ~80ms overhead per partition), and pass the pinned pointers to the C++ GPU function. Option B was more ambitious but more elegant: modify the synthesis pipeline itself to write directly into pinned memory, eliminating both the reallocation copies during synthesis (when Vec::push triggers internal reallocation and memcpy) and the H2D staging copy.

The user chose Option B, and made a crucial observation: the memcpy overhead of Option A would not be a clean 80ms under real workload conditions. With 20+ concurrent synthesis threads consuming 400+ GiB of system memory, host memory bandwidth was already heavily contested. A staged memcpy would itself become a bottleneck, competing with synthesis threads for the same memory bus. Direct synthesis into pinned memory was not just more elegant—it was necessary for the optimization to survive under production memory pressure.

Why This Grep?

With the decision made, the assistant needed to implement. But before writing a single line of code, a fundamental question had to be answered: what CUDA host allocation infrastructure already exists in this codebase?

The codebase is a layered architecture. At the bottom sits supraseal-c2, a C++ CUDA library (based on the Supranational sppark library) that provides the low-level GPU primitives. Above it sits cuzk-core, the Rust crate that orchestrates the proving pipeline. And above that sits bellperson, the fork of the Bellman library that handles constraint synthesis. The assistant had already read supraseal-c2/src/lib.rs in the previous message ([msg 3070]) and seen that it uses sppark::cuda_error!() to define CUDA error types, but the actual CUDA FFI bindings—the raw extern "C" declarations for functions like cudaMalloc, cudaMemcpy, and crucially cudaHostAlloc—were not visible in that file.

The grep command searched for four patterns: cudaHostAlloc, cudaFreeHost, cudaMallocHost, and cuda_host. The first three are the standard CUDA runtime API functions for allocating and freeing pinned host memory. The fourth, cuda_host, is a broader pattern that would catch any wrapper, type alias, or macro that incorporates the term "cuda_host" in its name—a defensive search to catch any abstraction layer that might already exist under a different naming convention.

The result was definitive: No files found.## Input Knowledge: What You Need to Understand This Message

To grasp the significance of this grep, a reader must understand several layers of context:

  1. The CUDA memory model: CUDA distinguishes between pageable (standard malloc) and page-locked (pinned) host memory. Pageable memory requires a staged transfer through a driver-managed bounce buffer, which limits throughput to the speed of that staging copy. Pinned memory allows the GPU to DMA directly from the source buffer, achieving near line-rate PCIe bandwidth. This is the core technical insight that drives the entire optimization.
  2. The codebase architecture: The proving pipeline has three layers—supraseal-c2 (C++ CUDA primitives), cuzk-core (Rust orchestration), and bellperson (Rust constraint synthesis). CUDA FFI bindings live in supraseal-c2, but the assistant had already read that file and found no host allocation functions there. The grep confirms this absence across the entire workspace.
  3. The problem domain: Zero-knowledge proofs, specifically Groth16 proofs using the Bellman/Bellperson constraint system. The a/b/c vectors represent the evaluations of the three linear constraint systems (A, B, C) over the field elements, and they are the primary data that must be transferred to the GPU for the multi-scalar multiplication (MSM) and number-theoretic transform (NTT) operations that dominate GPU proving time.
  4. The budget-based memory manager: A recently implemented system that tracks and limits memory usage across the proving pipeline. The pinned pool must integrate with this system, which means understanding how MemoryBudget tracks allocations and how try_acquire and release_internal work.
  5. The PCIe Gen5 context: The theoretical bandwidth of ~50 GB/s for PCIe Gen5 x16 is the target. The observed 1–4 GB/s represents a 90%+ efficiency loss, making this the single largest performance bottleneck in the pipeline.

Output Knowledge: What This Message Created

The grep produced a single, unambiguous fact: no CUDA host-pinned allocation infrastructure exists in this codebase. This negative result is profoundly generative. It means:

Assumptions Embedded in the Search

The grep encodes several assumptions about the codebase:

Assumption 1: The function names follow CUDA's standard naming. The patterns cudaHostAlloc, cudaFreeHost, and cudaMallocHost are the exact CUDA runtime API function names. The assistant assumes that if any pinned allocation exists, it will use one of these standard names rather than a custom wrapper name. The inclusion of cuda_host as a broader pattern mitigates this assumption somewhat, but it would still miss a function named something like allocate_pinned_host_memory or pin_malloc.

Assumption 2: The grep is comprehensive enough. The command searches across all files in the workspace. If the pinned allocation code existed in a dependency not included in the workspace (e.g., a system-installed CUDA library header), it would not be found. But the assistant is looking for usage of these functions in the codebase's own source, not in external libraries, so this is appropriate.

Assumption 3: The codebase is self-contained for this purpose. The assistant assumes that any pinned allocation infrastructure would be part of the visible codebase, not hidden behind a build-time dependency or a generated binding. This is a reasonable assumption given that supraseal-c2 is the designated CUDA FFI layer and the assistant has already read its main source file.

The Thinking Process: What the Reasoning Reveals

The assistant's reasoning in [msg 3070] (the message immediately preceding this grep) reveals a rich thinking process. The assistant walks through the design space methodically:

  1. Problem restatement: "The memcpy wouldn't just be 80ms, because under heavy memory pressure (20+ synthesis threads, 400+ GiB used), host memory bandwidth is already contested." This shows the assistant internalizing the user's point and extending it.
  2. Architecture mapping: "The tricky part is handling ProvingAssignment—I need to figure out the cleanest way to integrate pinned memory there." The assistant identifies the core design challenge: ProvingAssignment is generic and lives in bellperson, while the PinnedPool lives in cuzk-core, creating a dependency inversion problem.
  3. Iterative refinement: The reasoning cycles through multiple approaches—custom allocator, PinnedVec wrapper, enum abstraction, function pointer callbacks, ManuallyDrop<Vec>—before settling on the PinnedBacking + release_abc approach. Each iteration identifies a flaw in the previous approach and refines it.
  4. Memory model awareness: "For scalar types like Fr that are Copy, this is straightforward—they're just plain data with no destructors to worry about." The assistant carefully considers the Rust ownership and drop semantics, ensuring the solution avoids undefined behavior.
  5. Budget integration thinking: "The pool should hold permanent reservations for allocated buffers, releasing them only when buffers are freed through shrinkage or eviction." The assistant maps the pinned buffer lifecycle onto the existing MemoryBudget system, showing awareness of the broader architecture. This grep sits at the transition between design and implementation. The assistant has a clear plan in mind, but needs the concrete answer to "what exists?" before writing code. The grep is the reality check that turns the design into an actionable implementation plan.## Mistakes and Incorrect Assumptions The grep itself is a factual operation—it either finds matches or it doesn't—so it cannot be "wrong" in the conventional sense. However, the interpretation of its result carries risk: The silent assumption of sufficiency. The grep confirms that no existing code uses cudaHostAlloc or similar functions, but it does not confirm that adding such functions is straightforward. The assistant assumes that supraseal-c2 can easily be extended with new FFI bindings, but this depends on the build system, the CUDA toolkit availability, and the linkage model. If supraseal-c2 links CUDA statically or uses a custom CUDA wrapper library, adding raw extern "C" declarations might not work without additional build configuration. The assumption of FFI simplicity. The assistant's reasoning in [msg 3070] mentions "the actual CUDA FFI bindings for cudaHostAlloc and cudaFreeHost" as if adding them is a mechanical task. In practice, CUDA FFI bindings require careful attention to error handling (CUDA functions return error codes that must be checked), pointer lifetime management, and thread safety. The cudaHostAlloc function in particular has flags (cudaHostAllocDefault, cudaHostAllocPortable, cudaHostAllocMapped, cudaHostAllocWriteCombined) that affect behavior across multi-GPU systems and NUMA domains—the assistant's reasoning does not yet engage with these details. The assumption of zero-cost abstraction. The assistant's reasoning treats pinned memory as a pure win: "eliminates both the reallocation copies during synthesis AND the slow staged H2D transfer." While true for the H2D transfer, pinned memory has costs: it consumes page-locked memory that cannot be swapped, it can fragment the physical memory address space, and excessive pinning can degrade overall system performance by starving the page cache. The assistant's reasoning does not yet grapple with these trade-offs. The assumption about Fr being Copy. The assistant notes that "for scalar types like Fr that are Copy, this is straightforward—they're just plain data with no destructors to worry about." This is correct for the field element type in the BLS12-381 curve used in this project, but it is a property of the specific type, not a general truth. If the proving assignment were ever to hold types with destructors, the ManuallyDrop approach would leak those resources. The assistant correctly identifies this as a safe assumption for the current use case, but it is an assumption worth documenting.

Conclusion

Message [msg 3071]—a single grep returning "No files found"—is a microcosm of the engineering process. It is the moment of confirmation that precedes creation. Before you can build, you must know what exists. Before you can design the new, you must survey the old. This grep, for all its brevity, answered the most important question the assistant faced at that moment: "Am I building on existing infrastructure, or am I building from scratch?"

The answer was "from scratch," and that answer shaped everything that followed. The PinnedPool struct, the PinnedBacking mechanism, the release_abc method, the custom Drop implementation, the budget integration—all of these were designed and implemented in the messages that followed, and all of them trace their origin to this single negative result.

In the broader narrative of the opencode session, this message represents the pivot point between diagnosis and treatment. The investigation had identified the disease (H2D bottleneck at 1–4 GB/s). The design discussion had chosen the treatment (direct synthesis into pinned memory). But the surgery could not begin until the assistant knew what tools were already in the operating room. The grep was the inventory check, and its empty result was the signal to proceed with building everything from the ground up.

For the reader, this message is a reminder that the most impactful engineering decisions are often not the grand architectural pronouncements, but the quiet, factual answers to simple questions. "Does this exist?" asked of a codebase can change the trajectory of an entire optimization effort. The answer "No" is not a dead end—it is a foundation.