The Capacity Hint: A Pivotal Read in the Hunt for GPU Throughput

The Message

[assistant] Now let me look at how the capacity hint system works in cuzk to understand the exact sizes: [read] /tmp/czk/extern/cuzk/cuzk-core/src/pipeline.rs <path>/tmp/czk/extern/cuzk/cuzk-core/src/pipeline.rs</path> <type>file</type> <content>168: // ─── Capacity Hint Cache ──────────────────────────────────────────────────── 169: // 170: // After the first synthesis for a circuit type, we cache the actual Vec sizes 171: // (num_constraints, num_aux, num_inputs) so subsequent syntheses can pre-allocate 172: // to full capacity. This elimina...

At first glance, this appears to be a routine code-reading message—an assistant issuing a read tool call to examine a source file. But in the context of a multi-day debugging odyssey into GPU underutilization, this message represents a critical transition point: the moment when abstract architectural design meets concrete implementation constraints. The assistant is not merely browsing code; it is searching for the exact numbers that will determine whether a zero-copy pinned memory pool is feasible, and if so, how it must be built.

Context: The GPU Utilization Mystery

To understand why this message matters, we must step back. The team had been investigating why GPU utilization hovered around 50% in the cuzk proving daemon—a zero-knowledge proof system for Filecoin. Through painstaking instrumentation of both Rust and C++ code paths, they had identified the bottleneck: the Host-to-Device (H2D) transfer of the a, b, and c synthesis vectors inside the execute_ntts_single CUDA function. These vectors, each roughly 512 MiB for a SnapDeals partition (totaling ~1.5 GiB per partition), were being copied from standard heap-allocated Rust Vec<Fr> objects to GPU memory at a paltry 1–4 GB/s, instead of the ~50 GB/s that the PCIe Gen5 x16 link could theoretically deliver.

The root cause was memory allocation strategy. The SRS (Structured Reference String) points used in MSM (Multi-Scalar Multiplication) operations were allocated via cudaHostAlloc, which creates pinned (page-locked) host memory directly accessible by the GPU via Direct Memory Access (DMA). The a/b/c synthesis vectors, by contrast, were plain Rust Vec allocations on the heap. When CUDA copies data from unpinned memory, the driver must first stage the data through a small internal pinned bounce buffer—a process that throttles throughput by 10–50x, especially under memory bandwidth contention from 20+ concurrent synthesis threads.

The user's observation from nvtop confirmed this: during compute bursts, RX bandwidth hit 50 GB/s (the MSM reading pinned SRS points); during gaps, it dropped to 1–4 GB/s (the NTT H2D copies). The user then directed the assistant to investigate pinning the vectors, suggesting a dynamic pinned memory pool integrated with the existing MemoryBudget system.

Why This Message Was Written

The assistant had just completed a comprehensive research task ([msg 3053]) that catalogued every cudaHostAlloc, cudaHostRegister, and cudaMallocHost call site in the codebase, and traced the lifecycle of the a/b/c vectors from synthesis through prove_start. With that research complete, the assistant updated its todo list to mark "Design pinned memory pool for a/b/c synthesis output vectors" as in-progress ([msg 3054]). But design requires numbers. You cannot design a memory pool without knowing the allocation sizes.

This message is the bridge between research and design. The assistant needs to understand the capacity hint system—a caching mechanism that records the actual Vec sizes (num_constraints, num_aux, num_inputs) after the first synthesis for each circuit type, so subsequent syntheses can pre-allocate to full capacity. This system is critical because:

1. Buffer sizing: The pinned pool must allocate buffers large enough to hold the a/b/c vectors. If the assistant can determine exact sizes from the hint cache, it can pre-allocate pinned buffers of precisely the right capacity, avoiding both waste and reallocation.
2. Lifecycle integration: The capacity hint system already solves the problem of knowing vector sizes before synthesis begins. If the assistant can redirect ProvingAssignment to use pinned buffers at construction time (via new_with_capacity), the vectors would be DMA-ready from the moment they are populated—no copy needed.
3. Budget accounting: The pinned pool must integrate with MemoryBudget. Knowing exact sizes lets the assistant design a pool that can try_acquire the right amount of budget for each buffer checkout.

Input Knowledge Required

To understand this message, one must already grasp several layers of the system:

• The GPU proving pipeline: Synthesis produces ProvingAssignment objects containing a: Vec<Fr>, b: Vec<Fr>, and c: Vec<Fr>. These are consumed by gpu_prove_start, which passes pointers to the C++ groth16_cuda.cu code that copies them to the GPU for NTT (Number Theoretic Transform) computation.
• The capacity hint cache: A mechanism in cuzk-core/src/pipeline.rs that records the actual allocation sizes after the first synthesis for a given circuit type (PoRep, SnapDeals, WinningPoSt, WindowPoSt). Subsequent syntheses use these hints to pre-allocate vectors to full capacity via ProvingAssignment::new_with_capacity, eliminating ~27 reallocation cycles and ~32 GiB of redundant memory copies per synthesis.
• The MemoryBudget system: A unified byte-level budget manager that tracks SRS (pinned), PCE (heap), and synthesis working set (heap) under a single budget auto-detected from system RAM.
• CUDA memory models: The difference between cudaHostAlloc (pinned, directly DMA-able) and regular malloc/Vec allocations (require staged copy through a bounce buffer).

The Thinking Process Visible in This Message

The assistant's reasoning, visible across the surrounding messages, reveals a careful weighing of design alternatives. In the message immediately following ([msg 3062]), the assistant's extended reasoning shows it considering multiple approaches:

Option A: A custom allocator that redirects Vec<Fr> allocations to pinned memory. Rejected because Rust's allocator interface (GlobalAlloc) operates per-type, not per-allocation-site, making it impossible to selectively pin only the a/b/c vectors without pinning all Fr allocations.

Option B: Register existing heap vectors as pinned via cudaHostRegister after synthesis. Rejected because cudaHostRegister is expensive—it can cost as much as the transfer itself, effectively doubling the overhead.

Option C: Copy synthesized vectors into pre-allocated pinned buffers before prove_start. The extra memcpy is cheap (~80ms for 8 GiB at DDR5 speeds) compared to the current 2–9 second H2D bottleneck, but it adds latency and holds the GPU mutex.

Option D (chosen): Synthesize directly into pinned buffers from the pool. This requires knowing buffer sizes upfront, which the capacity hint system provides. The ProvingAssignment would be constructed with a PinnedVec type backed by pool-allocated cudaHostAlloc memory, making the vectors DMA-ready from the moment they are populated.

The capacity hint read is the linchpin of Option D. Without knowing the exact sizes, the assistant cannot pre-allocate pinned buffers of the right capacity, and the entire approach collapses.

Assumptions and Potential Pitfalls

The assistant makes several assumptions in pursuing this read:

1. The capacity hint cache is populated before the pinned pool is needed. This is true for steady-state operation (after the first synthesis), but the very first synthesis for a new circuit type will not have hints. The assistant must handle this cold-start case, perhaps by falling back to a maximum-size allocation or using a heuristic.
2. The hint sizes are stable across runs. The assistant assumes that num_constraints, num_aux, and num_inputs are deterministic for a given circuit topology. The code comments confirm this ("always ~130M for 32 GiB PoRep"), but any variability would cause buffer overruns or waste.
3. Pinned memory can be safely used as Vec backing storage. This is the trickiest assumption. When a Vec is created via Vec::from_raw_parts pointing to pinned memory, Rust's deallocator will attempt to free it through the global allocator, causing undefined behavior. The assistant recognizes this challenge and plans to use ManuallyDrop and a custom Drop implementation to safely return buffers to the pool.
4. The system has enough free memory to pin aggressively. The user clarified ([msg 3063]) that no other workloads compete for system RAM, so pinning 15–25 GiB for staging buffers is safe. Without this clarification, the assistant might have designed a more conservative pool.

Output Knowledge Created

This message produces no code changes, no configuration updates, and no running binary. Its output is purely informational: the assistant now knows the structure and location of the capacity hint cache. But this knowledge is immediately actionable. In the subsequent messages ([msg 3060] onward), the assistant proceeds to:

• Read groth16_ntt_h.cu to understand exactly how execute_ntts_single handles the vector data
• Research cudaHostRegister vs cudaHostAlloc performance characteristics
• Design the PinnedPool struct with exact-size allocation, free list, and budget integration
• Implement PinnedBacking in bellperson's ProvingAssignment with safe drop semantics
• Modify prove_start to call release_abc after extracting pointers The capacity hint read is the foundation upon which all of this implementation is built. Without it, the pinned pool would be guessing at sizes, and the elegant zero-copy design would be impossible.

The Deeper Significance

This message exemplifies a pattern that recurs throughout engineering: the moment when a team stops theorizing about a solution and starts grounding it in concrete numbers. The GPU bottleneck had been diagnosed, the root cause identified, and the high-level approach agreed upon. But between "we should pin the vectors" and "here is the implementation" lies a gulf of practical questions. How big are the vectors? How are they allocated? What systems already track their sizes? The assistant's read of the capacity hint cache answers these questions, transforming an abstract design goal into a concrete implementation plan.

In the broader narrative of this debugging session, this message is the turning point where investigation ends and construction begins. The preceding 80+ messages were about measurement, instrumentation, and diagnosis. The following messages are about implementation, compilation, and deployment. This single read command—this one message—is the hinge between those two phases.