The Decision That Wired Zero-Copy Pinned Memory into the cuzk Proving Pipeline

In message [msg 3115] of a lengthy debugging and optimization session, the AI assistant made a pivotal architectural decision that determined how a zero-copy pinned memory pool would be integrated into the cuzk proving engine's synthesis path. This single message — a brief block of reasoning followed by a file edit — represents the critical juncture where a working but disconnected component (the PinnedPool) was finally wired into the live code paths that produce ProvingAssignment instances for GPU proving.

The Problem: A Bottleneck Identified and a Solution Built But Not Connected

The broader context is essential. The cuzk proving daemon had been suffering from persistent GPU underutilization — the GPU was active only about 50% of the time per partition. Through detailed C++ timing instrumentation (CUZK_TIMING, CUZK_NTT_H), the root cause was traced to the H2D (host-to-device) transfer phase inside the ntt_kernels CUDA kernel. The a/b/c vectors (the constraint system's linear-combination evaluations) were being copied from unpinned Rust heap memory (Vec<Scalar>) via cudaMemcpyAsync. When the source memory is regular heap pages, CUDA must stage the transfer through a tiny internal pinned bounce buffer, achieving only 1–4 GB/s of throughput instead of the PCIe Gen5 line rate of approximately 50 GB/s.

The solution was a zero-copy pinned memory pool (PinnedPool), designed and implemented in earlier messages. The pool pre-allocates pinned (page-locked) host memory buffers that can be transferred to the GPU at full PCIe bandwidth, eliminating the bounce-buffer bottleneck. The core components — PinnedPool, PinnedBacking (a RAII wrapper that returns buffers to the pool on drop), release_abc() (a method on ProvingAssignment that extracts the a/b/c vectors for return), and new_with_pinned() (a constructor for ProvingAssignment that uses pinned backing) — were all implemented and compiling cleanly.

But there was a gap: the pool was not actually used anywhere. The synthesis functions in pipeline.rs still called synthesize_circuits_batch_with_hint, which internally created ProvingAssignment instances using new_with_capacity or new() — standard heap allocations. The PinnedPool existed as an island of functionality with no connections to the live code paths.

The Reasoning Process: Three Approaches Evaluated

Message [msg 3115] captures the assistant's reasoning as it confronted this integration problem. The message opens with a clear statement of the core issue:

The key insight is: synthesize_circuits_batch_with_hint creates each prover with ProvingAssignment::new_with_capacity or ProvingAssignment::new(). To use pinned memory, I need to add a third option: ProvingAssignment::new_with_pinned.

This observation frames the entire design space. The existing synthesis function is a black box that both creates provers and runs synthesis on them. To use pinned memory, the assistant must somehow inject pinned-backed provers into this function, or bypass it entirely.

The assistant then walks through three possible approaches:

Approach 1: A ProverFactory callback parameter. The idea is to add an optional callback to synthesize_circuits_batch_with_hint that the caller provides to create provers. The assistant quickly identifies a problem: "since this is per-circuit in a par_iter, the factory needs to be Fn + Sync." The rayon par_iter parallelism used during synthesis means the factory must be safe to call from multiple threads simultaneously, which constrains the type signature and makes the approach awkward.

Approach 2: A new synthesize_circuits_batch_with_pinned function. This variant would accept pinned buffer information directly. The assistant considers this but doesn't elaborate on why it's rejected — the reasoning is implicit in the final choice.

Approach 3: A new synthesize_circuits_batch_with_provers function. This is the approach the assistant ultimately selects. The key insight is to separate prover creation from synthesis execution. The caller (pipeline.rs) creates provers with pinned backing, then passes them to a function that only runs synthesis. The assistant explains: "The caller (pipeline.rs) creates them with pinned backing, and this function just runs synthesis on them."

The message then concludes with the implementation action: [edit] /tmp/czk/extern/bellperson/src/groth16/prover/supraseal.rs — and reports that the edit was applied successfully.

Assumptions and Trade-offs

The assistant makes several assumptions in this decision:

That separating prover creation from synthesis is safe and clean. The existing synthesize_circuits_batch_with_hint function does two things: (1) create ProvingAssignment instances (with appropriate capacity), and (2) run the bellpepper constraint synthesis on each circuit, populating those assignments. The new synthesize_circuits_batch_with_provers function would only do step (2), accepting pre-built provers. This separation is architecturally clean, but it assumes that the synthesis logic doesn't depend on any internal state set up during prover creation beyond what's already in the ProvingAssignment struct.

That the pinned pool checkout/return pattern integrates cleanly with rayon parallelism. The pool's checkout() method returns a PinnedBacking RAII guard. Multiple rayon threads will be checking out buffers concurrently. This requires the pool to be thread-safe (which it is, using an internal Mutex), but it also means the pool's capacity must be sufficient for the worst-case concurrent checkout count. The assistant doesn't discuss this sizing concern in this message, but it's an implicit assumption.

That the par_iter in the synthesis function doesn't need modification. The new function would still use par_iter internally to parallelize circuit synthesis. The pre-built provers are passed in as a Vec and distributed across threads. This assumes that the pinned buffers, once assigned to a prover, can be safely moved between threads (which they can, since PinnedBacking contains an Arc to the pool and a raw pointer/length pair).

That the fallback path (standard heap allocation) is still needed. The assistant doesn't explicitly state this in message [msg 3115], but the overall design includes a fallback: if the pool is exhausted (all buffers checked out), the synthesis falls back to new_with_capacity with standard heap memory. This is a robustness assumption — the pool is an optimization, not a correctness requirement.

Input Knowledge Required

To understand this message, one needs knowledge of several components:

  1. The ProvingAssignment struct — a bellperson type that holds the a/b/c evaluations, density trackers, and witness assignments for a single circuit. It has new(), new_with_capacity(), and the newly added new_with_pinned() constructors.
  2. The synthesize_circuits_batch_with_hint function — the existing synthesis entry point in bellperson's supraseal module. It takes circuits and a capacity hint, creates provers, runs synthesis via par_iter, and returns the provers along with input/aux assignment vectors.
  3. The PinnedPool and PinnedBacking types — the pool implementation in cuzk-core/src/pinned_pool.rs. PinnedPool manages a fixed set of pinned memory buffers. PinnedBacking is a RAII guard that returns the buffer to the pool on destruction.
  4. The rayon par_iter parallelism model — synthesis is parallelized across circuits using rayon's work-stealing thread pool. Any callback or factory used inside par_iter must be Fn + Sync.
  5. The pipeline architecture — synthesize_auto is the main entry point in pipeline.rs, which dispatches to either the PCE fast path or the standard synthesize_with_hint path. The per-partition functions (synthesize_partition, synthesize_snap_deals_partition) call synthesize_auto and are themselves called from the engine's synthesis worker threads.

Output Knowledge Created

This message produces a concrete design decision that cascades through the subsequent implementation:

  1. A new function signature: synthesize_circuits_batch_with_provers(circuits, provers) -> Result<(Instant, Vec<ProvingAssignment>, Vec<Arc<Vec<Scalar>>>, Vec<Arc<Vec<Scalar>>>), SynthesisError>. This function takes pre-built provers and runs synthesis on them, returning the same tuple as the existing function.
  2. A clear separation of concerns: Prover creation (with pinned backing) happens in pipeline.rs, synthesis execution happens in bellperson. This is a cleaner architecture than threading a factory callback through the synthesis function.
  3. A wiring plan: The pool will flow from Engine::new() through PartitionWorkItem into the per-partition synthesis functions, which will check out pinned buffers, create provers with new_with_pinned, and call the new bellperson function.

The Significance of the Decision

This message matters because it resolves a tension between two design goals: keeping the bellperson library's API general-purpose (not pinned-memory-specific) while enabling the zero-copy optimization in the cuzk engine. By choosing to add a function that accepts pre-built provers rather than modifying the existing function with a callback or a pinned-memory-specific variant, the assistant preserves backward compatibility and keeps the pinned-memory concern isolated in the caller.

The decision also reflects a practical engineering sensibility. The assistant explicitly rejects the ProverFactory callback approach because of the Fn + Sync constraint — a type-system complication that would ripple through the synthesis code. Instead, it chooses the "simplest and most self-contained approach": pass pre-built provers. This is a classic trade-off between generality (a factory callback could be used for any prover-creation strategy) and simplicity (pre-built provers require no type-system gymnastics).

In the messages that follow ([msg 3116] through [msg 3155]), the assistant executes this plan: it adds the new function to bellperson, exports it from the groth16 module, adds a pinned_pool parameter to synthesize_auto, threads it through all nine call sites, adds a pinned_pool field to PartitionWorkItem, creates the pool in Engine::new(), and wires the evictor callback. The entire integration is driven by the decision made in this single message.