The Evictor Integration: Wiring a Pinned Memory Pool into the CUZK Memory Budget System

The Message

In the course of a multi-session optimization effort for the CUZK proving daemon, the assistant produced the following message (global index 3154):

[assistant] Now I need to integrate the pinned pool with the evictor callback so it can shrink the pool when memory pressure occurs. Let me find the evictor setup: [read] /tmp/czk/extern/cuzk/cuzk-core/src/engine.rs <path>/tmp/czk/extern/cuzk/cuzk-core/src/engine.rs</path> <type>file</type> <content>991: 992: info!( 993: pipeline_enabled = self.pipeline_enabled, 994: "starting cuzk engine" 995: ); 996: 997: // SRS and PCE are loaded on demand — no preload. 998: // Wire evictor callback so budget.acquire() can evict idle SRS/PCE. 999: { 1000: let srs_for_evict = self.srs_manager.clone(); 1001: #[cfg(feature = "cuda-su...

This message, at first glance, appears to be a simple read operation — the assistant is loading a file to inspect the evictor callback setup. But within the broader context of the pinned memory pool implementation, this read represents a critical architectural decision point. The assistant is about to integrate a new memory consumer (the PinnedPool) into the existing memory budget eviction system, and it needs to understand the existing evictor pattern before it can modify it.

Context: The GPU Underutilization Problem

To understand why this message matters, we must step back to the problem it was solving. The CUZK proving daemon was suffering from persistent GPU underutilization — approximately 50% — despite having a powerful RTX 5090 connected via PCIe Gen5 x16 (theoretically capable of ~50 GB/s). Through detailed C++ timing instrumentation (using CUZK_TIMING and CUZK_NTT_H flags), the team had confirmed the root cause: the GPU mutex was held for 1.6–7.0 seconds per partition, but the GPU was only actively computing for approximately 1.2 seconds. The remaining time was spent in ntt_kernels performing H2D (host-to-device) transfers of the a/b/c vectors from unpinned Rust heap memory (Vec<Scalar>) via cudaMemcpyAsync. Because the source memory was unpinned, CUDA was forced to stage through a tiny internal pinned bounce buffer, achieving only 1–4 GB/s instead of the PCIe Gen5 line rate.

The chosen solution was a zero-copy pinned memory pool — the PinnedPool. Instead of allocating a/b/c vectors as regular Rust Vec<Scalar> on the heap (which would later need to be copied to the GPU via slow staged transfers), the synthesis process would write directly into CUDA-pinned host memory allocated from a pool. This would eliminate both the reallocation copies during synthesis and the slow staged H2D transfer, because the GPU could DMA directly from the pinned buffers at full PCIe bandwidth.

What Came Before: The Wiring Effort

By the time message 3154 was written, the assistant had already completed a substantial portion of the wiring. The core PinnedPool implementation in cuzk-core/src/pinned_pool.rs was complete and compiling cleanly. The PinnedBacking struct, release_abc() method, and new_with_pinned() constructor had been added to bellperson/src/groth16/prover/mod.rs. A new function synthesize_circuits_batch_with_prover_factory had been created in the bellperson supraseal module to support creating ProvingAssignment instances with pinned backing.

In the pipeline layer (pipeline.rs), the assistant had modified synthesize_auto and synthesize_with_hint to accept an optional Arc<PinnedPool>. All nine call sites of synthesize_auto had been updated — the two per-partition functions (synthesize_partition and synthesize_snap_deals_partition) now passed through the pinned_pool reference, while the other seven call sites passed None. The PartitionWorkItem struct in engine.rs had gained a pinned_pool: Option<Arc<PinnedPool>> field. The process_batch function had been updated to accept and forward the pool reference. The Engine struct itself had been modified to hold an Arc<PinnedPool>, and Engine::new() had been updated to create the pool at startup.

But one critical integration point remained: the evictor callback.## The Evictor Callback: A Delicate Integration Point

The evictor callback is the mechanism by which the memory budget system (MemoryBudget) can reclaim memory from idle subsystems when a new allocation request cannot be satisfied. In the CUZK daemon, there are two major consumers of memory that participate in this system: the SRS manager (which holds the Structured Reference Strings in GPU pinned memory via cudaHostAlloc) and the PCE cache (which holds Pre-Compiled Constraint Evaluators on the heap). When budget.acquire() is called and the budget is exhausted, it invokes the evictor callback to ask these subsystems to free their idle resources.

The assistant's reasoning, visible in the message text, is precise: "Now I need to integrate the pinned pool with the evictor callback so it can shrink the pool when memory pressure occurs." This is not merely a nice-to-have feature — it is essential for correctness. Without evictor integration, the PinnedPool would be a memory consumer that is invisible to the budget system. If the pool holds onto large amounts of pinned memory (potentially tens of GiB) while SRS or PCE needs to be loaded, the budget system would have no way to reclaim that memory. The pool's shrink() method, which frees idle buffers, would never be called, and the system could run out of memory or fail to load critical resources.

The assistant reads the evictor setup code around line 991 of engine.rs. The comment on line 997–998 is telling: "SRS and PCE are loaded on demand — no preload. Wire evictor callback so budget.acquire() can evict idle SRS/PCE." This reveals the design philosophy: resources are loaded lazily, and the evictor ensures that memory pressure triggers graceful degradation. The assistant needs to extend this pattern to include the PinnedPool.

The Thinking Process Visible in the Message

The message reveals several layers of the assistant's reasoning:

Awareness of the integration gap: The assistant explicitly states "Now I need to integrate the pinned pool with the evictor callback." This shows it has a mental checklist of remaining work items and recognizes that the evictor integration is the next logical step after having wired the pool through the synthesis and dispatch paths.
Understanding of the evictor's purpose: By saying "so it can shrink the pool when memory pressure occurs," the assistant demonstrates a clear understanding of what the evictor callback does and why the pool needs to participate in it. The pool is not just a performance optimization — it is a memory consumer that must be managed within the overall budget.
Methodical approach: The assistant reads the evictor setup code before making any changes. This is a deliberate choice — it needs to understand the existing pattern (how the SRS manager registers its eviction callback) before it can replicate that pattern for the PinnedPool. The read is not idle curiosity; it is reconnaissance for the edit that follows.
Recognition of the pool's shrink() method: The assistant knows that PinnedPool has a shrink() method that frees idle buffers. This method was presumably designed specifically for this purpose — to be called during eviction to release memory back to the system.

Assumptions Made

The assistant makes several assumptions in this message:

That the evictor callback pattern is extensible: It assumes that the existing evictor setup (which currently handles SRS and PCE) can be straightforwardly extended to include the PinnedPool. This is a reasonable assumption given the architecture, but it depends on the evictor being designed as a list or chain of callbacks rather than a single-purpose hook.
That PinnedPool::shrink() is safe to call from the evictor context: The evictor callback is invoked from within budget.acquire(), which may be called from various contexts (including async). The assistant assumes that shrink() is non-blocking and safe to call without holding locks that could conflict with the synthesis path.
That the pool is the only remaining integration point: The assistant's phrasing "Now I need to integrate" implies that everything else is already wired. This assumption proved correct — the subsequent messages show that after this evictor integration, the only remaining steps were updating PartitionWorkItem construction sites and then running cargo check.

Input Knowledge Required

To understand this message fully, one needs knowledge of:

The CUZK memory budget system: How MemoryBudget::acquire() works, what the evictor callback is, and how SRS and PCE register their eviction handlers.
The PinnedPool API: Specifically that it has a shrink() method that releases idle pinned buffers back to the OS.
The Engine lifecycle: That the evictor is wired during engine startup (in Engine::run() or similar), before any proof processing begins.
The Rust async model: Why the evictor callback needs to be careful about blocking (it's called from async acquire()), and why try_lock() patterns are used for the SRS manager.
The Arc<PinnedPool> ownership model: That the pool is shared via Arc and can be cloned cheaply for use in closures and callbacks.

Output Knowledge Created

This message, combined with the subsequent edit (message 3156), creates:

A new eviction path: When memory pressure occurs, the evictor callback now calls pool.shrink() to release idle pinned buffers, making that memory available for SRS loading, PCE extraction, or other budget-managed allocations.
A complete integration: With the evictor wired, the PinnedPool is now fully integrated into the engine lifecycle — created at startup, threaded through synthesis dispatch, used for pinned allocation in ProvingAssignment, and cleaned up under memory pressure.
A foundation for deployment: The evictor integration was the last piece of the wiring puzzle. After this, the assistant would proceed to update the PartitionWorkItem construction sites (messages 3157–3163), run cargo check to verify compilation, and build a Docker image for deployment.

Significance in the Larger Narrative

This message, while seemingly small, represents the moment when the pinned memory pool solution transitioned from a collection of independent components to an integrated system. The evictor callback is the glue that connects the pool to the memory budget, ensuring that the pool does not become a memory leak or a source of resource starvation. Without this integration, the pool could hold onto pinned memory indefinitely, preventing SRS or PCE from loading when needed.

The assistant's methodical approach — reading the existing code before editing, understanding the pattern, and extending it consistently — is characteristic of the entire pinned pool implementation. Each component was designed, implemented, compiled, and then wired into the larger system in a deliberate sequence: first the pool data structures, then the ProvingAssignment modifications, then the synthesis function changes, then the engine dispatch path, and finally the evictor integration.

The message also illustrates a key principle of systems programming: performance optimizations must be designed within the resource management framework of the system, not as isolated improvements. The pinned memory pool could have been implemented as a simple cache that grows unbounded, but the integration with the evictor ensures it participates in the cooperative memory management that keeps the entire proving daemon stable under varying workloads.