The Memory Manager Manifesto: A Strategic Checkpoint in the cuzk GPU Proving Engine

Introduction

In the high-stakes world of zero-knowledge proof generation for Filecoin, memory is not merely a resource—it is the fundamental constraint that shapes the entire architecture. The cuzk (CUDA ZK proving daemon) engine operates at the intersection of GPU compute, cryptographic protocol complexity, and the unforgiving physics of system RAM. Message <msg id=2151> captures a pivotal moment in the development of this system: a comprehensive status update that functions simultaneously as a design manifesto, a progress report, and a battle plan for the most critical refactoring in the project's history.

This message, written by the AI assistant to itself (and to any human observer reading the conversation), documents the state of a sweeping architectural transformation: replacing a fragile, static concurrency-limiting system with a robust, budget-based memory manager. It is a document of unusual density and precision, packing into a single message the distilled knowledge from hundreds of preceding tool calls, file edits, and debugging sessions. To understand this message is to understand the entire memory management philosophy of the cuzk proving engine.

The Context: Why This Message Exists

The message arrives at a specific inflection point in the development workflow. The assistant has just completed modifications to five core source files (memory.rs, lib.rs, config.rs, srs_manager.rs, pipeline.rs) and is about to embark on the most complex and risky change: rewriting engine.rs, the central orchestrator of the proving pipeline. This is a 2837-line file that coordinates partition dispatch, GPU worker management, SRS loading, PCE extraction, and synthesis orchestration. Touching it is like performing open-heart surgery on a running system.

The message serves several distinct purposes:

First, it is a checkpoint. Before making invasive changes to the engine, the assistant pauses to document exactly what has been accomplished, what remains, and in what order the remaining work must proceed. This is a defensive programming practice: when the changes to engine.rs inevitably require adjustments to the already-modified files, the assistant needs a clear reference point to understand the current state of the entire system.

Second, it is a design consolidation. The "Discoveries" section captures empirical measurements about memory consumption—70 GiB baseline RSS, 44 GiB for SRS, 26 GiB for PCE, 13.6 GiB per partition working memory. These numbers were not plucked from documentation; they were discovered through instrumentation and debugging of the running system. The message formalizes these findings into design constraints.

Third, it is a coordination document. The message lists "Key Code Patterns" that span multiple files—how SrsManager is locked, how the evictor callback works, how synthesize_auto() was updated across nine call sites. These patterns represent implicit contracts between the modified modules. By documenting them explicitly, the message ensures that the engine integration respects these contracts.

Fourth, it is a risk register. The "In Progress" section for engine.rs is organized as a numbered checklist with specific line-number references to the original file. This is not casual planning; it is a surgical protocol for a high-risk refactoring where a single mistake could crash the proving daemon and halt Filecoin storage proof generation.

The Architecture: A Unified Memory Budget

The core design documented in this message is the replacement of the partition_workers semaphore with a unified memory budget. To understand why this matters, one must understand the old architecture.

In the original system, the number of concurrent partition proofs was limited by a static configuration value: partition_workers. This was a crude throttle—it prevented too many GPU proofs from running simultaneously, but it had no awareness of actual memory pressure. The system could have 70 GiB of SRS and PCE loaded, plus 13.6 GiB per partition, and the semaphore would happily dispatch partitions until it hit its worker limit, regardless of whether the system had sufficient RAM. If the limit was set too high, the process would be OOM-killed. If set too low, GPU utilization suffered.

The new architecture, as described in the message, replaces this with a single byte-level budget auto-detected from system RAM:

Single unified budget (SRS pinned + PCE heap + working set all share one pool) Auto-detect from system RAM (/proc/meminfo MemTotal - safety_margin) SRS/PCE loaded on-demand — configurable preload removed entirely Evict under pressure — only evict SRS/PCE entries idle ≥5 minutes AND when budget is needed

This is a fundamental shift in philosophy. Instead of asking "how many partitions can we run?", the system now asks "how much memory do we have, and how should we allocate it across competing consumers?" The budget is a single pool that all memory consumers—SRS (pinned GPU memory), PCE (heap-allocated pre-compiled circuits), and per-partition working memory—draw from.

The message reveals a crucial design subtlety: the budget is not merely a counter but a smart allocator with LRU eviction. When a partition needs memory and the budget is exhausted, the system can evict idle SRS or PCE entries (those unused for ≥5 minutes) to free space. This is a dramatic improvement over the old system, where SRS and PCE were loaded at startup and never released.

The Discoveries: Empirical Memory Architecture

One of the most valuable contributions of this message is the "Discoveries" section, which documents the actual memory consumption patterns of the proving engine. These numbers were not known a priori—they were discovered through the debugging process that preceded this message.

Baseline RSS ≈ 70 GiB: SRS ~44 GiB (CUDA pinned) + PCE ~26 GiB (heap) Per-partition working memory ≈ 13.6 GiB: a/b/c vectors ~12.5 GiB + aux ~0.74 GiB + density ~48 MB Two-phase GPU release: prove_start drops a/b/c synchronously (~12.5 GiB), prove_finish drops rest (~1.1 GiB)

These numbers are remarkable. A single 32 GiB PoRep proof consumes approximately 13.6 GiB of working memory per partition, on top of a 70 GiB baseline. This means that on a machine with, say, 256 GiB of RAM, the system can run at most ~13 concurrent partitions (256 - 70 = 186 GiB available, divided by 13.6 GiB per partition). But this calculation changes if SRS or PCE can be evicted, because those 70 GiB are not fixed—they can be released under pressure.

The two-phase release pattern is particularly elegant. The GPU proving API (prove_start / prove_finish) has a specific memory lifecycle: the large a/b/c vectors can be freed immediately after prove_start completes, while the remaining aux and density memory must persist until prove_finish. By splitting the release into two phases, the system reclaims ~12.5 GiB (92% of the working memory) earlier, making it available for the next partition sooner.

The Code Patterns: Contracts Between Modules

The message documents several critical code patterns that represent contracts between the modified modules. These are worth examining in detail because they reveal the design's sophistication.

SrsManager Locking Protocol

SrsManager is behind Arc<Mutex<SrsManager>> (tokio mutex) — use blocking_lock() from sync contexts, .lock().await from async

This is a subtle but important distinction. The SrsManager is shared between async code (the main proving pipeline, which uses tokio tasks) and synchronous code (the evictor callback, which runs in a synchronous context). The tokio mutex (tokio::sync::Mutex) provides both .lock().await for async contexts and blocking_lock() for synchronous contexts. The message explicitly documents this dual access pattern to prevent deadlocks.

The Evictor Callback

The evictor callback is Arc<dyn Fn(u64) -> u64 + Send + Sync> — synchronous, calls blocking_lock() on SrsManager

The evictor is a closure that the budget calls when it needs to free memory. It receives the number of bytes needed and returns the number of bytes actually freed. The callback is synchronous because it's called from within the budget's acquire() method, which may be called from async contexts. This means the evictor must use blocking_lock() when accessing the SrsManager.

MemoryReservation Lifecycle

MemoryReservation::into_permanent() is used to transfer budget ownership from temporary reservations to long-lived caches (SRS/PCE)

This is a key design pattern. When a partition proof needs SRS data, it first acquires budget for the SRS size, then calls SrsManager::ensure_loaded() with the reservation. The SrsManager calls into_permanent() on the reservation, transferring ownership of the budget bytes from the temporary partition context to the permanent SRS cache. This ensures that the budget accurately reflects who is using memory at all times.

The Accomplished Work: Five Files Transformed

The message documents the completion of five files, each representing a significant engineering effort:

memory.rs (Created)

This is the foundation of the entire system. It contains MemoryBudget (the central allocator with acquire/release/eviction), MemoryReservation (a RAII guard that releases budget on drop or via explicit release()), detect_system_memory() (reads /proc/meminfo), and all estimation constants for different proof types (PoRep, WinningPoSt, WindowPoSt, SnapDeals). The inclusion of comprehensive tests indicates that this module was built with confidence in its correctness.

config.rs (Updated)

The configuration system was restructured to remove deprecated fields (pinned_budget, working_memory_budget, partition_workers, preload) and add new fields (total_budget, safety_margin, eviction_min_idle). The backward compatibility strategy is noteworthy: deprecated fields are kept as Option<T> with #[serde(default)], and warnings are logged when they appear in config files. This allows existing deployments to upgrade without breaking their configuration.

srs_manager.rs (Updated)

The SRS manager was made budget-aware. It now tracks last_used timestamps for each circuit ID, supports evictable_entries() (which checks Arc::strong_count() == 1 to ensure no other references exist), and evict() which returns the number of bytes freed. The ensure_loaded() method now accepts an optional MemoryReservation and calls into_permanent() to transfer budget ownership.

pipeline.rs (Updated)

This was a major refactoring. The four static OnceLock<PreCompiledCircuit<Fr>> variables were removed and replaced with a PceCache struct. All four extract_and_cache_pce_from_* functions (c1, winning_post, window_post, snap_deals) were updated to take &PceCache. The synthesize_auto() function was updated to accept Option<&PceCache>, and all nine call sites were updated to pass None temporarily (the real PceCache will be wired through from engine.rs).

The In Progress Work: engine.rs — The Heart of the System

The message's "In Progress" section for engine.rs is the most detailed and reveals the complexity of the remaining work. The assistant has already made three changes:

  1. Added reservation: Option<MemoryReservation> to SynthesizedJob
  2. Added budget: Arc<MemoryBudget> and pce_cache: Arc<PceCache> to Engine
  3. Updated Engine::new() to create budget, PceCache, and pass budget to SrsManager But the bulk of the work remains in Engine::start(), which is described as requiring: - Removal of SRS preload block (lines ~866-906) - Removal of PCE preload block (lines ~929-937) - Wiring the evictor callback after GPU detection - Replacing partition_workers channel capacity sizing with budget-derived values - Removing partition_semaphore creation (lines ~1074-1078) The partition dispatch paths (PoRep at lines ~1317-1568, SnapDeals at lines ~1580-1831) need to be converted from semaphore-based gating to budget-based gating. The GPU worker loop (lines ~2238-2529) needs the two-phase release pattern implemented. The message organizes these changes as a numbered checklist with specific line numbers. This is not just a todo list; it is a surgical protocol. Each item references exact locations in the source file, and the order matters—preload removal must happen before evictor wiring, which must happen before dispatch conversion.

Assumptions and Potential Pitfalls

The message contains several assumptions that are worth examining critically:

Assumption: Auto-detected memory is reliable. The system reads /proc/meminfo MemTotal and subtracts a safety margin. This assumes that the system's total RAM is the relevant constraint. In practice, other processes consume memory, and the kernel's memory overcommit behavior can complicate this picture. The safety margin is intended to account for this, but the margin's adequacy depends on the deployment environment.

Assumption: LRU eviction with a 5-minute idle threshold is safe. The eviction policy only evicts entries that have been idle for at least 5 minutes. This assumes that the cost of reloading an evicted entry (reading from disk, potentially re-downloading from the network) is acceptable if the entry is needed again. For frequently used circuit types, this could cause performance degradation if entries are repeatedly evicted and reloaded.

Assumption: Backward compatibility via serde's ignore-unknown-fields is sufficient. The message notes that "Serde ignores unknown fields by default, so old TOML configs with removed fields parse fine." This is true for serde's default behavior, but it assumes that users haven't made typographical errors in field names that happen to match the old field names. The warn_deprecated() mechanism mitigates this by alerting users to update their configs.

Assumption: The memory estimation constants are accurate. The constants for PoRep, WinningPoSt, WindowPoSt, and SnapDeals proof sizes are based on empirical measurements. If these measurements were taken under specific conditions (e.g., specific sector sizes, specific GPU models), they may not generalize to all deployments.

The Thinking Process Visible in the Message

The message reveals a structured, methodical thinking process. The assistant is operating under a clear mental model:

  1. Decomposition: The problem is broken into discrete, independently verifiable steps. Each file change is a self-contained unit that can be reviewed and tested.
  2. Risk management: The most complex change (engine.rs) is deferred until the supporting infrastructure (memory.rs, config.rs, srs_manager.rs, pipeline.rs) is complete and tested. This minimizes the risk of cascading failures.
  3. Traceability: Every change is linked to specific line numbers in the original files. This is not just for the assistant's benefit—it creates an audit trail that a human reviewer could follow.
  4. Contract documentation: The "Key Code Patterns" section documents the implicit contracts between modules. This is the assistant's way of saying "I've designed these interfaces to work together, and here are the rules that must be followed."
  5. Progressive refinement: The message shows evidence of iterative thinking. The synthesize_auto() call sites were first updated manually, then the assistant realized the pattern was uniform and used sed to batch-update the remaining eight calls. This demonstrates adaptive problem-solving.

The Output Knowledge Created

This message creates several forms of knowledge:

For the assistant itself: It serves as a working memory extension. The assistant cannot remember every line of every file across hundreds of tool calls, so this message externalizes the critical state.

For human reviewers: It provides a comprehensive overview of the memory manager implementation, including design decisions, empirical measurements, and the current status of each component.

For future debugging: If something goes wrong during the engine.rs integration, this message provides a baseline. The assistant can compare the actual behavior against the documented design to identify discrepancies.

For the todo system: The message updates the todo list (visible in the conversation as todowrite blocks) with the current status of each task. This creates a feedback loop between the assistant's planning and execution.

Conclusion

Message <msg id=2151> is far more than a status update. It is a design document, a risk assessment, a coordination protocol, and a surgical plan rolled into one. It captures the moment when a complex refactoring transitions from infrastructure work to core integration—from modifying supporting modules to rewriting the central orchestrator.

The message's density reflects the complexity of the system it describes. Every sentence encodes design decisions that were discovered through debugging, measured through instrumentation, and validated through testing. The "Discoveries" section alone represents hours of empirical investigation into memory consumption patterns that were previously opaque.

For anyone seeking to understand the cuzk proving engine's memory architecture, this message is the definitive reference. It explains not just what the system does, but why it does it, what assumptions it makes, and how the pieces fit together. It is a testament to the value of structured, documented thinking in complex software engineering—and a reminder that the most important code is sometimes the code that never gets compiled, but instead organizes the mind of the engineer writing it.