The Accumulating Baseline: Why SRS and PCE Never Leave Memory in the cuzk GPU Proving Engine

Introduction

In the middle of a deep architectural design session for a new memory management system in the cuzk GPU proving engine, a seemingly simple question from the user triggered one of the most revealing exchanges in the conversation. The user asked: "Is PoRep SRS/PCE unloaded when e.g. Snap proof needs to be made? Are both kept in memory?" The assistant's response—message 2068 in the conversation—appears straightforward on its surface, but it exposes a critical architectural blind spot that had been lurking in the codebase since its inception. This article examines that message in depth, unpacking the reasoning, evidence, assumptions, and design implications it contains.

Context: The Memory Management Architecture Design

The subject message is part of a larger sub-session (segment 14) dedicated to designing a comprehensive memory management architecture for the cuzk GPU proving engine. The assistant had been conducting a forensic audit of the entire memory lifecycle for 32 GiB PoRep proofs, tracing every allocation and deallocation point. A critical finding had already emerged: the existing working_memory_budget configuration option was entirely dead code—never enforced—leaving a static partition_workers semaphore as the only throttle on memory consumption. This semaphore was memory-unaware and fragile, incapable of preventing out-of-memory (OOM) conditions when multiple proof types were processed concurrently.

In the message immediately preceding the subject (msg 2066), the assistant had provided a comprehensive analysis of how SRS (Structured Reference String) and PCE (Pre-Compiled Constraint Evaluator) fit into the memory picture. That analysis detailed loading paths, memory characteristics, transient spikes, and per-proof-type sizes. The user absorbed this information and then asked the natural follow-up: when the system switches between proof types—say from PoRep to SnapDeals—are the old proof type's SRS and PCE unloaded to make room?

The Subject Message: A Direct Answer with Far-Reaching Implications

The assistant's response in message 2068 is concise but dense with evidence and implication. Here is the message in full:

Both are kept in memory. Neither SRS nor PCE is ever evicted to make room for another circuit type. Here's the evidence:

SRS: SrsManager (srs_manager.rs:117-126) stores loaded entries in a HashMap<CircuitId, Arc<SuprasealParameters>>. ensure_loaded (srs_manager.rs:148-208) checks the map and loads if absent — it never evicts an existing entry to make room. The evict() method exists (srs_manager.rs:237-249) but is only wired to the gRPC EvictSRS RPC — it's never called internally by the engine. The budget check at line 173 is advisory (logs a warning, loads anyway).

PCE: Each circuit type has its own static OnceLock (pipeline.rs:293-305 — POREP_32G_PCE, WINNING_POST_PCE, WINDOW_POST_PCE, SNAP_DEALS_PCE). OnceLock by definition is write-once, never cleared. There is no eviction path at all.

So if the daemon processes both PoRep and SnapDeals proofs over its lifetime:

``` SRS PoRep 32G ......... 44 GiB (pinned) SRS SnapDeals 32G ..... 33 GiB (pinned) PCE PoRep 32G ......... 26 GiB (heap) PCE SnapDeals 32G ..... ~17 GiB (heap, estimated from ~81M constraints) ────────────────────────────────── Baseline ............. ~120 GiB

On a 256 GiB machine: 256 - 120 - 10 (OS) = ~126 GiB for working set → only ~9 partitions at 13.6 GiB each (vs ~12 with PoRep alone) ```

And if WindowPoSt also shows up:

`` + SRS WindowPoSt 32G ... 57 GiB (pinned) + PCE WindowPoSt 32G ... ~26 GiB (heap) ────────────────────────────────── Baseline ............. ~203 GiB → only ~3 partitions fit on 256 GiB ``

This is another argument for the memory budget being dynamic rather than a fixed partition_workers count — as SRS/PCE accumulate for additional proof types, the available working memory shrinks, but nothing adjusts the partition concurrency downward.

It also raises a design question for the memory management work: should we add LRU eviction for SRS and/or PCE, or is it acceptable to require operators to preload only the circuit types they expect to use? PCE can be reloaded from disk in ~5s, and SRS in ~30-60s, so eviction would be feasible if we accept that latency on the first proof after re-load.

Why This Message Was Written: The Reasoning and Motivation

The message was motivated by a gap in the user's understanding that, once filled, would fundamentally reshape the memory management architecture being designed. The user had just received a detailed breakdown of SRS and PCE memory characteristics, but the natural next question was about dynamics: do these structures come and go as proof types change, or do they accumulate permanently? The answer to this question determines whether the baseline memory footprint is fixed or grows over the lifetime of the daemon.

The assistant's motivation was not merely to answer a yes/no question but to demonstrate, through concrete code evidence, that the current architecture has an unbounded baseline memory growth problem. Every new proof type encountered during the daemon's lifetime adds its SRS and PCE permanently, shrinking the available working memory for partition synthesis. This is a classic "memory leak" pattern—not a leak in the traditional sense of allocated-but-lost pointers, but a permanent retention of data that could be evicted and reloaded on demand.

The assistant structured the answer in three parts: (1) direct evidence from the code proving no eviction occurs, (2) concrete memory calculations showing the cumulative impact, and (3) a design question that reframes the problem as an explicit architectural choice rather than an oversight.

How Decisions Were Made: Evidence-Driven Analysis

No decisions were made in this message—it is an analysis, not a decision point. However, the message sets the stage for decisions that will be made later in the session. The assistant's approach is methodical: cite specific line numbers, quote data structures (HashMap, OnceLock), and trace the code paths that confirm the behavior.

The evidence for SRS is particularly thorough. The assistant identifies three distinct code locations that collectively prove the point:

srs_manager.rs:117-126 — the HashMap storage that never removes entries
srs_manager.rs:148-208 — the ensure_loaded function that only adds, never evicts
srs_manager.rs:237-249 — the evict() method that exists but is only reachable via an external gRPC call, never invoked internally This tripartite evidence structure—storage, loading, eviction—leaves no ambiguity. The assistant is effectively saying: "I've checked every path that could possibly unload SRS, and none of them fire automatically." For PCE, the evidence is even more definitive. The OnceLock type in Rust is, by its very semantics, a write-once, never-clear synchronization primitive. The assistant cites the specific static variables (POREP_32G_PCE, WINNING_POST_PCE, etc.) to show that each circuit type has its own independent OnceLock, and there is no mechanism—not even an unused one—to clear them.

Assumptions Made by the Assistant

The assistant makes several assumptions in this message, most of which are well-founded but worth examining:

The codebase is the authoritative source of truth: The assistant assumes that the current behavior as expressed in the code is the intended behavior. This is a reasonable assumption for an analysis, but it's worth noting that the evict() method on SrsManager exists and is wired to a gRPC endpoint—suggesting that someone, at some point, anticipated the need for eviction. The fact that it's never called internally may be an oversight rather than a deliberate design choice.
The memory sizes are accurate: The assistant provides specific numbers (44 GiB for PoRep SRS, 26 GiB for PoRep PCE, 33 GiB for SnapDeals SRS, etc.). These are drawn from earlier analysis in the session (msg 2066) and are based on file sizes and code inspection. The SnapDeals PCE estimate of ~17 GiB is explicitly marked as "estimated from ~81M constraints," acknowledging uncertainty.
256 GiB is the reference machine: The assistant assumes a 256 GiB machine for the calculations, which is consistent with the hardware context of the session. This is a reasonable baseline for high-end GPU proving machines.
The operator can control which circuit types are encountered: The assistant's design question—"is it acceptable to require operators to preload only the circuit types they expect to use?"—assumes that the set of proof types a daemon will encounter is predictable. In a production Filecoin storage provider, this may or may not be true depending on the sector portfolio.
Reload times are acceptable: The assistant states that PCE can be reloaded from disk in ~5s and SRS in ~30-60s. These numbers come from earlier analysis and assume warm caches and adequate I/O bandwidth. They do not account for the transient memory spikes during reload (which the assistant had documented in msg 2066).

Potential Mistakes or Incorrect Assumptions

While the message is factually accurate based on the code as written, there are some nuances worth examining:

The eviction question is framed as binary: The assistant presents LRU eviction as the alternative to "keep everything forever." But there are intermediate strategies—for example, keeping only the most recently used circuit type's SRS/PCE in memory and evicting others, or using a time-based expiration rather than LRU. The assistant's framing is intentionally provocative to highlight the problem, but the design space is richer than the message suggests.
The WindowPoSt numbers may overstate the problem: The assistant adds WindowPoSt SRS (57 GiB) and PCE (~26 GiB) to the baseline, arriving at ~203 GiB. This assumes all three proof types (PoRep, SnapDeals, WindowPoSt) are loaded simultaneously. In practice, a single daemon instance may only handle a subset of proof types. The assistant is presenting a worst-case scenario, which is valid for design purposes but could be misinterpreted as the typical case.
The "baseline" framing conflates pinned and heap memory: The assistant adds pinned CUDA memory (SRS) and regular heap memory (PCE) together as "baseline." While both consume physical RAM, they have different characteristics: pinned memory cannot be swapped and has different pressure on the system. The unified "baseline" number is useful for high-level reasoning but glosses over these distinctions.
The assumption that PCE extraction is always ~26 GiB: The PCE size varies by circuit type. The assistant provides ~26 GiB for PoRep and WindowPoSt, and ~17 GiB for SnapDeals. But these are estimates based on constraint counts, not measured values. The actual memory footprint could differ based on CSR matrix density and other factors.

Input Knowledge Required to Understand This Message

To fully grasp the subject message, a reader needs knowledge in several domains:

Rust programming language: Understanding of HashMap, Arc, OnceLock, and static variables. The OnceLock type is particularly important—it's a Rust 1.70+ synchronization primitive that allows lazy initialization with a write-once semantic. The fact that it can never be cleared is central to the PCE analysis.
CUDA memory model: Understanding of pinned (page-locked) host memory, why it's used for SRS (DMA transfers to GPU), and why it cannot be swapped. The distinction between pinned and regular heap memory is crucial for interpreting the memory calculations.
Filecoin proof types: Knowledge of PoRep (Proof of Replication), WindowPoSt (Window Proof of Spacetime), WinningPoSt, and SnapDeals. Each has different circuit sizes and SRS requirements. Without this context, the specific sizes (44 GiB vs 57 GiB vs 184 MiB) are just numbers.
Groth16 proving system: Understanding of SRS as the structured reference string (the proving key in a Groth16 setup) and PCE as a pre-compiled constraint evaluator that caches the fixed R1CS structure. The distinction between the proving key (SRS) and the circuit structure (PCE) is fundamental.
The cuzk architecture: Knowledge of the engine's partition dispatch system, the partition_workers semaphore, the synthesis pipeline, and the GPU worker loop. The message references these implicitly through the "working set" and "partitions" terminology.
The broader memory management design context: The message is part of a session designing a MemoryBudget/MemoryReservation system. The assistant's argument about dynamic budgets versus fixed partition_workers counts only makes sense within this context.

Output Knowledge Created by This Message

The message creates several pieces of new knowledge that advance the design session:

Confirmation of unbounded baseline growth: The message definitively establishes that the baseline memory footprint grows monotonically with the number of distinct proof types encountered. This is not a leak in the traditional sense but a design property that must be addressed.
Quantified cumulative impact: The message provides concrete numbers showing that a daemon handling PoRep + SnapDeals has a ~120 GiB baseline, leaving only ~126 GiB for working set on a 256 GiB machine. Adding WindowPoSt pushes the baseline to ~203 GiB, leaving only ~3 partitions' worth of working memory. These numbers make the problem tangible.
A design question that reframes the problem: By asking "should we add LRU eviction or require preloading only expected types?" the assistant transforms the problem from a technical limitation into an explicit architectural choice with known trade-offs. The reload latency numbers (~5s for PCE, ~30-60s for SRS) provide the data needed to evaluate that choice.
Evidence that the existing eviction mechanism is incomplete: The discovery that SrsManager::evict() exists but is only wired to a gRPC RPC (never called internally) is itself a finding. It suggests that eviction was considered at some point but never fully integrated into the engine's lifecycle.
A concrete recommendation for the memory budget design: The message reinforces the argument from msg 2066 that the memory budget must be dynamic and auto-computed from available RAM minus baseline consumption, rather than a static configured value.

The Thinking Process Visible in the Message

The assistant's reasoning process is visible in the structure of the message. It follows a clear logical flow:

Direct answer: "Both are kept in memory. Neither SRS nor PCE is ever evicted." This is the thesis statement.
Evidence for SRS: The assistant traces the code paths methodically—storage (HashMap), loading (ensure_loaded), and the unused eviction path (evict() method). This is a forensic approach: check every possible mechanism that could remove SRS from memory.
Evidence for PCE: The assistant cites the OnceLock type semantics and the specific static variables. This is even more definitive than the SRS case because OnceLock has no clearing mechanism at all by design.
Quantified impact: The assistant then shifts from code analysis to mathematical modeling, showing the cumulative baseline for different proof type combinations. This makes the abstract problem concrete.
Design implication: The assistant connects the finding back to the ongoing memory budget design, arguing that a dynamic budget is necessary because the baseline is not fixed.
Design question: Finally, the assistant opens a new design question about LRU eviction, providing reload latency data to inform the decision. This structure—answer, evidence, quantification, implication, question—is characteristic of the assistant's analytical style throughout the session. It moves from specific code facts to general design principles, always grounding abstractions in concrete evidence.

Conclusion

Message 2068 is a pivotal moment in the memory management design session. It answers a straightforward question with a finding that has profound implications: the cuzk engine's baseline memory footprint grows without bound as new proof types are encountered, because SRS and PCE are never evicted. The message transforms this from a vague concern into a quantified problem with specific numbers, code evidence, and a clear design question that must be answered. It exemplifies the kind of forensic code analysis that drives good architectural decisions—finding the hidden assumptions in existing code and exposing them to explicit design scrutiny. The message's legacy is the LRU eviction design that was subsequently incorporated into the cuzk-memory-manager.md specification, directly addressing the accumulating baseline problem it identified.