The Specification That Replaces Fragile Concurrency Limits with Memory-Aware Admission Control

In the course of a deep engineering session on the cuzk GPU proving engine, a single message arrived that marked the transition from design exploration to concrete deliverable. Message [msg 2078] is the assistant's confirmation that a comprehensive specification document, cuzk-memory-manager.md, has been written and saved to disk. On its surface, the message is a brief status update — a bullet-point summary of what the document covers, followed by a note dismissing unrelated LSP errors. But to understand why this message matters, one must trace the chain of reasoning that led to it: a forensic audit of memory allocation across the proving pipeline, the discovery that a critical configuration knob was entirely dead code, and a collaborative refinement of a new architecture that replaces a fragile static concurrency limit with a robust, memory-aware admission control system.

The Problem: A Proving Engine Built on Fragile Assumptions

The cuzk engine is a high-performance GPU proving system for Filecoin proofs. It handles several proof types — PoRep (Proof of Replication), WindowPoSt, WinningPoSt, and SnapDeals — each with different memory footprints. The engine's memory lifecycle involves multiple phases: SRS (Structured Reference String) loading consumes roughly 44 GiB of pinned CUDA memory; PCE (Pre-Compiled Constraint Evaluator) caching occupies about 26 GiB on the heap; per-partition synthesis allocates approximately 13.6 GiB for a/b/c/aux data; and GPU proving adds its own demands. For a full 10-partition PoRep batch, the total can exceed 136 GiB.

The existing system managed this complexity with a set of static knobs: a working_memory_budget config option, a pinned_budget setting, a partition_workers semaphore that limited concurrent partition processing, and an SRS preload list. In theory, these knobs allowed operators to tune memory usage. In practice, as the assistant discovered during the audit, the working_memory_budget was entirely dead code — it was parsed from configuration but never enforced anywhere in the engine. The only real throttle was the partition_workers semaphore, a static integer that limited how many partition proofs could run concurrently. This semaphore was memory-unaware: it didn't know how much RAM was actually available, couldn't adapt to different proof types with different memory footprints, and provided no mechanism for evicting cached data under pressure. The system was one configuration mistake away from an OOM kill.

From Audit to Architecture

The assistant's investigation ([msg 2073]) traced every allocation and deallocation point across the entire pipeline. The findings were stark: the SRS preload loop eagerly loaded every listed circuit at startup, pinning tens of gigabytes before any work began; the PCE was similarly preloaded into static OnceLock globals that could never be freed; and the partition_workers semaphore, set to a fixed value like 12, would admit that many concurrent proofs regardless of whether the system had 256 GiB or only 64 GiB of RAM.

The assistant proposed a unified memory budget system: a single MemoryBudget struct that auto-detects total system RAM from /proc/meminfo, subtracts a safety margin, and tracks used memory atomically. Instead of a static semaphore, the budget would gate admission: each partition or proof would acquire() an estimated amount of memory before proceeding, and if the budget was exhausted, the acquirer would either trigger eviction of idle cached data (SRS or PCE entries that hadn't been used in 5 minutes) or block until memory was freed. The design introduced MemoryReservation, an RAII handle that travels with a job through the pipeline, supporting partial release — the a/b/c buffers (~12.5 GiB) are freed immediately after GPU prove start, while the remaining ~1.1 GiB is released only after proof finalization.

The User's Directive

The user's response ([msg 2074]) was concise but consequential: "write down cuzk-memory-manager.md with all details needed for a new agents to implement correctly. Keep synth concurrency config, memory-bw/cpu contention is a separate constraint, partition-workers probably not needed."

This directive resolved the one open question from the assistant's plan. The assistant had asked whether both synthesis_concurrency and partition_workers should be removed, noting that partition_workers was purely a memory gate (directly replaceable by the budget) while synthesis_concurrency controlled CPU thread contention — a separate concern. The user confirmed this distinction: keep synthesis_concurrency as a CPU-scheduling knob, remove partition_workers since the budget subsumes its role. This decision reflects a clear architectural principle: separate concerns should have separate controls. Memory pressure is handled by the budget; CPU contention is handled by synthesis concurrency. Mixing them in a single semaphore was the original design's flaw.

What the Message Delivers

Message [msg 2078] reports that the specification has been written. The assistant enumerates eight major sections of the document:

MemoryBudget + MemoryReservation — the core of the new system, implemented in a new memory.rs module. MemoryBudget provides atomic accounting of used memory, an evictor callback for reclaiming cached data, and a tokio::sync::Notify mechanism for waking blocked acquirers. MemoryReservation is an RAII guard that supports partial release and automatically releases any remaining reservation on drop.
PceCache — replaces the four static OnceLock<PreCompiledCircuit> globals (one per proof type) with an evictable HashMap<CircuitId, Arc<PreCompiledCircuit>>. Each entry tracks its last_used timestamp and size in bytes, enabling LRU eviction under memory pressure.
SrsManager changes — adds last_used tracking, an evictable_entries() method that returns entries idle for at least 5 minutes with a strong count of 1 (meaning no in-flight proof is using them), budget-gated ensure_loaded(), and budget-releasing evict().
Config simplification — removes srs.preload, memory.pinned_budget, memory.working_memory_budget, and synthesis.partition_workers. Adds memory.total_budget (default "auto"), memory.safety_margin, and eviction_min_idle.
Engine integration — partition dispatch uses budget.acquire() instead of the semaphore. SynthesizedJob carries a MemoryReservation. The GPU worker performs a partial release after prove_start (freeing a/b/c buffers) and drops the reservation after prove_finish.
Eviction policy — oldest-first LRU, only entries idle ≥5 minutes with Arc::strong_count == 1, triggered when acquire() cannot be fulfilled from the remaining budget.
Memory estimation constants — per proof type, rounded up to GiB, covering PoRep partitions, batches, WindowPoSt, WinningPoSt, SnapDeals, and PCE extraction transient.
Acquire loop pseudocode, logging specification, testing strategy, and migration/backward-compatibility notes.

The Thinking Behind the Summary

The message's structure reveals the assistant's priorities. The bullet points are ordered by architectural importance: first the core types that everything else depends on, then the two caches that need eviction support, then the config changes that simplify the operator experience, then the engine integration that ties it all together. The eviction policy and estimation constants come later — they are details within the larger framework, not foundational decisions.

The final sentence — "The LSP errors in the output are pre-existing in unrelated Go files — not caused by this change" — is a small but telling detail. The assistant is preemptively addressing a concern the user might have. When the write tool executed, the system reported LSP diagnostics from unrelated Go source files. Rather than letting those errors cause confusion or concern, the assistant explicitly notes they are pre-existing and unrelated. This is the mark of an experienced engineer who understands that clean communication includes managing the signal-to-noise ratio of tool output.

Assumptions Embedded in the Specification

The specification, as summarized, rests on several assumptions that are worth examining. First, it assumes that /proc/meminfo is available and reliable for auto-detecting system RAM. This is true on Linux but would fail in containerized environments with memory limits — a container sees the host's MemTotal, not its own cgroup limit. The spec may need a fallback to cgroup/memory.max or an explicit configuration override for such environments.

Second, it assumes that the memory estimation constants are accurate enough to prevent OOM. The constants are rounded up to GiB, which provides some safety margin, but they are static estimates. If a new proof type is added or an existing one changes, the constants must be updated. There is no dynamic measurement or self-calibration.

Third, the 5-minute idle threshold for eviction is a heuristic. In a system processing proofs continuously, 5 minutes of idle time for an SRS or PCE entry suggests it is unlikely to be needed again soon. But if the workload is bursty — periods of intense activity followed by idle periods — the eviction might be too aggressive, causing reload overhead when activity resumes.

Fourth, the design assumes that Arc::strong_count == 1 is a sufficient check for "not in use." This is correct for the common case where each in-flight proof holds a clone of the Arc. But if there are other long-lived references (e.g., held by background tasks or diagnostic code), the eviction might be blocked even when the cache entry is effectively unused.

Why This Message Matters

Message [msg 2078] is the moment when an architectural vision becomes a concrete artifact. The assistant had spent the preceding messages analyzing, designing, and debating. The user had provided critical feedback. Now, the assistant has produced a document that is "directly actionable for implementation" — a specification that a new agent could pick up and implement without needing to re-discover all the reasoning that led to it.

The message also demonstrates a key skill in AI-assisted engineering: the ability to synthesize a complex design into a clear, structured specification. The eight bullet points are not just a checklist; they are a map of dependencies. You cannot implement the engine integration without first having MemoryBudget and MemoryReservation. You cannot implement eviction without PceCache and the SrsManager changes. The ordering reflects the build order.

Finally, the message closes a chapter. The next step is implementation — translating the spec into code across seven files, removing dead configuration, and wiring the new memory-aware admission control into the proving pipeline. The specification ensures that this implementation can be done by a different agent, or by the same agent after a context reset, without losing the design intent. That is the mark of a well-crafted specification: it captures not just what to build, but why it is built that way, so that the implementer can make sound decisions when encountering edge cases the spec did not anticipate.