From Compilation to OOM: The Complete Journey of Deploying a Budget-Based Memory Manager

The arc of a complex software engineering project rarely follows a clean line from specification to completion. More often, it traces a jagged path through compile errors, unit tests, real-world validation, deployment surprises, and production debugging. This is the story of one such arc: the final integration, validation, and deployment of a budget-based memory manager for the cuzk GPU proving daemon — a journey that began with fixing two compile errors and ended with an OOM-killed daemon and a crucial insight about safety margins.

The Final Integration Fixes

The chunk opens with the assistant in the closing stages of a massive refactoring [1][4]: replacing the static partition_workers semaphore with a unified MemoryBudget that tracks all major memory consumers (SRS pinned, PCE heap, and synthesis working set) under a single byte-level budget. The specification had been written, the core implementation completed across seven files, and the major architectural changes were in place. What remained were the loose ends — the final API migration points that would make the whole thing compile.

Three specific tasks needed attention. First, the third extract_and_cache_pce_from_c1 call in cuzk-bench/src/main.rs needed to accept the new PceCache parameter. Second, the SrsManager::new call in the same function needed conversion from the old (PathBuf, u64) signature to the new (PathBuf, Arc<MemoryBudget>) signature. Third, the entire codebase needed to be checked for any remaining references to the removed get_pce API.

These were straightforward mechanical changes, but they represented the final bridge between the old architecture and the new. Every call site that still used the old API was a potential compile error waiting to surface. The assistant methodically updated each one, then ran cargo check to see what remained.

Two Compile Errors That Nearly Derailed Everything

The first cargo check [19] revealed two compile errors — small in scope but significant in what they revealed about the architecture:

synth_job mutability: In engine.rs:2416, the synth_job binding needed to be mut to allow .take() on the reservation field. This was a Rust ownership issue: the MemoryReservation inside SynthesizedJob was being moved out via .take(), which requires mutable access. The fix was a single word — adding mut — but it highlighted how the new budget system threaded ownership through the entire synthesis pipeline.
The 'static lifetime ghost: In pipeline.rs:1143, the pce_arc variable had a lifetime issue. The function synthesize_with_pce required a 'static lifetime on its pce parameter, but the local Arc<PreCompiledCircuit> returned from PceCache.get() didn't satisfy this constraint. This was a leftover from the old architecture, where PCE data was stored in static OnceLock variables and could be borrowed for the program's entire lifetime. The new PceCache stores Arc-wrapped data, but the 'static requirement was still baked into the function signature. The second error was particularly instructive [24][25]. It revealed how architectural assumptions can become fossilized in type signatures. The 'static bound made perfect sense when PCE data lived in a global OnceLock — it was literally static. But when the architecture shifted to a runtime-managed PceCache, the data's lifetime became dynamic. The bound was no longer valid, and removing it was the correct fix. This is a classic pattern in large refactorings: the code that should have been updated during the main implementation phase but wasn't, because it was hidden behind a function signature that "worked" until the new types were plugged in.

The Moment of Compilation

With those two fixes applied, cargo check returned clean — zero errors [28][29]. The assistant then ran the full test suite: all 8 memory module tests passed, all 7 config tests passed. The implementation was not just syntactically correct but semantically validated by the unit tests.

But the real validation came next. The assistant ran pce-bench [51][52] — a real-world benchmark that loads a 32 GiB PoRep C1 output, extracts the PCE (Pre-Compiled Circuit), and validates correctness across all 10 circuits (130 million constraints each). This was the acid test: would the new PceCache correctly store and retrieve the Arc<PreCompiledCircuit>? Would the MemoryBudget track usage correctly? Would the PCE data be correct end-to-end?

The results were clean. Baseline synthesis completed in 50.9 seconds. PCE extraction took 68.2 seconds — a one-time cost, with the 25.7 GiB PCE data cached in the new PceCache. PCE synthesis completed in 52.1 seconds, using PceCache.get() to retrieve the cached circuit. All 10 circuits matched between baseline and PCE paths. The memory manager worked.

The Commit That Sealed It

With validation complete, the assistant committed the changes [57]. The commit message tells the story concisely:

Replace the static partition_workers semaphore with a unified MemoryBudget that tracks all major memory consumers (SRS pinned, PCE heap, synthesis working set) under a single byte-level budget auto-detected from system RAM.

The diff statistics are staggering: 9 files changed, 2,629 insertions, 592 deletions. Two new files created (memory.rs and the specification document). This was not a small refactoring — it was a fundamental re-architecture of how the GPU proving daemon manages its most critical resource.

The Deployment Directive

The user's next instruction shifted the focus from development to deployment [56][58]: "Commit, test on ssh -p 40612 root@141.0.85.211 which is running curio and cuzk, curio is cordoned; upload new cuzk (built in docker) and test if memory constraints work correctly."

This was the moment the implementation would meet reality. The remote machine had 755 GiB RAM, an RTX 5090 (32 GB VRAM), and 64 cores. Curio was running (cordoned, meaning it wasn't accepting new work). The existing cuzk binary was at /usr/local/bin/cuzk, and the config at /tmp/cuzk-run-config.toml still used the deprecated preload and partition_workers fields.

The assistant first performed reconnaissance [59][60][61] — checking free memory, running processes, GPU configuration, and the existing config. The machine had 755 GiB total, with 227 GiB used and 528 GiB available. The RTX 5090 was detected with 32 GiB VRAM. Curio was running but no cuzk process was active. The config needed updating for the new memory manager.

Building and Deploying

The build process used a Docker rebuild workflow. The Dockerfile.cuzk-rebuild produced a minimal FROM scratch image containing only the cuzk-daemon binary. The assistant built the image, extracted the binary using docker cp, and uploaded it to the remote machine via SCP.

The first deployment attempt revealed a subtle issue: the binary replacement failed because the old daemon was still running ("Text file busy"). The assistant had to kill the old process, wait for it to die, then replace the binary. This is a classic deployment gotcha — you can't overwrite a running executable on Linux.

The Runtime Panic: `blocking_lock` in an Async Context

With the new binary deployed and started with a test config (100 GiB budget), the daemon immediately panicked [54][55]:

thread 'tokio-runtime-worker' panicked at engine.rs:913:45:
Cannot block the current thread from within a runtime.

The root cause was in the evictor callback. The MemoryBudget.acquire() method is an async function that runs on tokio worker threads. When the budget is exhausted, it calls the evictor callback to free memory. The evictor, in turn, calls srs_for_evict.blocking_lock() on the SrsManager's tokio Mutex. But blocking_lock() is designed for use from synchronous contexts (like spawn_blocking). Calling it from an async context on a tokio worker thread triggers the runtime's panic — you cannot block a thread that's driving async tasks.

This was a fundamental design tension. The evictor callback is synchronous (Arc<dyn Fn(u64) -> u64 + Send + Sync>), but it needs to access the SrsManager which is protected by a tokio async mutex. The original implementation assumed the evictor would only be called from synchronous contexts, but the budget system calls it from within the async acquire() loop.

The fix was elegant: replace blocking_lock() with try_lock(). If the mutex is held (because another async task is currently using the SrsManager), the evictor simply skips SRS eviction for this cycle. The acquire loop will retry, and eventually the mutex will be available. This is a classic trade-off: instead of risking a panic, we accept that eviction might be delayed under contention.

Testing with a Tight Budget

With the fix deployed, the assistant ran a controlled test: 100 GiB total budget, 3 concurrent proofs, each requiring 10 partitions. The goal was to verify that the budget system actually constrained concurrency.

The results were revealing. The RSS trace showed the daemon climbing from 0 GiB to 103 GiB over 40 seconds, then oscillating between 98 GiB and 114 GiB. The budget was working — it was keeping memory usage within the 100 GiB limit. But the user noticed a problem: "Seems there is only one synthesis running at a time, previously the machine was handling 10 or 8."

The assistant investigated and found two issues:

The budget was too tight: SRS (44 GiB) + PCE (26 GiB) = 70 GiB baseline, leaving only ~30 GiB for working sets. At 14 GiB per partition, that's barely two partitions. With 30 partitions to schedule across three proofs, serialization was inevitable.
An SRS pre-acquisition race: When three proofs arrived simultaneously, each independently checked if SRS was loaded. Since none had loaded it yet, all three pre-acquired 44 GiB from the budget. This temporarily consumed 132 GiB of budget for SRS alone, starving the working set allocations. The race resolved once the first proof finished loading SRS and converted its reservation to permanent, but the damage to concurrency was done. The assistant noted that the SRS race needed a proper fix — a loading flag to prevent duplicate pre-acquisitions — but the immediate priority was testing with a realistic budget.

The OOM Kill

Switching to total_budget = "auto" (which auto-detected 750 GiB from the 755 GiB system RAM minus 5 GiB safety margin) transformed the behavior. All 30 partitions started within one second. The RSS climbed rapidly: 0 → 35 → 72 → 114 → 157 → 185 → 229 → 273 GiB in under two minutes.

Then the daemon died. The bench client reported: "Error: Prove RPC failed — transport error — connection error — stream closed because of a broken pipe." The daemon had been OOM-killed.

The RSS trace showed a peak of 498 GiB. Adding Curio's ~87 GiB and the kernel's overhead, the system had exhausted its 755 GiB of physical RAM. The safety margin of 5 GiB was woefully inadequate — it only accounted for the gap between auto-detected memory and the budget limit, not for the memory consumed by co-located processes like Curio.

The Critical Insight

The assistant's conclusion was measured and precise: "The budget system works correctly but requires a larger safety margin (e.g., 250 GiB) to account for co-located processes."

This is the kind of insight that only emerges from real-world deployment. Unit tests can validate correctness. Benchmarks can validate performance. But only running the actual system on a production machine with real co-located workloads reveals the configuration parameters that make or break the system.

The budget-based memory manager was architecturally sound. It correctly tracked memory usage, gated concurrency, and prevented over-allocation. But the safety margin — a single configuration parameter — needed to account for the entire system's memory footprint, not just the proving daemon's. On a machine where Curio, the OS, and other processes consume 250+ GiB, a 5 GiB safety margin is a rounding error.

Themes and Reflections

Several themes emerge from this chunk of work:

The hidden complexity of API migration: What appeared to be simple mechanical changes — updating a function signature, adding a parameter — revealed deep architectural assumptions (like the 'static lifetime on PCE data) that had to be unwound.

The gap between validation and deployment: Passing cargo check, unit tests, and even a real-world benchmark on the development machine did not guarantee the system would work on the target machine. The runtime panic (blocking_lock in async context) and the OOM kill were both deployment-specific issues that no amount of local testing could have caught.

The importance of safety margins: The budget system's correctness was never in doubt. It tracked every byte, gated every allocation, and enforced the limit precisely. But precision without adequate margin is dangerous. The system needed to know not just how much memory the proving daemon could use, but how much was actually available after accounting for everything else running on the machine.

The iterative nature of systems engineering: Each deployment cycle revealed new issues. Fix the compile errors → find the runtime panic. Fix the runtime panic → find the concurrency bottleneck. Fix the concurrency bottleneck → find the OOM. Each iteration brought the system closer to production readiness, but each also revealed new dimensions of the problem.

Conclusion

The budget-based memory manager for cuzk represents a significant architectural improvement over the static partition_workers semaphore. It provides fine-grained control over memory usage, on-demand loading of SRS and PCE, LRU eviction under pressure, and a unified budget that prevents any single consumer from starving the system.

But the journey from implementation to production deployment revealed that architectural correctness is only part of the story. The system also needs robust configuration defaults, proper handling of async/sync boundaries, and safety margins that reflect the realities of a shared production environment. The final lesson is one that every systems engineer learns eventually: the difference between a system that works and a system that works in production is measured not in features but in the hard-won knowledge of what the safety margins should be.## References

[1] "The Implementation Status Report: How an AI Assistant Documented a Complex Memory Manager Refactoring" — Article on message 2240 documenting the full scope of the memory manager implementation.

[4] "The Loose Ends: A Pivotal Transition in the cuzk Memory Manager Implementation" — Article on message 2243 covering the final API migration points.

[19] "The Moment of Truth: Running cargo check After a Complex Refactoring" — Article on message 2258 analyzing the first compilation attempt.

[24] "The Static Lifetime That Outlived Its Purpose" — Article on message 2263 examining the 'static lifetime issue in the PCE cache.

[25] "Unwinding a Static Lifetime: How a Rust Compile Error Revealed the Architecture Shift from OnceLock to PceCache" — Article on message 2264 tracing the root cause of the lifetime error.

[28] "The Moment of Compilation: How a Single cargo check Validated an Entire Memory Manager Architecture" — Article on message 2267 covering the successful build.

[29] "The Quiet Validation: When \"Build Passes Cleanly\" Marks a Turning Point" — Article on message 2268 analyzing the clean build signal.

[51] "The Moment of Truth: Validating a Memory Manager at Production Scale" — Article on message 2290 covering the pce-bench run.

[52] "The Moment of Truth: Validating a Memory Manager with Production-Scale Data" — Article on message 2291 analyzing the benchmark results.

[54] "The Bridge Between Implementation and Production: Deploying the cuzk Memory Manager" — Article on message 2294 covering the deployment workflow.

[55] "The Pivot Point: How a Simple Todo List Captures the Transition from Development to Deployment" — Article on message 2295 analyzing the deployment todo list.

[56] "The Pivot Point: A Todo List Update That Marks the Transition from Implementation to Deployment" — Article on message 2297 covering the todo list update.

[57] "The Commit That Sealed a Memory Manager: A Pivot Point in the cuzk GPU Proving Engine" — Article on message 2296 analyzing the commit.

[58] "The Pivot Point: A Todo List Update That Marks the Transition from Implementation to Deployment" — Article on message 2297.

[59] "The Reconnaissance SSH: Probing the Deployment Target for a Budget-Based Memory Manager" — Article on message 2298 covering the initial SSH reconnaissance.

[60] "The Reconnaissance Before Deployment: Understanding the Target Environment" — Article on message 2299.

[61] "The Reconnaissance SSH: How a Single Command Uncovered the Deployment Landscape for cuzk's Memory Manager" — Article on message 2300.

[62] "The Config That Bridges Two Eras: Reading the Remote Configuration Before Deploying cuzk's Memory Manager" — Article on message 2301 analyzing the old config.

[63] "Reconnaissance Before Deployment: Auditing Remote Resources for a Memory Manager Rollout" — Article on message 2302.