The Production Baptism of a GPU Memory Manager: From Clean Compilation to OOM in Three Acts

The arc of a complex systems engineering project rarely follows a clean line from specification to production deployment. More often, it traces a jagged path through compile errors, unit tests, real-world validation, deployment surprises, and production debugging. This is the story of one such arc: the final integration, validation, and deployment of a budget-based memory manager for the cuzk GPU proving daemon — a journey that began with fixing two compile errors and ended with an OOM-killed daemon and a crucial insight about safety margins on shared machines.

In the span of a few dozen messages covering barely an hour of wall-clock time, a brand-new budget-based memory manager underwent its production baptism. It survived a runtime panic, proved its admission control logic worked correctly, demonstrated massive concurrency at scale, and then was killed by the Linux OOM killer. Each failure revealed a different layer of the gap between a theoretically sound design and the messy reality of a shared machine running multiple co-located processes. This article traces that arc — from the first deployment to the final diagnostic conclusion — examining the reasoning, assumptions, and discoveries that emerged when a carefully engineered memory management system met real hardware running real workloads.

The Stage: A Memory Manager Born from Fragility

The cuzk GPU proving engine processes Filecoin's zero-knowledge proofs, each of which demands enormous memory. A single 32 GiB PoRep proof requires synthesizing 10 partitions of 130 million constraints each, consuming approximately 14 GiB of working memory per partition on top of a 44 GiB Structured Reference String (SRS) and a 26 GiB Pre-Compiled Constraint Evaluator (PCE) cache. The old system managed this appetite with a static partition_workers limit — a hard cap on concurrent partitions that was simple but fragile. It could neither adapt to varying proof sizes nor protect against OOM when workloads spiked.

The assistant had spent several segments designing and implementing a replacement: a unified memory budget system that tracked every major memory consumer under a single byte-level limit. The MemoryBudget struct used an acquire/release protocol where each component — SRS loading, PCE extraction, partition synthesis — had to reserve space before allocating. An evictor callback could free memory by unloading idle SRS or PCE entries when the budget ran low. The design was clean, testable, and conceptually superior to the old static limit.

By the start of segment 17, the implementation was nearly complete. All 15 unit tests passed. A real-world pce-bench run on a 754 GiB development machine successfully extracted a 25.7 GiB PCE and validated correctness across all 10 circuits. The changes were committed. The stage was set for deployment to the target production machine: a remote host with 755 GiB of RAM, an RTX 5090 GPU, 64 CPU cores, and a running Curio instance.

But "nearly complete" is not the same as "complete." Three loose ends remained — the final API migration points that would make the whole thing compile.

The Final Integration Fixes

The first chunk of segment 17 opens with the assistant in the closing stages of a massive refactoring: replacing the static partition_workers semaphore with a unified MemoryBudget that tracks all major memory consumers under a single byte-level budget auto-detected from system RAM. The specification had been written, the core implementation completed across seven files, and the major architectural changes were in place. What remained were the loose ends.

Three specific tasks needed attention. First, the third extract_and_cache_pce_from_c1 call in cuzk-bench/src/main.rs needed to accept the new PceCache parameter. Second, the SrsManager::new call in the same function needed conversion from the old (PathBuf, u64) signature to the new (PathBuf, Arc<MemoryBudget>) signature. Third, the entire codebase needed to be checked for any remaining references to the removed get_pce API.

These were straightforward mechanical changes, but they represented the final bridge between the old architecture and the new. Every call site that still used the old API was a potential compile error waiting to surface. The assistant methodically updated each one, then ran cargo check to see what remained.

Two Compile Errors That Nearly Derailed Everything

The first cargo check revealed two compile errors — small in scope but significant in what they revealed about the architecture:

synth_job mutability: In engine.rs:2416, the synth_job binding needed to be mut to allow .take() on the reservation field. This was a Rust ownership issue: the MemoryReservation inside SynthesizedJob was being moved out via .take(), which requires mutable access. The fix was a single word — adding mut — but it highlighted how the new budget system threaded ownership through the entire synthesis pipeline.
The 'static lifetime ghost: In pipeline.rs:1143, the pce_arc variable had a lifetime issue. The function synthesize_with_pce required a 'static lifetime on its pce parameter, but the local Arc<PreCompiledCircuit> returned from PceCache.get() didn't satisfy this constraint. This was a leftover from the old architecture, where PCE data was stored in static OnceLock variables and could be borrowed for the program's entire lifetime. The new PceCache stores Arc-wrapped data, but the 'static requirement was still baked into the function signature. The second error was particularly instructive. It revealed how architectural assumptions can become fossilized in type signatures. The 'static bound made perfect sense when PCE data lived in a global OnceLock — it was literally static. But when the architecture shifted to a runtime-managed PceCache, the data's lifetime became dynamic. The bound was no longer valid, and removing it was the correct fix. This is a classic pattern in large refactorings: the code that should have been updated during the main implementation phase wasn't, because it was hidden behind a function signature that "worked" until the new types were plugged in.

The Moment of Compilation

With those two fixes applied, cargo check returned clean — zero errors. The assistant then ran the full test suite: all 8 memory module tests passed, all 7 config tests passed. The implementation was not just syntactically correct but semantically validated by the unit tests.

But the real validation came next. The assistant ran pce-bench — a real-world benchmark that loads a 32 GiB PoRep C1 output, extracts the PCE, and validates correctness across all 10 circuits (130 million constraints each). This was the acid test: would the new PceCache correctly store and retrieve the Arc<PreCompiledCircuit>? Would the MemoryBudget track usage correctly? Would the PCE data be correct end-to-end?

The results were clean. Baseline synthesis completed in 50.9 seconds. PCE extraction took 68.2 seconds — a one-time cost, with the 25.7 GiB PCE data cached in the new PceCache. PCE synthesis completed in 52.1 seconds, using PceCache.get() to retrieve the cached circuit. All 10 circuits matched between baseline and PCE paths. The memory manager worked.

With validation complete, the assistant committed the changes. The commit message tells the story concisely:

Replace the static partition_workers semaphore with a unified MemoryBudget that tracks all major memory consumers (SRS pinned, PCE heap, synthesis working set) under a single byte-level budget auto-detected from system RAM.

The diff statistics are staggering: 9 files changed, 2,629 insertions, 592 deletions. Two new files created (memory.rs and the specification document). This was not a small refactoring — it was a fundamental re-architecture of how the GPU proving daemon manages its most critical resource.

The Deployment Directive

The user's next instruction shifted the focus from development to deployment: "Commit, test on ssh -p 40612 root@141.0.85.211 which is running curio and cuzk, curio is cordoned; upload new cuzk (built in docker) and test if memory constraints work correctly."

This was the moment the implementation would meet reality. The remote machine had 755 GiB RAM, an RTX 5090 (32 GB VRAM), and 64 cores. Curio was running (cordoned, meaning it wasn't accepting new work). The existing cuzk binary was at /usr/local/bin/cuzk, and the config at /tmp/cuzk-run-config.toml still used the deprecated preload and partition_workers fields.

The assistant first performed reconnaissance — checking free memory, running processes, GPU configuration, and the existing config. The machine had 755 GiB total, with 227 GiB used and 528 GiB available. The RTX 5090 was detected with 32 GiB VRAM. Curio was running but no cuzk process was active. The config needed updating for the new memory manager.

Act I: The Runtime Panic at engine.rs:913

The first deployment was a study in the hazards of async Rust. The assistant built the binary via Docker, extracted it from a scratch-based image, uploaded it via SCP, and hot-swapped it into the running daemon's path. The daemon started cleanly — 12 MiB RSS, no preload, on-demand loading active. The memory budget was initialized at 100 GiB, deliberately tight to stress-test the admission control logic. The assistant launched three concurrent proofs and waited.

What came back was not a clean validation but a runtime panic:

thread 'tokio-runtime-worker' panicked at /build/extern/cuzk/cuzk-core/src/engine.rs:913:45:
Cannot block the current thread from within a runtime.

This is a classic tokio pitfall. The evictor callback — a synchronous closure passed to MemoryBudget::set_evictor() — used blocking_lock() on a tokio::sync::Mutex guarding the SrsManager. The evictor was called from within the async acquire() loop running on a tokio worker thread. blocking_lock() blocks the current thread, which is forbidden on worker threads because it prevents them from driving other async tasks. The panic was immediate and unambiguous.

The fix, applied in subsequent messages, was to replace blocking_lock() with try_lock(). If the SRS mutex is held by another task, try_lock() returns None and the evictor skips SRS eviction for that iteration. The acquire loop retries, and the mutex eventually becomes available. This trades a guaranteed panic for a graceful degradation — the system might take an extra iteration to free memory, but it won't crash.

This fix was the first lesson of production deployment: code that compiles cleanly and passes all unit tests can still panic at runtime when exercised under conditions that were never tested. The evictor had never been called during development because the budget was never exhausted in single-proof tests. Only the deliberate stress test with a constrained budget and concurrent proofs triggered the evictor path, exposing the bug.

Act II: The Concurrency Cliff

With the evictor fix deployed, the assistant re-ran the benchmark. The daemon no longer panicked, but the user's observation revealed a different kind of failure: "Seems there is only one synthesis running at a time, previously the machine was handling 10 or 8."

The assistant's investigation uncovered a multi-layered problem. The 100 GiB budget was simply too tight. The SRS (44 GiB) plus PCE (26 GiB) baseline consumed 70 GiB, leaving only 30 GiB for working sets — enough for roughly two partitions at 14 GiB each. But the actual behavior was even worse: the daemon logs showed budget_used_gib=88 right after SRS loading, suggesting that multiple proofs were each reserving 44 GiB for SRS simultaneously.

A grep of the source code confirmed the race. Each proof independently checked is_loaded() and, if the SRS wasn't loaded yet, acquired a full 44 GiB reservation before loading. When three proofs arrived simultaneously, all three saw is_loaded() == false and each reserved 44 GiB, temporarily consuming 132 GiB of budget for a resource that only needed 44 GiB. During the ~6-second SRS loading window, these overlapping reservations starved the budget, allowing only one partition to slip through at a time.

The assistant traced the SYNTH_START/SYNTH_END timeline from the daemon logs: partition 0 started at T=22216, ended at T=64190 (42 seconds). Partition 7 started at T=68531 — only after partition 0 finished. Partition 1 started at T=114932 — only after partition 7 finished. Each partition ran sequentially because the budget never had room for more than one at a time.

The assistant faced a triage decision: fix the SRS race condition (a code change requiring rebuild and redeployment) or increase the budget to match the machine's actual capacity (a configuration change). The assistant chose the latter, reasoning that the race was only a problem under tight budgets and that validating the overall system at scale was the higher priority.

Act III: Thirty Partitions in One Second

The assistant reconfigured the daemon with total_budget = "auto" and safety_margin = "5GiB". The auto-detection mechanism read the system's 755 GiB of RAM and, after subtracting the 5 GiB margin, set the budget to 750 GiB. The daemon logged max_partitions_in_budget=53 — a dramatic increase from the 2-partition limit under the 100 GiB budget.

The benchmark was re-launched: three proofs, concurrency 3, 10 partitions each. Thirty seconds later, the assistant checked the logs and found 30 SYNTH_START entries, all with timestamps within the same second. Every partition across all three proofs had started simultaneously. The budget system was working exactly as designed — when given adequate headroom, it admitted work freely without artificial serialization.

The assistant's message captured the moment of triumph: "Now we're talking. All 30 partitions (3 proofs × 10 partitions) started within ~1 second — massive concurrency." The budget numbers looked healthy: total_budget_gib=750, budget_available_gib dropping from 706 to 622 as partitions were dispatched, RSS climbing rapidly from 0 to 273 GiB within 30 seconds.

But the triumph was short-lived. The assistant waited 120 seconds for completion and instead received a transport error: the connection was broken, the stream was closed. The daemon had been killed.

Act IV: The OOM Kill and the Safety Margin That Wasn't

The assistant's forensic analysis reconstructed the failure. The RSS trace showed memory climbing to approximately 498 GiB before the daemon disappeared. The machine had 755 GiB total, but earlier free -h output had shown only ~528 GiB available — the gap was consumed by kernel overhead, filesystem cache, and the Curio daemon, which alone used ~87 GiB of shared memory.

The arithmetic was damning: 30 partitions × 14 GiB each = 420 GiB working set, plus 44 GiB SRS, plus 26 GiB PCE = 490 GiB for cuzk. Adding Curio's ~87 GiB brought the total to ~577 GiB. With kernel overhead and other processes, the system exceeded its 755 GiB physical memory, and the OOM killer selected the cuzk daemon as the victim.

The root cause was the safety_margin = "5GiB" configuration. This parameter was intended to reserve memory for other processes, but it was orders of magnitude too small. The assistant had set it based on an assumption that cuzk was the primary memory consumer on a dedicated machine — an assumption that was false. The machine ran Curio, system services, and the kernel itself, all of which consumed significant memory outside the budget system's visibility.

The budget system had worked correctly in the narrow sense: it limited cuzk's own allocations to the configured 750 GiB. But that number was based on total system RAM minus a trivial safety margin, not on actual available memory after accounting for co-located processes. The budget system had no mechanism to discover or account for memory consumed by other processes on the same machine.

The Critical Insight

The assistant's conclusion was measured and precise: "The budget system works correctly but requires a larger safety margin (e.g., 250 GiB) to account for co-located processes."

This is the kind of insight that only emerges from real-world deployment. Unit tests can validate correctness. Benchmarks can validate performance. But only running the actual system on a production machine with real co-located workloads reveals the configuration parameters that make or break the system.

The budget-based memory manager was architecturally sound. It correctly tracked memory usage, gated concurrency, and prevented over-allocation. But the safety margin — a single configuration parameter — needed to account for the entire system's memory footprint, not just the proving daemon's. On a machine where Curio, the OS, and other processes consume 250+ GiB, a 5 GiB safety margin is a rounding error.

Lessons Learned

This arc of deployment — runtime panic, concurrency collapse, triumphant scale, OOM kill — is a microcosm of the challenges in bringing a resource management system from design to production. Several lessons emerge.

First, the evictor callback was a synchronous interface in an async world. The MemoryBudget::set_evictor() method accepted a synchronous Fn(u64) -> u64 closure, but invoked it from within an async acquire() loop. This forced the evictor implementation to either use blocking primitives (which panic on tokio worker threads) or restructure its internals. A more robust design would have used an async callback or a channel-based approach where the evictor sends eviction requests to a background task.

Second, the SRS pre-acquisition race was a classic check-then-act concurrency bug. Each proof independently checked is_loaded() and, if false, acquired budget for the full SRS size. Without an atomic "check and mark as loading" operation, multiple proofs could pass the check before any completed the load. The fix — a loading flag or a compare-and-swap pattern — was deferred but not forgotten.

Third, the safety margin is the most critical configuration parameter in a shared machine. The assistant's instinct to set safety_margin = "5GiB" was reasonable for a dedicated machine, but the machine was not dedicated. The budget system's "auto" mode detected total system RAM, but total RAM is not the same as available RAM. The gap — consumed by Curio, kernel structures, filesystem cache, and other processes — must be measured and accounted for.

Fourth, the budget system's accounting was incomplete. The budget tracked explicit reservations for SRS, PCE, and partition working sets, but it did not track CUDA pinned memory, transient allocations during PCE extraction, shared library overhead, or the Rust runtime itself. The discrepancy between budget-tracked memory and actual RSS was a warning sign that the assistant initially overlooked.

Fifth, the hidden complexity of API migration. What appeared to be simple mechanical changes — updating a function signature, adding a parameter — revealed deep architectural assumptions (like the 'static lifetime on PCE data) that had to be unwound. The 'static lifetime was a fossil of the old OnceLock architecture, and removing it required understanding not just what the code did, but why it was written that way.

Sixth, the gap between validation and deployment. Passing cargo check, unit tests, and even a real-world benchmark on the development machine did not guarantee the system would work on the target machine. The runtime panic (blocking_lock in async context) and the OOM kill were both deployment-specific issues that no amount of local testing could have caught.

Conclusion: The Iterative Nature of Systems Engineering

The segment ends with the assistant having identified the correct diagnosis: the budget system works, but the safety margin must be increased to approximately 250 GiB to account for co-located processes. The fix is a configuration change, not a code change. The SRS race condition remains as a known issue to tackle later.

This arc demonstrates that no design survives first contact with production unchanged. The memory manager was architecturally sound — its core acquire/release protocol, its eviction mechanism, its budget gating all functioned correctly. But each deployment revealed a new dimension of the problem: the async/sync boundary in the evictor, the race condition in SRS loading, the gap between total RAM and available RAM, the incomplete accounting of memory consumers.

The iterative cycle — deploy, observe, diagnose, fix, redeploy — is not a sign of failure. It is the fundamental mechanism by which systems become robust. Each failure eliminated a blind spot. Each fix added a layer of resilience. By the end of the segment, the assistant had a complete understanding of what the memory manager needed to operate safely in its production environment: a larger safety margin, a fix for the SRS race, and possibly a mechanism to discover co-located process memory usage.

The production baptism was brutal, but it was effective. The memory manager that emerges from this session will be more robust than the one that entered it — not because the design was wrong, but because the design was tested against reality.

References

[1] "From Compilation to OOM: The Complete Journey of Deploying a Budget-Based Memory Manager" — Chunk article covering the final integration fixes, compilation, validation, and initial deployment reconnaissance.

[2] "From Runtime Panic to OOM: The Production Baptism of a GPU Memory Manager" — Chunk article covering the runtime panic fix, concurrency bottlenecks, and OOM diagnostic analysis.