The Art of Budget Arithmetic: How One Assistant Calculated 400 GiB of Safety

In the middle of a sprawling coding session spanning dozens of messages, one message stands out not for its code changes, not for its architectural decisions, but for a moment of quiet arithmetic. Message 2380 is a single SSH command that writes a configuration file to a remote machine, preceded by a paragraph of reasoning about memory budgets. On its surface, it is mundane: an engineer adjusting a total_budget parameter from "auto" to "400GiB". But beneath that surface lies a rich tapestry of systems thinking, failure analysis, and the delicate art of provisioning resources in a shared environment.

This message represents the culmination of a long debugging journey—a journey that began with a promising new memory manager, passed through a panic caused by a blocking_lock() call in an async context, and ended with a process killed by the Linux OOM killer at 498 GiB RSS. The message is the moment when the assistant, having diagnosed the root cause of the OOM, applies the fix: not a code change, but a configuration change informed by careful reasoning about the machine's actual available memory.

The Context: A Memory Manager Under Test

To understand why this message was written, we must first understand what came before it. The assistant had been implementing a unified budget-based memory manager for the cuzk daemon—a CUDA-based ZK proving engine used in the Filecoin network. The old system used a static partition_workers semaphore that limited the number of concurrent proof partitions based on a hardcoded count, not on actual memory usage. The new system tracked every major memory consumer—SRS parameters (~44 GiB), PCE caches (~26 GiB), and per-partition working sets (~13.6 GiB each)—under a single byte-level budget, auto-detected from system RAM, with on-demand loading and LRU eviction under pressure.

The implementation had been completed and committed. A critical bug—the evictor callback calling blocking_lock() on a tokio mutex from within an async context, causing a runtime panic—had been found during deployment and fixed with a try_lock() approach. The binary was deployed to a remote test machine with 755 GiB of RAM, a single RTX 5090 GPU, and 64 cores.

The first real test with the auto-detected budget of 750 GiB (755 GiB detected minus 5 GiB safety margin) was dramatic: all 30 partitions across 3 proofs started within one second, RSS climbed rapidly to 498 GiB, and then the process was killed by the OOM killer. The transport error—"stream closed because of a broken pipe"—was the unmistakable signature of a process terminated by the kernel.

The Diagnosis: Why Auto-Detection Failed

The assistant's reasoning in message 2380 begins with a clear-eyed assessment of the machine's actual state. The auto-detection logic had done its job: it detected 755 GiB of total RAM, subtracted a 5 GiB safety margin, and set the budget to 750 GiB. But this logic was fundamentally flawed because it only accounted for the cuzk process itself. The machine was running Curio, another large process in the Filecoin ecosystem, which was using approximately 226 GiB of memory (combining RSS and shared memory).

This is a classic failure mode in resource management: the assumption that "total system memory" equals "memory available for my process." In reality, the operating system's available memory—the amount that can be allocated without triggering OOM—is total RAM minus what other processes are using. The free -h command had shown 528 GiB available, not 755 GiB. The 5 GiB safety margin was laughably insufficient when a co-resident process was consuming over 200 GiB.

The assistant's reasoning in the message shows this understanding clearly: "The math: 755 GiB total - 226 GiB (Curio) = 529 GiB available for cuzk." This is the critical insight that the auto-detection logic lacked. The budget system could not know about other processes on the machine; it could only see its own allocations. The solution was not to make the budget system more aware of other processes (a complex and fragile undertaking), but to set an explicit budget that left room for them.

The Arithmetic of Safety

The message then walks through a careful calculation. The assistant considers three different budget levels:

500 GiB: This would leave 755 - 226 - 500 = 29 GiB headroom. With SRS and PCE consuming ~70 GiB baseline, that leaves 430 GiB for working sets, which at 13.6 GiB per partition means ~31 partitions—enough for all 30 partitions to run concurrently. But the assistant recognizes this as too aggressive: "memory accounting isn't perfect." The previous run had shown that 30 partitions consumed 498 GiB RSS, not the 70 + 30×13.6 = 478 GiB that the simple model predicted. The 20 GiB discrepancy came from allocator overhead, fragmentation, stack memory, and other unaccounted costs. With only 29 GiB of headroom against a 226 GiB Curio process, any accounting error could trigger another OOM.

400 GiB: This gives 755 - 226 - 400 = 129 GiB headroom. With 70 GiB baseline, that's 330 GiB for working sets, allowing ~24 concurrent partitions. The assistant estimates RSS would land around 70 + 24×14.3 ≈ 413 GiB, well within the 529 GiB available. The 129 GiB headroom provides a generous cushion for allocator overhead, kernel caches, and other unaccounted memory.

The assistant's choice of 400 GiB reveals a conservative engineering philosophy. Rather than pushing to the limit (500 GiB, which would barely fit all 30 partitions), the assistant deliberately throttles to ~24 partitions, accepting lower throughput in exchange for reliability. This is a judgment call that prioritizes stability over peak performance—a wise choice given that the previous run had already demonstrated the consequences of overcommitment.

The Assumptions Embedded in the Calculation

Every engineering decision rests on assumptions, and this message is rich with them. Let me examine the key assumptions the assistant makes:

Assumption 1: Curio's memory usage is stable at ~226 GiB. The assistant uses the free -h output showing 226 GiB "used" as a fixed value. In reality, Curio's memory usage could fluctuate. If Curio's usage spikes during proving—perhaps because it's also processing proofs or serving API requests—the available memory could shrink, potentially causing OOM even with the 129 GiB headroom.

Assumption 2: The per-partition cost is ~14.3 GiB. This is derived from the previous OOM run: (498 - 70) / 30 ≈ 14.3 GiB. But this is an empirical average that includes allocator overhead and fragmentation effects that may not scale linearly. With 24 partitions instead of 30, the overhead per partition might be different (possibly lower, due to reduced contention for allocator locks and page cache).

Assumption 3: The baseline 70 GiB (SRS + PCE) is fixed. The assistant assumes SRS and PCE caches consume a constant 70 GiB regardless of how many partitions are being processed. This is reasonable—these are loaded once and shared across all partitions—but the PCE cache could grow if new circuit types are encountered, and the SRS manager's eviction policy could free some of that memory under pressure.

Assumption 4: The safety margin of 0 GiB in the config is sufficient. The assistant sets safety_margin = "0GiB" because the explicit budget of 400 GiB already incorporates a de facto safety margin through the conservative choice. But the budget system still has a safety_margin field that could provide additional protection against accounting errors. Setting it to 0 means the budget system will allow allocations up to exactly 400 GiB, with no further buffer.

Assumption 5: The eviction policy will work correctly under the new budget. The assistant doesn't explicitly discuss eviction in this message, but the config includes eviction_min_idle = "5m", meaning SRS and PCE entries that haven't been used for 5 minutes are candidates for eviction. The assistant assumes that under a 400 GiB budget, the eviction policy will not trigger aggressively (since 24 partitions should fit comfortably), and that if it does trigger, the try_lock() fix will prevent panics.

The Input Knowledge Required

To fully understand this message, a reader needs knowledge spanning several domains:

System memory accounting: Understanding the difference between total RAM, used RAM, available RAM, RSS, and shared memory. The assistant references free -h output and distinguishes between Curio's RSS (160 MiB) and its shared memory usage (87 GiB), recognizing that both contribute to memory pressure even though they appear in different columns.

Linux OOM behavior: Knowing that the kernel's OOM killer terminates processes when the system runs out of memory, and that the symptom is a broken pipe or transport error rather than a clean process exit. The assistant had previously confirmed OOM via dmesg (though permission was denied) and inferred it from the RSS trajectory and error pattern.

GPU proving memory model: Understanding the memory lifecycle of a ZK proof: SRS parameters (~44 GiB, CUDA pinned memory), PCE caches (~26 GiB, heap), per-partition working sets (~13.6 GiB including a/b/c vectors, aux data, and density information), and the two-phase GPU release pattern (synchronous drop of a/b/c vectors at prove_start, then remaining ~1.1 GiB at prove_finish).

The cuzk architecture: Knowing that the daemon processes proofs in partitions (10 partitions per 32 GiB PoRep proof), that partitions can be processed concurrently, and that the synthesis phase (CPU work) and GPU proving phase have different memory profiles.

The Curio context: Understanding that Curio is a separate daemon in the Filecoin ecosystem that runs alongside cuzk, consuming significant shared memory for its own operations. The assistant had previously checked ps aux and free -h to establish Curio's footprint.

The Output Knowledge Created

This message produces several forms of knowledge:

A validated configuration: The config file written to the remote machine is the primary output. It sets total_budget = "400GiB", safety_margin = "0GiB", and eviction_min_idle = "5m". This configuration becomes the basis for the next test run.

A reasoning framework: The message establishes a methodology for setting memory budgets in shared environments. The key insight—that total system RAM minus co-resident process usage gives the true available budget—is a reusable principle. The assistant's arithmetic (755 - 226 = 529 available, 400 budget leaves 129 headroom) provides a template for future provisioning decisions.

An empirical correction: The previous OOM run provided data that refined the per-partition cost estimate from the theoretical 13.6 GiB to the empirical 14.3 GiB. This correction is implicitly applied in the message's reasoning ("(498 - 70) / 30 ≈ 14.3 GiB") and informs the budget calculation.

A decision record: The message documents why 400 GiB was chosen over 500 GiB. This is valuable for future debugging: if the next run is stable, the record explains why; if it still OOMs, the record helps identify which assumption was wrong.

The Thinking Process in Action

The message's reasoning section is a window into the assistant's cognitive process. Let me trace the steps:

Step 1: Establish the constraints. The assistant lists the key numbers: 755 GiB total, 226 GiB for Curio, 70 GiB baseline for cuzk, 13.6 GiB per partition. This is the factual foundation.

Step 2: Calculate the available budget. 755 - 226 = 529 GiB. This is the hard upper bound—the amount of memory cuzk can consume before the system runs into trouble.

Step 3: Test the 500 GiB candidate. 500 GiB leaves 29 GiB headroom. The assistant calculates that 500 - 70 = 430 GiB for working sets, allowing ~31 partitions. But then hesitates: "that's still all 30 partitions which may be too aggressive since memory accounting isn't perfect." This is the key moment of judgment. The assistant recognizes that the theoretical model doesn't perfectly predict reality (as the 20 GiB discrepancy in the previous run demonstrated) and decides that 29 GiB of headroom is insufficient.

Step 4: Test the 400 GiB candidate. 400 GiB leaves 129 GiB headroom. The assistant calculates ~24 concurrent partitions and estimates RSS around 413 GiB. The 129 GiB headroom provides a comfortable margin for accounting errors and fluctuations.

Step 5: Validate against empirical data. The assistant cross-checks the per-partition cost using data from the OOM run: (498 - 70) / 30 ≈ 14.3 GiB. This empirical value is close to the theoretical 13.6 GiB, confirming that the model is reasonably accurate.

Step 6: Make the decision. The assistant chooses 400 GiB, then immediately acts by writing the config file to the remote machine. The decision is not deferred or second-guessed; the reasoning leads directly to action.

What's notable is what the assistant doesn't do. It doesn't consider dynamic budget adjustment based on Curio's current usage. It doesn't implement a feedback loop that measures actual RSS and adjusts the budget in real-time. It doesn't add Curio-awareness to the auto-detection logic. Instead, it applies a simple, robust solution: set an explicit budget that leaves ample room for the known co-resident process. This is pragmatic engineering—solve the immediate problem with the simplest adequate solution.

The Broader Significance

This message is a microcosm of the entire coding session. The session began with ambitious architectural work: designing and implementing a unified memory manager that replaced a static semaphore with a sophisticated byte-level budget system. But the real-world deployment revealed that no amount of architectural sophistication can substitute for understanding the actual operating environment. The auto-detection logic was elegant but wrong because it didn't know about Curio. The fix was not more sophistication but more knowledge—knowledge about the machine's actual workload.

The message also illustrates the importance of empirical validation in systems engineering. The assistant didn't just trust the theoretical 13.6 GiB per-partition estimate; it used the OOM run data to derive an empirical 14.3 GiB estimate. This cross-check caught a 5% error in the model, which, when multiplied across 30 partitions, accounted for the 20 GiB discrepancy between predicted and actual RSS.

Finally, the message demonstrates the value of conservative engineering. The assistant had the option to set the budget to 500 GiB and squeeze out maximum throughput. Instead, it chose 400 GiB, accepting ~20% lower concurrency in exchange for a 4x larger safety margin (129 GiB vs 29 GiB). This is the kind of judgment that separates reliable systems from fragile ones.

Conclusion

Message 2380 is a study in practical systems engineering. It is not flashy—it does not introduce new algorithms, refactor large codebases, or implement novel architectures. It is a quiet moment of arithmetic, a configuration change, a single SSH command. But it encapsulates everything that makes the broader session work: the careful diagnosis of a failure, the understanding of the system's actual constraints, the willingness to adjust based on empirical data, and the judgment to prioritize reliability over peak performance.

The 400 GiB budget that the assistant settled on would prove to be the right choice. In subsequent messages, the assistant would deploy this configuration, run the benchmark, and watch all 30 partitions complete successfully with peak RSS at 488 GiB—safely under the 529 GiB available. The proofs would pass verification. The memory manager would work.

Sometimes the most important engineering decision is not what to build, but what number to write in a config file.