The Push That Fixed the OOM: Deploying Hardware-Aware Benchmark Concurrency

docker tag curio-cuzk:latest theuser/curio-cuzk:latest && docker push theuser/curio-cuzk:latest 2>&1 | tail -10
3c02fbcccafe: Layer already exists
b7ec8fd7c485: Layer already exists
dab2da33fe40: Layer already exists
b5c4de646bba: Layer already exists
b62eaeb8df36: Pushed
18241330a2d1: Layer already exists
073ec47a8c22: Layer already exists
415560502554: Pushed
41c7798f0421: Pushed
latest: digest: sha256:4727f1fbd18f89bfbf53964113595076c036a81fb4ac45798d79453de627e421 size: 4929

At first glance, message [msg 1130] looks like a routine Docker push — ten lines of layer status, a digest, and a size. But in the context of this coding session, it represents the culmination of a high-stakes debugging marathon. This single docker push deployed a fix for a catastrophic Out-of-Memory (OOM) bug that had been silently killing GPU proving instances, wasting money, and threatening the viability of the entire distributed proving pipeline. The message is the moment a hard-won insight about memory budgeting was shipped to production.

The OOM Crisis

The story begins with a US-based GPU instance running on Vast.ai: a machine with 376GB of RAM and two RTX 3090 GPUs. The instance was running a PoRep (Proof of Replication) benchmark — the standard performance test used to qualify proving workers. The benchmark script (benchmark.sh) would start the cuzk daemon, submit a warmup proof to generate the Pre-Compiled Constraint Evaluator (PCE) cache, then run a batch of 12 proofs at a concurrency of 5 to measure throughput.

The first OOM fix had already been deployed in a previous round: the warmup phase now used partition_workers=2 to avoid the memory spike of simultaneous partition synthesis during PCE extraction. That fix worked — the warmup completed successfully, the PCE file was generated, and the daemon restarted with the full partition_workers=10. But then the actual benchmark began, and the process was killed again.

As the assistant discovered in [msg 1115], the problem was deeper than just the warmup: "5 concurrent proofs x 10 partition workers = too much memory, even with 376GB RAM." Each proof running 10 partition workers simultaneously consumed roughly 30-40GB of RAM. With concurrency set to 5, that meant 150-200GB just for synthesis, plus the SRS file (44GB), the PCE cache (26GB), and OS overhead. The machine had 376GB total, but the combined demand exceeded what the kernel could supply, and the OOM killer struck.

The Reasoning Behind the Fix

The assistant's thinking in [msg 1122] reveals the diagnostic process: "The issue isn't just the warmup/PCE extraction — it's also the steady-state operation with high concurrency." The hardcoded BENCH_CONCURRENCY=5 in entrypoint.sh (line 19) was the root cause. It assumed all machines could handle five simultaneous proofs, which was catastrophically wrong for lower-RAM instances.

The fix, developed across messages [msg 1123] through [msg 1128], involved a fundamental redesign of how the entrypoint configures benchmark parameters. Instead of hardcoded values, the script would now:

Detect GPU count by parsing nvidia-smi output
Calculate available RAM by reading /proc/meminfo and subtracting a 100GB safety reserve
Estimate per-proof memory demand as partition_workers * 3GB (later adjusted from an initial * 4GB)
Compute maximum safe concurrency as the floor of available RAM divided by per-proof demand
Cap concurrency by GPU capacity using a formula of num_gpus * 3 (allowing a small queue per GPU)
Take the minimum of the RAM-limited and GPU-limited values This was a classic engineering trade-off: the assistant could have simply lowered the hardcoded concurrency to 2 for all machines, but that would leave high-RAM machines underutilized. The dynamic formula maximized throughput on every hardware configuration while guaranteeing safety.

Assumptions Made

The fix rested on several assumptions, some explicit and some implicit:

/proc/meminfo reflects actual available memory. The assistant noted in [msg 1127] that the Vast.ai API reports cpu_ram as a per-GPU fraction, which can differ wildly from the container-visible memory. The US instance showed 75GB in the API but 376GB in /proc/meminfo. The fix correctly reads the container's view, but this means the formula's accuracy depends on the container's memory limit being set correctly.
Each partition worker consumes approximately 3GB of RAM. This was an empirical estimate, adjusted from an initial 4GB after the assistant judged the formula too conservative. The actual memory footprint depends on proof parameters, GPU architecture, and the specific proving pipeline, so this is a heuristic, not a precise measurement.
GPU count detection via nvidia-smi is reliable. The script uses nvidia-smi --query-gpu=count --format=csv,noheader to count GPUs. This works in most Docker containers but could fail if the NVIDIA runtime is misconfigured.
The 100GB overhead reserve is sufficient. This covers the SRS file (44GB), PCE cache (26GB), OS/kernel overhead, and the cuzk daemon's base memory. For machines with less than ~150GB total RAM, this reserve would consume most of the available memory, forcing concurrency to 1.

Mistakes and Corrections

The most notable mistake was the initial per-proof memory estimate. In [msg 1124], the assistant first implemented the formula using partition_workers * 4 as the per-proof memory demand. But after reviewing the math for a 125GB machine — which would yield concurrency=1, deemed "way too conservative" — the assistant reconsidered. The reasoning in [msg 1127] shows the correction: "For safety, let me adjust the formula to be a bit less conservative — partition_workers * 3 instead of * 4."

This adjustment reveals an important tension in the design: the formula had to be conservative enough to prevent OOM kills on every target machine, but aggressive enough to keep high-RAM instances productive. The assistant had no benchmark data to calibrate the per-worker memory cost precisely, so it chose a middle ground. The 3GB estimate was a judgment call, not a measurement.

Another subtle issue was the handling of the 125GB edge case. The assistant realized that the Vast.ai API's cpu_ram field reports per-GPU-fraction memory, not total system RAM, so the 125GB case might never occur in practice. But the formula still had to handle it gracefully, which it did by capping concurrency to a minimum of 1.

Input Knowledge Required

To understand this message, one must grasp several layers of domain knowledge:

The cuzk proving pipeline: PoRep proofs are split into partitions, each synthesized by a worker thread. More partition workers means faster proof generation but higher memory usage. The PCE is a pre-compiled constraint evaluator that caches circuit structure to accelerate future proofs.
Vast.ai instance model: Instances report cpu_ram as a per-GPU fraction (e.g., 76800 MB for a 2-GPU machine with 376GB total), which is misleading. The container's /proc/meminfo gives the true limit.
Docker image management: The docker tag and docker push commands deploy the locally-built image to a registry (theuser/curio-cuzk), making it available for vastai create to pull onto new instances.
The overall system architecture: The entrypoint script manages the full lifecycle — tunnel setup, parameter fetch, benchmark, and supervisor mode. The benchmark results determine whether an instance is kept or destroyed.

Output Knowledge Created

This message produced a pushed Docker image with digest sha256:4727f1fbd18f89bfbf53964113595076c036a81fb4ac45798d79453de627e421. This image contains the fixed entrypoint.sh that will dynamically configure benchmark concurrency on every new instance. The output is not just a software artifact — it's a deployed solution to a systemic problem that had been wasting money and preventing reliable proving operations.

The Thinking Process

The assistant's reasoning across the preceding messages shows a clear diagnostic arc. In [msg 1115], it identified the OOM kill during the batch benchmark. In [msg 1122], it traced the root cause to the hardcoded concurrency of 5. In [msg 1123], it read the entrypoint to understand the current configuration. In [msg 1124]-[msg 1127], it iterated on the formula, adjusting the per-worker memory estimate from 4GB to 3GB. In [msg 1128], it verified the math for three target configurations. In [msg 1129], it built the Docker image. And finally, in [msg 1130], it pushed the image to the registry.

The most interesting thinking appears in [msg 1127], where the assistant catches itself being too conservative: "Wait, the 125GB case gives concurrency=1 which is way too conservative — the RTX 3090 should be able to handle at least 2 concurrent proofs." This moment of self-correction — adjusting the formula based on hardware intuition rather than blind calculation — is the hallmark of an engineer who understands both the system and the hardware it runs on.

Conclusion

Message [msg 1130] is a Docker push. But it is also the end of a debugging journey that transformed the proving pipeline from a fragile, OOM-prone system into a robust, hardware-aware one. The fix deployed here — dynamic concurrency scaling based on RAM and GPU count — would go on to enable reliable benchmarks on machines ranging from 125GB single-GPU instances to 2TB multi-GPU behemoths. In the broader arc of the session, this message marks the pivot from reactive firefighting to proactive system design, setting the stage for the data-driven experimental system that would follow.