"Still oom, go 32": The Four Words That Tell a Thousand Stories About Building CUDA Software

The subject message is breathtakingly brief — just four words, no capitalization, no punctuation, no pleasantries:

Still oom, go 32

Yet this tiny utterance, sent by the user at index 40 of an opencode coding session (and identically at index 39), sits at the climax of an hours-long struggle to build the flash-attn library on a machine with two NVIDIA RTX PRO 6000 Blackwell GPUs. To understand why these four words matter, we must unpack the entire saga that led to them — a saga of conflicting CUDA toolkits, memory-exhausted compilation processes, and the delicate art of tuning parallel build jobs on a machine that should, by all rights, have had plenty of headroom.

The Long Road to an OOM

The message does not exist in isolation. It is the culmination of a painful, iterative debugging process that began many messages earlier. The assistant had successfully installed NVIDIA drivers (version 590.48.01) and CUDA Toolkit 13.1 on a fresh Ubuntu 24.04 machine, verified two RTX PRO 6000 Blackwell GPUs (each with ~96GB of VRAM), created a Python virtual environment using uv, and installed PyTorch. Everything was going smoothly — until the assistant tried to install flash-attn, a high-performance attention kernel library that is essential for running modern large language models efficiently.

The first problem was a CUDA version mismatch. PyTorch 2.10 had been compiled against CUDA 12.8, but the system had CUDA 13.1 installed. The flash-attn build process, which compiles CUDA kernels from source, detected this mismatch and refused to proceed. The assistant solved this by installing a secondary CUDA 12.8 toolkit alongside the primary 13.1 installation, then pointing the build at the correct CUDA home directory.

The second problem — the one that leads directly to our subject message — was memory exhaustion. Building flash-attn from source is an extraordinarily memory-intensive operation. The library compiles dozens of CUDA kernel variants for different GPU architectures, attention head dimensions, and data types. Each compilation invokes nvcc (the NVIDIA CUDA compiler) and ptxas (the PTX assembler), and each such process can consume gigabytes of RAM. The MAX_JOBS environment variable controls how many of these compilation processes run in parallel.

The Descent: 128 → 64 → 32

The trajectory of MAX_JOBS values tells the story. The machine has 128 CPU cores, so setting MAX_JOBS=128 was the natural first choice — run one compilation job per core. The user explicitly requested this: "it's a 128 core machine, abort and run faster" (msg 27). The assistant complied, killing the previous lower-job-count build and relaunching with MAX_JOBS=128.

It OOM'd immediately. The machine's 288GB of RAM was simply not enough to run 128 parallel CUDA compilation processes.

The user then instructed "OOMing, try 64" (msg 36). The assistant killed the orphaned processes (which required escalating force — pkill -9 -f ptxas, pkill -9 -f nvcc, pkill -9 -f cicc, pkill -9 -f ninja) and relaunched with MAX_JOBS=64.

It OOM'd again.

This brings us to the subject message. The user, having watched two attempts fail, issues the next halving: "Still oom, go 32." The message is sent not once but twice — at both index 39 and index 40 — suggesting either impatience with the assistant's response time or a desire to ensure the instruction was received.

What This Message Reveals About the Debugging Process

The message is a masterclass in human-in-the-loop debugging. The user is acting as a real-time monitor, watching the build output (presumably via the tail -30 appended to each build command) and making decisions based on observed failure modes. The assistant cannot see the build output until the command completes (or fails), so the user serves as an external observer providing feedback that the assistant cannot gather autonomously.

The decision to halve MAX_JOBS from 64 to 32 is not arbitrary. It follows a classic binary-search approach to finding the maximum parallelism the system can sustain. The user could have suggested 48 or 40, but halving is the fastest way to find a working configuration when you don't know the exact memory budget per compilation job. Each CUDA kernel compilation has a variable memory footprint depending on the kernel complexity, so predicting the exact limit is impossible without empirical testing.

Assumptions and Their Consequences

Several assumptions underpin this message, and some proved incorrect. The user assumed that 288GB of RAM would be sufficient for 128 parallel CUDA compilations — a reasonable assumption given that each compilation typically needs 1-4GB, suggesting a theoretical capacity of 72-288 parallel jobs. The reality was different: some kernel compilations spike to 8-10GB or more, and the memory overhead of the build system (ninja, Python, linker invocations) adds significant baseline consumption.

The user also assumed that simply killing the build processes and restarting with fewer jobs would work. This proved partially true — but only after the machine was rebooted with expanded RAM (432GB) in a later iteration. The OOM condition had apparently left the system in a degraded state where even reduced parallelism failed.

Input Knowledge Required

To understand this message, a reader needs considerable context:

Output Knowledge Created

This message creates actionable knowledge: it establishes that MAX_JOBS=64 is too high for this system configuration, and it narrows the search space to MAX_JOBS=32 as the next candidate. It also implicitly documents the system's memory characteristics — the per-job memory footprint of flash-attn compilation is at least 4.5GB (288GB / 64 jobs), which is valuable diagnostic information for future builds.

The Broader Significance

This four-word message encapsulates a fundamental truth about building machine learning infrastructure: the hardware specifications that look impressive on paper (128 cores, 288GB RAM, two flagship GPUs) can still be insufficient for the peculiar demands of CUDA compilation. The build process for flash-attn is an outlier in its memory intensity, requiring more RAM than many production inference workloads. The message also illustrates the tight feedback loop between human and AI in opencode sessions — the assistant executes, the human observes, the human directs, and the cycle repeats until a working configuration is found.

In the end, even MAX_JOBS=32 would not work immediately. The machine had to be rebooted with expanded memory (432GB) before flash-attn could be successfully compiled. But the message "Still oom, go 32" marks the critical turning point where the debugging strategy shifted from "how many jobs can we run?" to "something deeper is wrong with the system." It is a testament to the patience required when building cutting-edge ML software, and a reminder that even the most powerful hardware can be humbled by a library's build system.