"Still oom, go 32": The Art of Collaborative Parameter Tuning in ML Infrastructure

The message is deceptively simple. Three words, a comma, and a number: "Still oom, go 32" ([msg 39]). In the context of an opencode coding session, this brief user utterance represents a critical inflection point in a multi-hour struggle to build the flash-attn CUDA extension from source. It is a message born of frustration, impatience, and the hard-won intuition that comes from watching a 128-core machine buckle under the weight of its own parallelism. To understand why this message was written, one must understand the entire chain of events that led to it — a chain that reveals deep truths about the fragility of ML infrastructure, the hidden costs of parallel compilation, and the unique collaborative dynamics between a human operator and an AI assistant.

The Full Context: A Battle Against Memory Pressure

The session began with the straightforward goal of setting up a machine learning environment on a remote Ubuntu 24.04 server equipped with two NVIDIA RTX PRO 6000 Blackwell GPUs ([msg 3]–[msg 12]). The assistant successfully installed NVIDIA driver 590.48.01 and CUDA Toolkit 13.1, configured environment paths, and created a Python virtual environment using uv. Core packages like PyTorch (2.10.0+cu128) were installed without issue. The trouble began when the assistant attempted to install flash-attn, a high-performance CUDA kernel for attention mechanisms that is a dependency of many modern LLM serving stacks.

Because PyTorch 2.10.0 was compiled against CUDA 12.8 but the system had CUDA 13.1 installed, there was no precompiled wheel for flash-attn that matched this combination. The package had to be built from source — a process that invokes nvcc (the CUDA compiler) and ptxas (the PTX assembler) to compile dozens of CUDA kernel variants for different GPU architectures and attention configurations. Each compilation job consumes significant memory, and with 128 CPU cores available, the default parallelism was aggressive.

The assistant's first attempt used MAX_JOBS=8 ([msg 20]), a conservative setting. But after the user noted "it's a 128 core machine, abort and run faster" ([msg 27]), the assistant escalated to MAX_JOBS=128 ([msg 29]). This immediately caused an Out-of-Memory (OOM) condition. The user responded with "OOMing, try 64" ([msg 36]). The assistant killed the orphaned processes and relaunched with MAX_JOBS=64 ([msg 38]). That too failed. And so we arrive at the subject message: "Still oom, go 32" ([msg 39]).

What the Message Reveals About the Thinking Process

The brevity of this message is itself a form of communication. The user does not explain why 64 failed, nor do they provide system metrics or memory diagnostics. They simply state the outcome ("Still oom") and issue the next command ("go 32"). This telegraphic style implies a shared understanding: both the user and the assistant have been iterating on this problem long enough that the pattern is clear. Each reduction in MAX_JOBS is a step in a binary search for the memory ceiling of the system.

But there is an important assumption embedded here. The user assumes that the OOM is caused purely by the number of parallel compilation jobs, and that reducing MAX_JOBS will eventually yield a successful build. This is a reasonable hypothesis — each ptxas invocation consumes a few hundred megabytes of RAM, and with 128 concurrent jobs, the total could easily exceed the 288GB available. However, the assumption ignores other possible contributors: memory fragmentation, kernel module overhead, or the fact that some compilation steps (like linking) may require more memory than others regardless of job count.

The user also assumes that the assistant can simply "go 32" — that is, kill the current build, adjust the parameter, and restart. This assumes the machine is still responsive enough to receive commands. As the subsequent messages show ([msg 41]–[msg 43]), this assumption was optimistic: the machine was so deeply thrashing that SSH commands timed out, and the user ultimately had to reboot ([msg 44]). The OOM condition had pushed the system past the point of remote recoverability.

Input Knowledge Required to Understand This Message

To parse "Still oom, go 32" correctly, the reader (and the assistant) must understand several layers of context:

  1. The build process: flash-attn compiles CUDA kernels using nvcc and ptxas, and MAX_JOBS controls how many of these compilation tasks run in parallel. This is not a standard make -j flag — it is a variable read by the flash-attn build system (and by extension, PyTorch's cpp_extension module).
  2. The memory profile: Each compilation job, particularly ptxas assembling PTX code into device code for the Blackwell architecture (sm_100), requires a non-trivial amount of RAM. With 128 cores, the default parallelism could spawn 128 simultaneous ptxas processes.
  3. The prior iterations: The message references a history that includes MAX_JOBS=8 (successful but slow), MAX_JOBS=128 (OOM), and MAX_JOBS=64 (also OOM). The user is now proposing MAX_JOBS=32 as the next step in a halving search.
  4. The system constraints: The machine has 288GB of RAM and 128 CPU cores. The user's earlier instruction to "run faster" ([msg 27]) reflected an assumption that more parallelism would speed things up — an assumption now being corrected by empirical evidence.
  5. The tooling: The assistant is using uv pip install with --no-build-isolation and --no-cache-dir flags, and the build is happening inside an SSH session to a remote machine. The user cannot directly observe the OOM; they infer it from the build failure output or from the assistant's report.

Output Knowledge Created by This Message

This message generates several important outcomes:

  1. A new parameter value: The assistant now knows to set MAX_JOBS=32 for the next build attempt. This is a concrete, actionable piece of information.
  2. A refined mental model: Both user and assistant now know that the system's memory ceiling for this build is somewhere between 32 and 64 parallel jobs. This narrows the search space.
  3. A diagnostic signal: The fact that 64 jobs also OOM'd tells us something about the memory pressure. If 64 jobs consume ~288GB, each job might be using ~4-5GB — which seems high for a typical ptxas invocation. This suggests either that some jobs are much heavier than others, or that there is additional memory overhead (e.g., from the Python process, the ninja build system, or kernel module allocations).
  4. A decision point: The message implicitly asks the assistant to kill the running build and restart. This is a command, not a suggestion. The assistant's response (in the following messages) shows it attempting to comply, but the machine's unresponsiveness reveals that the OOM has already caused systemic failure.

Mistakes and Incorrect Assumptions

Several assumptions in this exchange proved incorrect:

The user's assumption that the machine would remain responsive. After the 64-job build OOM'd, the user expected the assistant to kill processes and relaunch. But the OOM condition had already triggered kernel-level memory pressure that made SSH connections time out. The assistant's kill commands at [msg 41] and [msg 42] both timed out. The machine had to be physically rebooted ([msg 44]).

The assistant's assumption that pkill would work under OOM conditions. The assistant issued pkill -9 -f ptxas and similar commands, but these require the kernel to allocate memory for process management — something that becomes unreliable under extreme memory pressure. The -9 signal (SIGKILL) cannot be caught or ignored, but it still requires the kernel to schedule the signal delivery, which may not happen promptly when the system is thrashing.

The shared assumption that the problem was purely about parallelism. While MAX_JOBS was certainly a factor, the root cause may have been more nuanced. The build was targeting only the Blackwell architecture (TORCH_CUDA_ARCH_LIST="10.0"), which should have reduced the number of kernels to compile. But flash-attn compiles multiple kernel variants (different head dimensions, data types, etc.) for each architecture, and some of these variants may be much more memory-intensive than others. A single ptxas invocation for a large kernel could consume several gigabytes on its own.

The assumption that the machine had 288GB of usable RAM. The earlier nvidia-smi output showed two GPUs, but the system's total RAM was reported as 288GB. However, some of this memory may have been reserved by the kernel, the GPU driver (NVIDIA drivers can reserve significant system memory for GSP firmware and other purposes), or other processes. The actual free memory available for compilation may have been substantially less.

The Broader Significance

This message, for all its brevity, captures something essential about the practice of ML infrastructure engineering. Building CUDA extensions from source is a notoriously fragile process. The combination of PyTorch's version-specific CUDA compatibility requirements, the lack of precompiled wheels for cutting-edge GPU architectures (Blackwell/sm_100 in this case), and the memory-intensive nature of CUDA compilation creates a perfect storm of configuration hell.

The iterative reduction of MAX_JOBS — from 128 to 64 to 32 — is a textbook example of troubleshooting by parameter search. It mirrors the way engineers tune batch sizes, learning rates, or thread counts in production systems. Each failed attempt provides information, narrowing the feasible range. The user's message "Still oom, go 32" is the third step in what would become a longer search: after the reboot, the assistant would eventually succeed with MAX_JOBS=20 on a machine with 432GB of RAM ([chunk 0.0]).

The message also highlights the unique collaborative dynamic of the opencode session. The user is not writing code or issuing shell commands directly; they are directing an AI assistant that executes on their behalf. The communication must be compressed, relying on shared context and implicit understanding. "Still oom, go 32" works as a message only because both parties have been following the same thread, watching the same failures, and building the same mental model of the system's constraints. It is a message that could not exist without the preceding 38 messages, and it would be incomprehensible without them.

In the end, this tiny utterance — three words and a number — encapsulates the entire challenge of ML infrastructure: the gap between what should work (128 cores should compile faster!) and what actually works (32 jobs, maybe, after a reboot). It is a reminder that in systems engineering, theory must always yield to measurement, and that the most valuable tool in the debugging arsenal is the willingness to try a smaller number.