The 15.09 GiB Mystery: Debugging a Persistent OOM on Blackwell GPUs

In the high-stakes world of training large language models on bleeding-edge hardware, few experiences are as maddening as an out-of-memory (OOM) error that refuses to budge no matter what parameter you tweak. This is the story of one such moment in an opencode coding session — a single message (index 7878) from an AI assistant debugging a DFlash speculative decoding drafter training run on a 4× NVIDIA RTX PRO 6000 Blackwell GPU node. The message captures a pivotal realization: the OOM error was identical — exactly 15.09 GiB — regardless of whether the assistant used 512, 256, or 128 anchor points. This invariance meant the entire multi-hour debugging effort had been chasing the wrong variable.

The Setting: Training a DFlash Drafter on Blackwell

To understand the significance of this message, we need to set the stage. The project involved training a DFlash (Drafting with Flash Attention) model — a lightweight speculative decoding drafter designed to accelerate inference for the Qwen3.6-27B language model. The training setup was complex: two GPU pairs (DP=2) on a single machine with 4× RTX PRO 6000 Blackwell GPUs, each with 96 GB of VRAM. The target model (Qwen3.6-27B) ran on GPUs 0 and 1, while the drafter models occupied GPUs 2 and 3. The training pipeline involved packing multiple sequences into a single batch using a token budget of 8192, computing hidden states from the target model, and training the drafter with a combination of language modeling loss and distillation signals.

The session had already survived a cascade of hardware-specific failures. Earlier, the assistant had fixed six training script bugs, resolved FLA (Flash Linear Attention) Triton autotuner crashes on the new sm_120 Blackwell architecture by clearing corrupted disk caches and adding sequential warmup, and addressed an OOM from unfused flex_attention backward passes by implementing lazy compilation. After upgrading Triton from 3.6.0 to 3.7.0, the team had launched what they hoped would be a stable training run with 512 anchors, DP=2, and an 8192 token budget.

It crashed immediately with an OOM.

The Cascade of Misdiagnosis

What followed was a classic debugging spiral. The assistant, observing the OOM error pointing to a 15.09 GiB allocation failure with 84.07 GiB already in use, assumed the culprit was the attention score matrix size. The reasoning was straightforward: the unfused flex_attention backward pass materializes a full attention score matrix of shape [batch, heads, Q_len, KV_len] in float32. With 512 anchors and a block size of 16, the query length would be 512 × 16 = 8192 tokens. Combined with a packed sequence of ~8192 tokens, the KV length would be roughly 16,384. The score matrix alone would be 8192 × 16384 × 32 heads × 4 bytes ≈ 17.2 GB — well over budget.

The logical fix was to reduce the anchor count. The assistant tried 256 anchors (Q_len = 4096, expected score matrix ≈ 6.44 GB), then 128 anchors (Q_len = 2048, expected score matrix ≈ 2.68 GB). Each time, the training script was relaunched with pkill, cache clearing, and a fresh invocation. Each time, the same error returned: "Tried to allocate 15.09 GiB" with "84.07 GiB memory in use."

The Epiphany: Message 7878

Message 7878 is where the assistant stops and thinks. The agent reasoning section — a stream-of-consciousness analysis — reveals the critical insight:

"The OOM is EXACTLY the same — 'Tried to allocate 15.09 GiB' and '84.07 GiB memory in use' — regardless of whether we use 512, 256, or 128 anchors. This means the issue is NOT the anchor count but something else entirely."

This is the moment of diagnostic pivot. The assistant recognizes that the invariance of the allocation size across three different configurations invalidates the anchor-count hypothesis. If the score matrix size depended on anchor count, changing anchors would change the allocation. The fact that it didn't meant either (a) the anchor parameter wasn't actually being changed, (b) the OOM was coming from a different tensor entirely, or (c) some other component was dominating memory regardless of anchor settings.

The Forensic Memory Analysis

What follows is a remarkable piece of forensic debugging. The assistant performs a detailed memory accounting exercise, working backwards from the allocation size to identify the offending tensor.

First, the raw arithmetic: 15.09 GiB converts to 16,199,917,568 bytes. In float32, that's approximately 4 billion elements. In BF16, it's 8 billion elements. If this is an attention score matrix with 32 heads, the product Q_len × KV_len would be roughly 126 million per head. The assistant tries various combinations: with Q_len = 8192 (512 anchors), KV_len would need to be ~15,448 — far exceeding the 8192 token budget. With Q_len = 2048 (128 anchors), KV_len would need to be ~61,800 — an absurdly large packed sequence.

The assistant systematically rules out other candidates. The drafter's language model head output at 512 anchors would be 8192 × 248320 × 2 bytes ≈ 4.07 GB in BF16 — not 15 GB. The cross-entropy backward pass's softmax-minus-one-hot operation on logits of shape [2048, 248320] in float32 would be about 2 GB. The auxiliary hidden states from the target model are only 320 MB. None of these match.

The reasoning then explores more exotic possibilities: could the target model's hidden states be leaking onto the drafter GPU through the computation graph? Are the hooks that detach outputs failing to break autograd connections? Is the ThreadPoolExecutor causing tensor duplication across GPU pairs? The assistant even considers whether the verifier logits computation — a 4 GB tensor in BF16 — is being retained unnecessarily.

This is not just idle speculation. Each hypothesis is grounded in the assistant's understanding of the training code's architecture: the hook-based hidden state extraction, the no_grad context for verifier computation, the cross-entropy loss internals, and the memory management of PyTorch's autograd engine.

The Hidden Assumption

Throughout this reasoning, one assumption goes unexamined: that the training log being read corresponds to the most recent run. The assistant has been killing processes, clearing caches, and relaunching with different parameters, but each new invocation writes to the same file (/workspace/train.log) using shell redirect (>). If the new process crashes before writing any output — or if the log file isn't flushed before the crash — the old log content remains, creating a false impression that the new configuration also failed.

The assistant eventually arrives at this possibility, but only after an exhaustive memory analysis. The reasoning shows the assistant considering whether to add memory profiling directly into the training step, and whether to test DP=1 to isolate the issue. But the immediate action is pragmatic: "Let me get the full log to understand what's happening."

The bash command issued at the end of the message is telling:

ssh ... 'pkill -f train_dflash 2>/dev/null; grep -E "step|Training|Loaded|Drafter|Batches|anchors|ERROR|OOM|allocat" /workspace/train.log | head -30'

The pkill at the start kills any lingering process, and the grep searches for key indicators. The command returns no output — which in itself is suspicious. A training log with 90 lines (as revealed in the next message) should contain many of these keywords. The empty result is the first clue that something is wrong with the log.

What This Message Reveals About Debugging Methodology

Message 7878 is a masterclass in structured debugging under pressure. Several methodological insights stand out:

1. Invariance as a diagnostic tool. The assistant recognizes that a parameter change producing no change in the error signal is itself a signal. This is the debugging equivalent of the scientific method's control experiment: if varying X doesn't change Y, then Y doesn't depend on X.

2. Working backwards from quantitative evidence. Rather than guessing, the assistant converts the error's allocation size (15.09 GiB) into bytes, then into element counts, then into possible tensor shapes, systematically checking each against known model dimensions. This is memory forensics at the byte level.

3. The importance of understanding the full software stack. The reasoning traverses multiple layers: the training script's batching logic, PyTorch's autograd engine, Triton's kernel compilation, FLA's attention implementations, and even the shell's file redirect semantics. Debugging at this depth requires mental models of each layer.

4. The trap of stale state. The assistant's assumption that the log file reflects the latest run is a classic debugging pitfall. When processes crash silently or before flushing output, the filesystem preserves the last successful state, creating an illusion of continuity. The assistant falls into this trap despite being otherwise thorough — a reminder that even experienced debuggers have blind spots.

The Aftermath

The messages immediately following 7878 reveal the resolution. The grep output in message 7879 shows line 26: "max anchors: 512" — confirming the log was from an old run. The timestamp on the file (20:17) predates the latest launch. The process had died so quickly that the shell redirect never wrote any output. The assistant's subsequent commands — checking ps aux, verifying file timestamps, running with | head -60 instead of background redirect — all demonstrate the lesson learned: never trust a log file without verifying it was actually written by the process you're debugging.

Broader Lessons for ML Engineering on New Hardware

This episode illustrates several recurring themes in machine learning engineering on cutting-edge hardware:

The fragility of early-stage infrastructure. Training on new GPU architectures (Blackwell/sm_120) means encountering bugs that don't exist on mature platforms. The FLA autotuner race condition, the unfused flex_attention backward, and the Triton cache corruption are all architecture-specific issues that wouldn't appear on Ampere or Hopper GPUs.

The compounding effect of complexity. The training pipeline combines multiple novel components: DFlash's anchor-based attention, online distillation from a frozen target model, DP parallelism across GPU pairs, and FLA's Triton kernels. Each component introduces failure modes, and their interactions create emergent bugs that are difficult to isolate.

The importance of logging discipline. A single > redirect vs. >> append, or a missing flush=True in Python, can waste hours of debugging time. The assistant's eventual switch to piping output to stdout (| head -60) rather than writing to a shared file is a pragmatic workaround, but the root cause is a logging architecture that doesn't distinguish between runs.

The value of structured reasoning under pressure. When faced with a persistent error and mounting frustration (the user had already asked "crashed?" twice), the assistant resists the temptation to keep tweaking parameters and instead performs a systematic analysis. This discipline — stopping to think rather than acting — is what ultimately leads to the correct diagnosis.

Conclusion

Message 7878 captures a turning point in a complex debugging session. The assistant's realization that the OOM error was invariant across anchor counts represents a classic debugging breakthrough: recognizing when the evidence contradicts your hypothesis and having the intellectual honesty to abandon it. The subsequent memory analysis demonstrates deep systems knowledge, while the eventual discovery of the stale log file reveals a more mundane but equally important lesson about the reliability of debugging artifacts.

In the end, the 15.09 GiB allocation wasn't about anchors at all — it was about trusting the wrong log file. But the journey to that realization, captured in this single message, is a testament to the value of rigorous reasoning in the face of opaque hardware failures. For anyone debugging ML training on new GPU architectures, the lesson is clear: when the error doesn't change, change your assumptions — and always check your logs.