The Surgical Compile: Debugging OOM in DFlash Training on Blackwell GPUs

In the middle of a high-stakes debugging session on a 4× RTX PRO 6000 Blackwell node, the assistant issued a single, deceptively simple edit command:

[edit] /data/dflash/scripts/dflash_model.py Edit applied successfully.

This message, <msg id=7863>, is a tool call confirmation—the assistant modified a Python file and the system confirmed the change was applied. On its surface, it is the most mundane kind of message in any coding session: a file was edited, and it worked. But this edit was the culmination of an intense chain of reasoning about GPU memory management, PyTorch's torch.compile infrastructure, and the peculiarities of running speculative decoding training on NVIDIA's newest Blackwell architecture (sm_120). Understanding why this edit was made, what it attempted to do, and why it ultimately failed reveals a rich story about systems debugging at the frontier of hardware and software.

The Crisis: OOM on the Drafter GPU

To understand the edit, one must understand the crisis that precipitated it. The assistant was training a DFlash (Drafting with Flash Attention) model—a speculative decoding architecture where a small "drafter" model learns to predict the outputs of a larger "target" model. The training loop used data parallelism across two GPU pairs (DP=2), with GPUs 0–1 handling one pair and GPUs 2–3 handling the other. The target model (Qwen3.6-27B) ran on GPUs 0 and 1, while the drafter model ran on GPUs 2 and 3.

After clearing a corrupted Triton disk cache that had been causing FLA (Flash Linear Attention) autotuner crashes, the assistant launched a full training run. It crashed with an out-of-memory (OOM) error on GPU 2—one of the drafter GPUs. The error message reported that PyTorch had 84 GiB of memory in use and tried to allocate an additional 15.09 GiB, exceeding the 96 GiB capacity of the RTX PRO 6000 Blackwell GPU.

The assistant's reasoning, visible in <msg id=7862>, traced the OOM to a specific root cause: the unfused backward pass of flex_attention. PyTorch's flex_attention is a higher-order operator that supports flexible attention masks. When called outside a torch.compile context, its backward pass materializes the full attention score matrix as a dense tensor. With the training configuration using max_anchors=512 and block_size=16, the query length was 8192 tokens. The key-value length was the packed sequence (up to 8192 tokens) plus the query length, totaling approximately 16,384 tokens. With 32 attention heads and 4 bytes per element (float32), a single score matrix consumed roughly 16 GiB per layer. Across 5 drafter layers, that was 80 GiB—catastrophically exceeding the available memory.

The Reasoning: Why Compile Only Flex Attention?

The assistant's reasoning reveals a careful cost-benefit analysis. The obvious solution was to use torch.compile on the entire drafter model, which would fuse the attention backward pass and avoid materializing the score matrix. However, the assistant had just spent hours debugging FLA Triton autotuner crashes on sm_120 (Blackwell). The FLA library, used by the target model's Gated Delta Net (GDN) layers, had been crashing with TypeError: 'NoneType' object is not a mapping errors in its custom autotuner. Compiling the entire model risked triggering those crashes again.

The assistant therefore chose a surgical approach: compile only the flex_attention_forward function itself, not the whole drafter model. This was an elegant compromise. The target model's FLA kernels would run uncompiled (avoiding the autotuner race condition), while the drafter's attention computation would benefit from fused kernels that eliminated the memory-hungry score matrix materialization. The edit in <msg id=7863> added a @torch.compile decorator or equivalent compilation wrapper to the flex_attention_forward function defined in dflash_model.py.

The Assumption That Failed

The assistant made a reasonable but incorrect assumption: that compiling the flex_attention function at module definition time would cause the fused backward kernel to be selected during training. PyTorch's flex_attention documentation does indicate that compilation is required for fused kernel dispatch, and compiling the function directly seemed like the right approach.

But as the assistant discovered in <msg id=7867>, this assumption was wrong. The compiled flex_attention was still falling through to the unfused sdpa_dense_backward implementation. The error traces showed the backward pass calling the dense fallback, materializing the full score matrix, and exhausting GPU memory. The 15.09 GiB allocation persisted unchanged across runs with 512, 256, and even 128 anchors—confirming that the anchor count was irrelevant because the compilation wasn't taking effect.

The root cause, as the assistant later reasoned, was that torch.compile(flex_attention) at module level wasn't sufficient because flex_attention is a higher-order operator with special handling in PyTorch's compilation pipeline. The fused backward kernel requires the entire calling context to be compiled—not just the function itself. The create_block_mask call, which generates dynamic attention masks using closures, was creating graph breaks that prevented torch.compile from tracing through the full attention computation.

The Thinking Process Visible in the Reasoning

The assistant's reasoning in <msg id=7862> and the surrounding messages reveals a systematic diagnostic process. First, it calculated the exact memory footprint: "8192 16384 32 heads * 4 bytes per layer, which comes to 16 GB per layer. With 5 layers, we're looking at 80 GB total." This quantitative analysis grounded the debugging in concrete numbers.

The assistant then explored multiple solution paths in parallel: reducing max_anchors to shrink the query length, using PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True for more flexible memory management, gradient checkpointing to trade computation for memory, and selective compilation of just the attention function. The surgical compile approach was chosen as "the cleanest solution to try first" because it addressed the root cause (unfused backward) without disrupting the fragile FLA kernels.

When the approach failed, the assistant's reasoning in <msg id=7867> showed a sophisticated understanding of PyTorch internals: "The issue might be that torch.compile(flex_attention) needs to be called within a compiled context (i.e., the calling function also needs to be compiled) for the fused kernel to be selected." This diagnosis led to reverting the change and trying a different strategy—compiling the full drafter forward pass with --compile after confirming that the earlier FLA crashes were caused by a corrupted Triton cache, not by compilation itself.

Lessons for Systems Debugging

The edit in <msg id=7863> and its aftermath illustrate several principles of debugging at the frontier of ML hardware and software. First, when multiple bugs interact (FLA autotuner crashes + OOM from unfused attention), isolating them is critical. The assistant correctly separated the corrupted Triton cache issue from the OOM issue by clearing caches and observing that FLA worked but memory was still exhausted.

Second, surgical interventions are tempting but often insufficient when the underlying infrastructure (PyTorch's compilation pipeline) has specific requirements. The assumption that compiling a single function would fuse its backward pass was reasonable but wrong—the compilation context needed to encompass the entire calling scope.

Third, quantitative reasoning about memory is essential. The assistant's ability to calculate tensor sizes, trace memory allocation, and identify the 15.09 GiB allocation as the score matrix was what made the debugging tractable. Without this quantitative grounding, the OOM errors would have remained opaque.

The edit itself was a dead end—it was reverted in <msg id=7867> and replaced with full-model compilation. But the reasoning behind it, the assumptions it embodied, and the diagnostic process it was part of represent the real value of this message. It is a snapshot of a developer working at the edge of what current ML infrastructure supports, making calculated bets about how to balance competing constraints, and learning from each failure.