The Phantom OOM: How a Misattributed GPU Crash Revealed Deeper Training Pipeline Instability

Introduction

In the high-stakes world of large-scale machine learning training, few events are as disruptive and mystifying as the "phantom" out-of-memory (OOM) error—a crash that appears to have been fixed, only to resurface on a different component, in a different location, with a different signature. This is precisely the situation captured in message 9673 of an opencode coding session, where an AI assistant is attempting to restart a distributed DFlash drafter training run across 8 NVIDIA RTX PRO 6000 Blackwell GPUs. The message represents a critical diagnostic pivot: after restarting training with what the assistant believed were sufficient memory optimizations, the throughput collapsed to a quarter of its previous rate, and a routine error check revealed that the OOM had simply migrated from one GPU to another, rather than being resolved.

This article examines message 9673 in detail—the reasoning, assumptions, decisions, and discoveries contained within it—and explores how this single diagnostic moment exposed the fragility of the training pipeline and the limitations of the assistant's memory optimization strategy.

The Message in Context

To understand message 9673, one must appreciate the sequence of events that preceded it. The training run had been progressing smoothly at approximately 20.1 Ktok/s (thousands of tokens per second) with a 5-target, 3-drafter GPU topology, when GPU 6—one of the three drafter GPUs—suffered an OOM crash ([msg 9662]). Remarkably, the training pipeline continued operating on the two surviving drafters, maintaining 20.2 Ktok/s throughput with only a 5% performance penalty. The user, however, was not satisfied with this degraded state and explicitly instructed the assistant to restart from scratch, re-engage all three drafter GPUs, and preserve the critical training signal parameters—anchors=1024 and block_size=32—while tuning "non-harmful" parameters like batch size to resolve the memory pressure ([msg 9664]).

The assistant complied, killing the running session and crafting a new training script with two key modifications: token_budget was reduced from 49152 to 45056, and max_batch_size was cut from 64 to 48 ([msg 9669]). These changes were intended to reduce the number of sequences packed per batch, thereby lowering peak GPU memory consumption without altering the anchor/block configuration that defines the training signal. The assistant also added PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True,max_split_size_mb:512 and CUDA_MODULE_LOADING=LAZY as additional memory management measures.

The training restarted and initially appeared to be progressing, with steps advancing from 690 to 707. However, the throughput was alarmingly low—only 5.4 Ktok/s compared to the previous 20+ Ktok/s—and the ETA had ballooned to over 15 days. This is the state in which message 9673 begins.

The Subject Message: A Diagnostic Pivot

Message 9673 opens with the assistant's internal reasoning, which reveals a careful but ultimately incomplete analysis of the situation:

Hmm, the throughput is only 5.4 Ktok/s — much lower than before. And looking at GPU utilization: - GPU 0: 0% util, 92 GB — target, probably just finished a batch - GPU 1: 0% util, 72 GB — target - GPU 2: 0% util, 96 GB — target - GPU 3: 0% util, 79 GB — target - GPU 4: 100% util, 96 GB — target, actively processing - GPU 5: 100% util, 80 GB — drafter, active - GPU 6: 0% util, 59 GB — drafter, but only 59 GB? Seems low compared to 80GB on GPU 5 - GPU 7: 0% util, 58 GB — drafter, also low

The assistant's reasoning here is methodical and structured. It begins by observing the symptom (low throughput), then examines the GPU utilization data to identify which components are active. The observation that GPUs 6 and 7 are idle with significantly lower memory consumption (59 GB and 58 GB vs. 80 GB on GPU 5) is the critical clue. The assistant correctly hypothesizes that these drafters may have crashed or are stalled, but it also hedges appropriately: "I should remember that utilization snapshots are instantaneous, so this could just be a momentary state."

This hedging is important. In distributed training, GPU utilization snapshots are indeed point-in-time measurements, and it is possible to catch the system in a transient state where some GPUs are between batches. However, the combination of low throughput, high ETA, and persistently idle GPUs points to a systemic issue rather than a momentary blip.

The assistant then pivots to the next diagnostic step: checking the training logs for errors. It issues a bash command to grep for "error", "OOM", "out of memory", and "Traceback" in the training output log:

ssh -o ConnectTimeout=10 root@10.1.2.6 "pct exec 200 -- bash -c 'grep -i \"error\\|OOM\\|out of memory\" /workspace/checkpoints/train_stdout.log 2>/dev/null | tail -5; echo; grep \"Traceback\" /workspace/checkpoints/train_stdout.log 2>/dev/null | wc -l; echo tracebacks'"

The result is devastating in its clarity:

torch.OutOfMemoryError: CUDA out of memory. Tried to allocate 4.74 GiB. GPU 5 has a total capacity of 94.97 GiB of which 4.57 GiB is free. Including non-PyTorch memory, this process has 90.29 GiB memory in use. Of the allocated memory 88.96 GiB is allocated by PyTorch, and 640.21 MiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True to avoid fragmentation.

The Critical Discovery: A Migrating OOM

The most important finding in this message is that the OOM occurred on GPU 5, not GPU 6. This is a pivotal discovery because the entire memory optimization strategy had been predicated on the assumption that GPU 6 was the weak link. The assistant had reduced token_budget and max_batch_size to protect against the OOM that previously struck GPU 6, but the crash simply relocated to GPU 5.

This phenomenon—a "migrating OOM"—is a well-known challenge in distributed training debugging. When multiple GPUs are operating near their memory limits, small imbalances in workload distribution, memory fragmentation patterns, or the order of tensor allocations can cause different GPUs to hit the ceiling at different times. Reducing the overall memory pressure (via token_budget reduction) may delay the OOM but not eliminate it, because the underlying issue is not the total memory consumption but rather the distribution of that consumption across GPUs and the specific allocation patterns on each device.

The OOM error message itself is instructive. GPU 5 had 94.97 GiB total capacity, of which only 4.57 GiB was free when the allocation of 4.74 GiB was attempted. The process had 90.29 GiB in use, with 88.96 GiB allocated by PyTorch and 640.21 MiB reserved but unallocated. This is a razor-thin margin—the system was operating at approximately 95% of GPU memory capacity, and a single allocation request 170 MiB larger than available free memory triggered the crash.

The error message also contains a helpful suggestion: "If reserved but unallocated memory is large try setting PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True to avoid fragmentation." Notably, the assistant had already enabled this setting in the training script, yet the OOM still occurred. This suggests that the memory pressure was not primarily a fragmentation issue but rather a fundamental capacity problem.

Assumptions and Their Consequences

Message 9673 reveals several critical assumptions made by the assistant, some of which proved incorrect:

Assumption 1: The OOM would recur on GPU 6. The assistant's entire memory optimization strategy was designed to protect GPU 6, the drafter that had crashed in the previous run. The reduction in token_budget and max_batch_size was intended to reduce memory pressure on the drafter GPUs collectively, but the specific targeting of GPU 6's weakness meant that when the OOM migrated to GPU 5, the fix was revealed as insufficient.

Assumption 2: Reducing token_budget would proportionally reduce drafter memory consumption. The token_budget parameter controls how many tokens are packed into each batch for the target model's hidden state extraction. While this indirectly affects the drafter pipeline (fewer tokens means fewer hidden states to process), the drafter's peak memory is more directly tied to the anchor and block size configuration (1024 anchors × 32 block size = 32,768 tokens through the lm_head). Reducing the input token budget does not proportionally reduce the drafter's peak memory during the backward pass.

Assumption 3: The memory optimization was sufficient. The assistant calculated that the OOM was only ~200 MB short and believed that reducing token_budget by ~8% (49152 to 45056) and max_batch_size by 25% (64 to 48) would provide enough headroom. However, the OOM on GPU 5 required 4.74 GiB with only 4.57 GiB free—a 170 MiB shortfall—indicating that the memory reduction was either not applied to the right GPU or was consumed by other overheads.

Assumption 4: The low throughput was a symptom of the OOM rather than a separate issue. The assistant initially focused on the 5.4 Ktok/s throughput as the primary anomaly and investigated GPU utilization to understand it. The OOM discovery explained why GPUs 6 and 7 were idle (they had crashed), but it did not fully explain why GPU 5, which was still active at 100% utilization, was only producing 5.4 Ktok/s. The throughput collapse may have additional causes beyond the OOM—such as the reduced token_budget creating smaller batches that underutilize the pipeline, or the surviving drafter (GPU 5) being bottlenecked by the target GPUs' memory pressure.

Input Knowledge Required to Understand This Message

To fully grasp the significance of message 9673, one needs:

  1. Understanding of the DFlash training architecture: The system uses a pipeline where 5 target GPUs extract hidden states from a frozen target model (Qwen3.6-27B), and 3 drafter GPUs train a small drafting model using those hidden states. The drafters communicate with the targets via a shared queue of hidden states (q_hs).
  2. Knowledge of the anchor/block configuration: The training signal is defined by max_anchors=1024 and block_size=32, meaning the drafter processes 32,768 tokens per batch through its lm_head. These parameters are considered inviolable because they define the drafting window.
  3. Awareness of the dependency version history: The original stable configuration (5 targets + 3 drafters at 20+ Ktok/s) was achieved with PyTorch 2.11 compiled against CUDA 12.8. An upgrade to CUDA 13.0 added approximately 200 MB of overhead per GPU, which is believed to be the root cause of the OOM.
  4. Familiarity with distributed training diagnostics: The ability to interpret GPU utilization snapshots, memory consumption patterns, and training throughput metrics is essential for understanding why the assistant focused on GPU 6/7 idleness and the low throughput.

Output Knowledge Created by This Message

Message 9673 produces several critical pieces of knowledge:

  1. The OOM has migrated from GPU 6 to GPU 5. This is the most important finding, as it invalidates the assumption that reducing token_budget would resolve the memory pressure on the drafter GPUs.
  2. The memory optimization was insufficient. Despite reducing token_budget by 8% and max_batch_size by 25%, the system still OOMs—now on a different GPU. This suggests that the memory pressure is more fundamental than batch packing parameters.
  3. The training pipeline does not gracefully handle drafter crashes. When GPUs 6 and 7 OOM'd, the pipeline continued on GPU 5 alone, but at severely degraded throughput (5.4 Ktok/s vs. 20+ Ktok/s). The pipeline's fault tolerance is limited: it can survive drafter failures, but performance collapses.
  4. The expandable_segments setting alone is not sufficient. The assistant had enabled this PyTorch memory management feature, yet the OOM still occurred, indicating that the memory pressure is a capacity issue rather than a fragmentation issue.
  5. The target GPUs are under memory pressure as well. The assistant notes that GPUs 0 and 2 are using 92-96 GB (near the 94.97 GB capacity), suggesting that the target model inference is also memory-constrained with the longer sequences from the expanded dataset.

The Thinking Process: A Window into Diagnostic Reasoning

The assistant's reasoning in message 9673 is a textbook example of structured diagnostic thinking in machine learning engineering. It follows a clear pattern:

  1. Observe the symptom: Throughput is 5.4 Ktok/s, much lower than expected.
  2. Gather data: Examine GPU utilization and memory consumption across all 8 GPUs.
  3. Formulate hypotheses: GPUs 6 and 7 may have crashed; the low throughput may be due to drafter failures.
  4. Test hypotheses: Check the training logs for errors and OOM messages.
  5. Discover the root cause: The OOM is on GPU 5, not GPU 6.
  6. Revise understanding: The memory optimization was insufficient, and the OOM has migrated. However, the reasoning also reveals a blind spot. The assistant does not fully connect the dots between the dependency version upgrade (torch cu128 → cu130) and the OOM. The earlier analysis had identified that the cu130 upgrade added ~200 MB of overhead per GPU, but the assistant chose to work around this by reducing batch parameters rather than reverting the torch version. Message 9673 demonstrates that this workaround was insufficient, and the correct fix—reverting to cu128—would eventually be adopted in the subsequent messages.

Broader Implications

Message 9673 is a microcosm of a larger challenge in ML infrastructure: the tension between configuration optimization and dependency management. The assistant attempted to solve a memory problem by tuning configuration parameters (token_budget, max_batch_size) while preserving the training signal (anchors, block_size). This is a reasonable approach—configuration parameters are safer to modify than dependencies, and they don't require reinstalling packages or rebuilding environments.

However, the message reveals the limits of this approach. When the memory overhead comes from a fundamental change in the software stack (cu128 → cu130), configuration tuning can only provide marginal relief. The OOM will keep migrating across GPUs until the root cause—the version upgrade—is addressed. This is a lesson that applies broadly to ML engineering: configuration optimization can mask symptoms, but it rarely cures diseases.

Conclusion

Message 9673 captures a moment of diagnostic clarity in a complex distributed training scenario. The assistant's methodical investigation—observing low throughput, examining GPU utilization, checking error logs—reveals that a memory optimization strategy predicated on protecting GPU 6 was fundamentally flawed, as the OOM simply migrated to GPU 5. The message demonstrates both the power of structured diagnostic reasoning and the danger of anchoring on a specific hypothesis (GPU 6 is the weak link) without considering alternative failure modes.

The phantom OOM that moves from one GPU to another is a humbling reminder that in distributed systems, fixing a symptom is not the same as fixing the cause. The assistant's journey from this diagnostic moment to the eventual torch version rollback is a testament to the iterative nature of ML infrastructure debugging—and to the importance of knowing when to stop tuning parameters and start addressing root causes.