"Not oomed but crashed?" — A Diagnostic Pivot That Uncovered a Triton Shared-Memory Bug

Subject Message: [user] Not oomed but crashed?

Introduction

In the middle of a high-stakes EAGLE-3 draft model training run on four NVIDIA RTX PRO 6000 Blackwell GPUs, a single four-word user message — "Not oomed but crashed?" — served as a critical diagnostic pivot. This message, sent at a moment when the assistant was blindly waiting for a training job to initialize, reframed the failure mode and led directly to the discovery of a subtle Triton shared-memory limitation on the SM120 architecture. The message is a masterclass in concise, high-signal debugging communication: it distinguishes between two superficially similar failure modes (OOM vs. crash) and forces a re-examination of assumptions about where the bottleneck truly lies.

The Context: A Training Pipeline Pushed to Its Limits

To understand why this message matters, we must trace the events that led up to it. The team was training an EAGLE-3 speculative decoding drafter for the Kimi-K2.5 model on a machine with 8 GPUs (though training used 4). The training script used a packed-sequence approach: multiple training samples were concatenated into a single long sequence with block-diagonal attention masking, allowing the GPU to process more tokens per step.

The assistant had been iterating on the --max-seq-len parameter to improve GPU utilization. At max_seq_len=4096, the GPUs drew only 250W out of a 600W power budget — severely underutilized despite showing 100% GPU-Util (a metric that measures occupancy, not throughput). The assistant increased the sequence length to 8192, which boosted power draw to 412-476W and VRAM usage to 39.8 GB. But the number of batches per epoch remained high at 35,446, because each packed sequence held roughly only one sample.

The natural next step was to push further. The assistant tried max_seq_len=32768 — the user reported an OOM. Then max_seq_len=24576 — another OOM. Each time, the assistant killed the process, cleared the output directory, and relaunched with a smaller value. The pattern was clear: the assistant assumed the failure was GPU VRAM exhaustion, and the fix was to reduce the sequence length until it fit.

Then came the attempt at max_seq_len=16384. The assistant launched the job and issued a sleep 180 command to wait three minutes before checking the logs. But the user, monitoring the system in real time, saw something different: the job didn't OOM — it crashed.

Why "Not OOM but Crashed" Is a Fundamental Distinction

The user's message draws a sharp line between two failure modes that are easy to conflate but have entirely different root causes:

1. OOM (Out of Memory): GPU VRAM exhaustion. The CUDA driver returns a clear error: CUDA out of memory. The fix is typically to reduce batch size, sequence length, or model size — anything that reduces the memory footprint.
2. Crash: A non-memory failure. This could be a kernel compilation failure, a driver bug, a CUDA error, a Triton autotuning issue, or any number of software-level problems. The fix is entirely different and often requires understanding the specific error message. By explicitly distinguishing these two outcomes, the user told the assistant: "Your assumption that this is a memory problem is wrong. Look elsewhere." This was a crucial intervention because the assistant had been operating under the implicit assumption that all failures at high sequence lengths were VRAM OOMs. The previous two attempts (32768 and 24576) had indeed OOMed, reinforcing this assumption. The user's message broke that pattern.

The Assumptions Embedded in the Assistant's Approach

Before this message, the assistant had been making several assumptions:

• Assumption 1: Failure at high sequence length is VRAM OOM. This was reasonable given the two prior OOMs, but it turned out to be incorrect for the 16384 case.
• Assumption 2: Reducing max_seq_len monotonically will eventually find a working configuration. This is true for VRAM OOM (where memory usage scales roughly linearly with sequence length), but not for the actual failure mode (a Triton kernel constraint that has a hard cutoff).
• Assumption 3: The sleep 180 command would return useful status. The assistant expected to see either a running job (with GPU power and batch counts) or an OOM error. It did not anticipate a crash that looked different from OOM.
• Assumption 4: The training script's max_seq_len parameter is the only knob that matters for GPU utilization. In reality, the Triton compiler's autotuning was the limiting factor, and that's controlled by entirely different mechanisms (compiler flags, block sizes, num_stages).

Input Knowledge Required to Understand This Message

To interpret "Not oomed but crashed?" correctly, the reader (and the assistant) needed:

1. Knowledge of the training setup: That the script uses packed sequences with max_seq_len, that the GPUs are RTX PRO 6000 Blackwell (SM120 architecture), and that the training uses torch.compile for flex_attention.
2. Knowledge of the previous failures: That 32768 and 24576 both produced OOM errors, establishing a pattern.
3. Knowledge of the system behavior: The ability to distinguish between a CUDA OOM (which produces a specific error message) and a process crash (which might produce a different error or no error at all).
4. Knowledge of Triton compilation: The user implicitly understood that a crash without OOM could be a kernel compilation issue — a sophisticated insight that most operators would not have.

Output Knowledge Created

This message triggered a chain of investigation that produced several important insights:

1. The actual error was a Triton shared-memory OOM: The assistant investigated (in [msg 4278]) and found RuntimeError: No valid triton configs. OutOfMemoryError: out of resource: triton_per_fused__to_copy_add_div_expand_mul_pow_sum_view_1 Required: 163912 Hardware limit:101376. This is a Triton kernel compilation failure: the RMSNorm kernel's reduction dimension at seq_len=16384 requires more shared memory (163,912 bytes) than the SM120 architecture provides (101,376 bytes per block).
2. This is not a VRAM issue: The failure is in the Triton compiler's autotuning, not in GPU memory allocation. The fix is different: reduce block sizes, adjust num_stages, or use a different sequence length that avoids the problematic reduction dimension.
3. The boundary is between 12288 and 16384: The assistant later determined that 12288 works (by trying it in [msg 4280]), establishing the practical limit for this particular kernel on SM120 hardware.
4. The fix requires compiler-level intervention: Simply reducing max_seq_len is a workaround, but the real solution involves Triton compiler flags like TRITON_MAX_BLOCK or disabling max_autotune for specific kernels.

The Thinking Process Revealed

The user's message reveals a sophisticated mental model of GPU failures. The user was monitoring the system in real time and saw the process terminate without the characteristic CUDA OOM error message. Rather than simply reporting "it failed again," the user made a differential diagnosis: "not OOM, but crashed." This distinction shows that the user:

• Was watching the error output, not just the process exit code
• Knew what an OOM error looks like and recognized its absence
• Understood that a crash without OOM implies a different class of problem
• Communicated this distinction concisely to guide the assistant's debugging The assistant's response (in <msg id=4277-4280>) shows it immediately understood the significance. It went looking for tracebacks rather than assuming memory pressure, found the Triton shared-memory error, and correctly identified the root cause within minutes. The user's four-word message saved what could have been hours of fruitless max_seq_len tuning.

Conclusion

"Not oomed but crashed?" is a deceptively simple message that carries enormous diagnostic weight. In four words, it reframes a debugging problem, breaks a false pattern, and points toward a completely different root cause. The message exemplifies the kind of high-bandwidth, low-latency communication that makes human-AI collaboration effective: the user observed a subtle difference in failure mode, recognized its significance, and communicated it with surgical precision. The result was the rapid identification of a Triton shared-memory limitation on SM120 — a bug that would have been extremely difficult to find through trial-and-error parameter tuning alone.