The Fifteen-Second Failure: When TP4 EP2 Meets GPU Memory Limits

In the middle of an intensive parallelism exploration for the Kimi K2.6 model on 8× RTX PRO 6000 Blackwell GPUs, a single message arrives that contains no reasoning, no analysis, and no triumphant benchmark numbers. It is a polling script—a routine readiness check—and its output is devastatingly brief: after just fifteen seconds, the service has already failed. The journal reveals a CUDA out-of-memory error on GPU 2, with 94.83 GiB already consumed out of 94.97 GiB total. This message ([msg 11516]) marks the hard boundary where theory meets hardware constraints, and it speaks more through its silence than any verbose explanation could.

The Message

The assistant dispatches a bash script that polls every 15 seconds for the TP4 EP2 service status:

for i in $(seq 1 90); do
    sleep 15
    st=$(ssh -o ConnectTimeout=5 root@10.1.2.200 "systemctl is-active sglang-k26-tp4ep2.service" 2>&1)
    if [ "$st" = "failed" ]; then
        echo "[$((i*15))s] FAILED"
        ssh -o ConnectTimeout=5 root@10.1.2.200 "journalctl -u sglang-k26-tp4ep2.service --no-pager -n 20 | grep -E 'Error|error|FAILED|assert|Traceback' | tail -8" 2>&1
        break
    fi
    health=$(curl -s --max-time 5 "http://10.1.2.200:30001/v1/models" 2>/dev/null)
    if echo "$health" | grep -q '"id"' 2>/dev/null; then
        echo "[$((i*15))s] K2.6 TP4 EP2 READY!"
        break
    fi
    if [ $((i % 6)) -eq 0 ]; then echo "[$((i*15))s] loading..."; fi
done

The output arrives in the very first iteration:

[15s] FAILED
May 26 00:06:33 dflash-train python[92314]: torch.OutOfMemoryError: CUDA out of memory. Tried to allocate 672.00 MiB. GPU 2 has a total capacity of 94.97 GiB of which 136.94 MiB is free. Including non-PyTorch memory, this process has 94.83 GiB memory in use. Of the allocated memory 93.62 GiB is allocated by PyTorch, and 273.15 MiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True to avoid ...

There is no assistant commentary, no "hmm, that's unexpected," no fallback plan. The message is purely the tool call and its result. The failure is allowed to speak for itself.

The Path to TP4 EP2

To understand why this message exists, one must trace the parallelism journey that preceded it. The conversation had been systematically exploring how to best deploy K2.6—a Mixture-of-Experts model with 384 experts and Multi-head Latent Attention (MLA)—across eight PCIe-connected Blackwell GPUs.

The exploration began with TP8 (tensor parallelism over 8 GPUs), which achieved 26.3 tok/s at concurrency 1 but required expensive AllReduce communication across PCIe for every token, capping aggregate throughput at roughly 808 tok/s ([msg 11513]). Next came PP8 (pipeline parallelism over 8 stages), which improved single-request latency to 36.9 tok/s by eliminating per-token synchronization, but suffered from pipeline bubbles that limited aggregate throughput to roughly 396 tok/s ([msg 11506]). Then the user directed the assistant to try EP8 (expert parallelism over 8 GPUs), and the results were transformative: 65.3 tok/s at C=1—a 2.5× improvement over TP8—with aggregate throughput scaling to approximately 961 tok/s ([msg 11512], [msg 11513]).

The user, seeing EP8's dominance, then asked the assistant to try a hybrid configuration: TP4 EP2 ([msg 11514]). The intuition was plausible: by combining tensor parallelism (sharding attention and non-expert layers across 4 GPUs) with expert parallelism (distributing experts across 2 groups), perhaps the model could achieve even better throughput by balancing communication overhead against compute efficiency. The assistant dutifully created the service configuration and started it ([msg 11515]). Message 11516 is the readiness check that follows.

Anatomy of a Silent Failure

The polling script itself is a well-worn pattern in this conversation—nearly identical versions appear in messages 11500, 11511, and elsewhere. It checks two conditions: whether systemctl reports the service as "failed," and whether the HTTP health endpoint returns a valid model ID. The script is designed for patience, running up to 90 iterations (22.5 minutes total) with 15-second intervals.

But the failure is not patient. It arrives at the very first check, just 15 seconds after the service started. This immediacy is itself informative: the OOM occurs during model loading, not during inference. The model's weights cannot be allocated within the available GPU memory, and the process crashes before it ever serves a single request.

The script's error-handling branch is also revealing. Upon detecting failure, it reaches back to the remote host to extract the last 20 lines of the journal, filtered for error patterns. This diagnostic reflex—built into the polling script itself—demonstrates a workflow where failure is anticipated as a possibility but handled as an exception rather than the expected outcome. The assistant expected the service to start successfully, just as EP8 had done minutes earlier.

The Memory Arithmetic

The OOM error on GPU 2 provides a detailed snapshot that explains exactly why TP4 EP2 fails where EP8 succeeded. The key numbers are stark:

• Total capacity: 94.97 GiB
• Memory in use: 94.83 GiB
• Free memory: 136.94 MiB
• Failed allocation: 672.00 MiB The GPU is nearly full—99.85% utilized before the failed allocation. The 672 MiB request is the straw that breaks the camel's back, but the real story is in how the memory was consumed in the first place. The critical difference between EP8 and TP4 EP2 lies in how expert weights are distributed. K2.6 is a MoE model with 384 experts, and the expert parameters dominate the model's memory footprint. Under EP8 (as configured with --tp-size 8 --ep-size 8), each GPU holds one-eighth of the total parameters: one-eighth of the attention weights (sharded via TP) and one-eighth of the expert weights (sharded via EP). Under TP4 EP2, each GPU holds one-quarter of the attention weights (sharded via TP across 4 GPUs) and one-half of the expert weights (sharded via EP across 2 groups). The expert weights are roughly 4× larger per GPU under TP4 EP2 compared to EP8. For a model where experts constitute the vast majority of parameters—typical for large MoE models—this 4× increase in per-GPU expert memory is catastrophic. The 94.83 GiB already consumed represents the model's weights, KV cache reservations, and intermediate buffers all competing for space. The additional 672 MiB that triggered the OOM was likely a buffer allocation during the final stages of model initialization, and there was simply no room left.

What the Error Tells Us About the Hardware

The error message also reveals something about the GPUs themselves. The RTX PRO 6000 Blackwell cards have 96 GB of VRAM, and the error reports 94.97 GiB total capacity (the slight discrepancy is typical of memory reporting conventions). The fact that 94.83 GiB is already in use means the model loading process consumed virtually every available byte.

This near-exact fit is noteworthy. It suggests that EP8 was operating well within memory margins (each GPU held only 1/8 of the model), while TP4 EP2 pushed each GPU to its absolute limit. The 136.94 MiB of remaining free memory is essentially zero in the context of deep learning workloads, where activation tensors, temporary buffers, and framework overhead routinely require hundreds of megabytes.

The error also mentions that 273.15 MiB is "reserved by PyTorch but unallocated"—memory held in PyTorch's caching allocator for reuse. The suggestion to enable expandable_segments:True would not have helped here; the GPU was genuinely full of model weights, not suffering from fragmentation.

Implications for the Parallelism Search

This message serves as a critical data point in the broader parallelism exploration. It establishes that TP4 EP2 is not feasible on 8× 96 GB GPUs for K2.6, regardless of its theoretical performance advantages. The memory ceiling is absolute.

The failure also validates, indirectly, why EP8 worked so well: by distributing both attention and expert weights maximally across all 8 GPUs, EP8 minimized per-GPU memory pressure while simultaneously eliminating the AllReduce bottleneck on MoE layers that had plagued TP8. The user's intuition that expert parallelism avoids PCIe AllReduce bottlenecks was correct ([msg 11513]), and EP8's memory efficiency is an additional, unplanned benefit.

For the assistant, this message represents a moment where the exploration hits a wall. There is no graceful degradation, no partial loading, no fallback to a smaller configuration. The service simply fails, and the polling script dutifully reports the failure. The next message in the conversation will need to chart a new course—perhaps returning to EP8, perhaps exploring EP4 with TP2, or perhaps accepting that the optimal configuration has already been found.

Conclusion

Message 11516 is, on its surface, the most mundane of artifacts: a failed service start captured by a polling script. But within its fifteen-second lifespan, it encapsulates the fundamental tension of distributed inference: the tradeoff between parallelism strategy and memory capacity. The TP4 EP2 configuration was a reasonable hypothesis, born from the success of EP8 and the desire to explore the design space. Its failure is not a mistake but a discovery—a precise measurement of the hardware's limits. In a conversation filled with benchmark numbers and performance comparisons, this single line—[15s] FAILED—may be the most informative message of all.