The Art of the Pivot: Diagnosing an OOM and Reconfiguring Parallelism for Kimi K2.6

Introduction

In the high-stakes world of large language model deployment on multi-GPU clusters, few events are as simultaneously frustrating and informative as an out-of-memory (OOM) error. An OOM is not merely a failure—it is a signal, a diagnostic message from the hardware that the current strategy for distributing model weights across GPUs is mismatched to the available resources. How an engineer (or, in this case, an AI assistant) interprets that signal and decides on a next course of action reveals a great deal about their understanding of the underlying architecture.

Message [msg 11517] captures exactly such a moment. It is a pivot message—a single turn in a conversation where the assistant receives the news that a configuration it just deployed (TP4 EP2) has crashed with a CUDA OOM error, reasons through why it failed, and immediately deploys an alternative configuration (TP2 EP4). The message is deceptively short in its visible output—a single line reading "Started TP2 EP4"—but the reasoning section preceding it contains a dense, self-correcting chain of analysis about memory distribution across tensor parallelism and expert parallelism groups. This article unpacks that reasoning, examining the assumptions, the knowledge required, the decisions made, and the broader significance of this single message within the months-long effort to deploy Kimi K2.6 with speculative decoding on Blackwell GPUs.

The Conversation Leading Up to the Message

To understand [msg 11517], we must first understand the benchmarking campaign that preceded it. The user and assistant had been systematically exploring parallelism strategies for deploying Kimi K2.6—a Mixture-of-Experts (MoE) model with 384 experts—across 8× RTX PRO 6000 Blackwell GPUs connected via PCIe (not NVLink). This is a challenging environment because PCIe bandwidth is a severe bottleneck for strategies that require frequent all-reduce communication across GPUs.

The journey began with TP8 (tensor parallelism across all 8 GPUs), which achieved 26 tok/s at C=1 but scaled to ~808 tok/s aggregate. The user then requested PP8 (pipeline parallelism), which improved single-request latency to 37 tok/s but disappointed at high concurrency (~396 tok/s) due to pipeline bubbles. The user's intuition was that expert parallelism (EP) would outperform both because it avoids all-reduce on the MoE layers—instead of broadcasting activations across all GPUs, EP distributes experts across GPUs and uses all-to-all communication to route tokens to the GPU holding the needed expert.

The assistant deployed EP8 (expert parallelism across all 8 GPUs, with TP1 for attention) and the results were dramatic: 65 tok/s at C=1 (2.5× TP8) and ~960 tok/s aggregate. The user, clearly experienced with distributed inference, then asked the assistant to "try tp4 ep2" ([msg 11514])—a hybrid configuration that combines tensor parallelism and expert parallelism.

The Failed Configuration: TP4 EP2

The assistant deployed TP4 EP2 in [msg 11515]. The configuration used --tp-size 4 --ep-size 2, which with 8 GPUs means: 2 expert-parallel groups, each containing 4 GPUs that share tensor parallelism. Within each EP group, the attention weights are split across 4 GPUs (TP4), and the 384 experts are split into 2 groups of 192 experts each, with each group further split by TP4 across its 4 GPUs.

The result, reported in [msg 11516], was an OOM on GPU 2:

torch.OutOfMemoryError: CUDA out of memory. Tried to allocate 672.00 MiB. 
GPU 2 has a total capacity of 94.97 GiB of which 136.94 MiB is free. 
Including non-PyTorch memory, this process has 94.83 GiB memory in use.

The numbers are striking: 94.83 GiB out of 94.97 GiB capacity is in use, with only 137 MiB free. The process had exhausted the GPU's memory, leaving no room even for a modest 672 MiB allocation. This is the context that [msg 11517] responds to.

Anatomy of the Subject Message

The message [msg 11517] has three distinct parts:

A reasoning section where the assistant works through the memory distribution math for TP4 EP2, compares it to the working EP8 configuration, and identifies the root cause of the OOM.
A decision to try TP2 EP4 instead, justified by the reasoning.
A bash command that stops the failed service, creates a new systemd unit file for TP2 EP4, and starts it. The visible output is minimal—just "Started TP2 EP4"—but the reasoning section is where the real work happens. Let us examine it in detail.

The Reasoning Process: A Self-Correcting Memory Analysis

The reasoning begins with a clear statement of the configuration:

TP4 EP2 means: 8 GPUs total, divided into 2 EP groups of 4 GPUs each. Within each group, TP4 is used. So each EP group has 4 GPUs with TP4. The model's attention is split across 4 GPUs (TP4). The 384 experts are split into 2 groups of 192 experts each.

This is correct. The assistant then attempts to compute the per-GPU memory footprint:

But OOM on GPU 2 - with TP4 EP2, each GPU needs to hold: - 1/4 of attention weights (TP4) - 192 experts (half of 384) split across 4 GPUs = 48 experts per GPU? No wait...

The "No wait..." is a fascinating moment of self-correction. The assistant realizes that the expert distribution is more complex than it initially thought. It continues:

Actually, with TP4 EP2, both attention and experts are sharded by TP4 within each EP group. Each EP group holds 192 experts, and TP4 splits those across 4 GPUs, so that's 48 experts per GPU. The attention weights are also TP4 sharded, so each GPU holds 1/4 of the attention.

But then the assistant confronts a puzzle: if each GPU holds 48 experts (the same as EP8) and 1/4 of attention weights (vs 1/8 in EP8), why did TP4 EP2 OOM while EP8 worked? The attention difference alone—doubling from 1/8 to 1/4—should not account for 94+ GiB of usage.

Looking at the memory breakdown, with TP4 EP2 each GPU gets 1/4 of attention weights and 48 experts (each expert's weights also split by TP4). But the OOM error suggests something's off with how SGLang implements this combination.

The assistant identifies the key insight:

Comparing to pure EP8, each GPU would hold 1/8 of attention and 48 experts, which is actually less memory per GPU, so the issue might be in how the parallelism is being applied.

This is the critical realization. The assistant suspects that SGLang's implementation of combined TP+EP may not distribute memory as cleanly as the theoretical model suggests. There could be replication of certain buffers, intermediate activations that scale with TP degree, or other implementation-specific overheads.

The assistant then makes a practical decision:

OOM -- TP4 means each GPU holds 1/4 of attention instead of 1/8, doubled memory for that. Let me try TP2 EP4 which keeps attention smaller.

This is a pragmatic engineering decision. Rather than debugging the exact memory allocation pattern of SGLang's TP+EP implementation, the assistant reduces the TP degree from 4 to 2, which halves the attention memory per GPU (from 1/4 to 1/2... wait, actually TP2 means each GPU holds 1/2 of attention weights, which is more per GPU, not less).

Hmm, this is actually a point worth examining. Let me reconsider.

With TP4 EP2: each GPU holds 1/4 of attention weights (TP4 shards attention across 4 GPUs). With TP2 EP4: each GPU holds 1/2 of attention weights (TP2 shards attention across 2 GPUs).

So TP2 EP4 actually means more attention memory per GPU, not less. The assistant's stated reasoning ("keeps attention smaller") appears to be incorrect in the direction of the effect. However, the EP degree changes from 2 to 4, which means each EP group now has 2 GPUs instead of 4, and each group holds 96 experts (384/4) instead of 192. With TP2 within each group, each GPU holds 96/2 = 48 experts—the same number as before.

So the per-GPU memory for TP2 EP4 is:

1/2 of attention weights (more than TP4's 1/4)
48 experts (same as TP4 EP2 and EP8) This should use more memory per GPU, not less. Yet the assistant deploys it and it works (as we can see from subsequent messages in the segment). This suggests that either: 1. The assistant's reasoning about attention memory was wrong but the configuration worked for other reasons (e.g., SGLang's implementation handles TP2 EP4 more efficiently than TP4 EP2) 2. There's something about how SGLang allocates intermediate buffers that makes TP2 EP4 more memory-efficient despite larger attention weights 3. The assistant's assumption that "TP4 doubles attention memory vs TP8" was the wrong comparison—the real issue might have been something else entirely This is an excellent example of a message where the stated reasoning may not perfectly match the actual dynamics, but the engineering outcome (a working configuration) validates the decision pragmatically.

Assumptions and Potential Mistakes

The message contains several assumptions worth examining:

Assumption 1: The OOM is caused by attention weight distribution. The assistant attributes the OOM primarily to TP4 doubling the attention memory per GPU compared to EP8's TP1. While mathematically plausible, the 93.6 GiB usage suggests something beyond a simple doubling of attention weights. The attention layers of K2.6 are relatively small compared to the MoE layers (MLA has compact KV representations), so the attention weight difference between TP1 and TP4 should be on the order of a few GiB, not tens of GiB.

Assumption 2: SGLang distributes memory exactly according to the theoretical model. The assistant's reasoning assumes that TP4 cleanly divides attention weights by 4 and EP2 cleanly divides experts by 2, with no overhead. In practice, SGLang may have additional memory costs for intermediate buffers, activation memory, KV cache, and implementation-specific replication that scale differently with TP and EP degrees.

Assumption 3: TP2 EP4 will fit where TP4 EP2 did not. As noted above, the per-GPU attention memory is actually larger with TP2 (1/2) than with TP4 (1/4). The assistant's stated justification is internally inconsistent. However, the configuration may still work because of other factors—for example, EP4 means each EP group has only 2 GPUs, which may reduce communication buffer requirements, or the KV cache may be distributed differently.

Potential mistake: The reasoning about "keeps attention smaller" is directionally wrong. This is the most notable error in the reasoning. The assistant says "TP4 means each GPU holds 1/4 of attention instead of 1/8, doubled memory for that. Let me try TP2 EP4 which keeps attention smaller." But TP2 means each GPU holds 1/2 of attention—larger than TP4's 1/4. The assistant seems to be comparing TP2 to TP8 (1/2 vs 1/8) rather than TP2 to TP4 (1/2 vs 1/4), or perhaps conflating two different comparisons.

This mistake is worth dwelling on because it reveals something important about how the assistant thinks. The reasoning is happening under time pressure—the user is waiting, the OOM just happened, and the assistant needs to respond quickly. In such conditions, even sophisticated reasoning systems make arithmetic errors. What matters is that the overall engineering judgment (try a different TP/EP combination) was sound, and the configuration ultimately worked.

Input Knowledge Required

To understand this message, a reader needs knowledge spanning multiple domains:

Mixture-of-Experts architecture: Understanding that K2.6 has 384 experts, that only a subset (typically 8) are activated per token, and that expert parallelism distributes these experts across GPUs to avoid all-reduce.

Tensor parallelism vs. expert parallelism: TP shards individual weight matrices across GPUs (requiring all-reduce after every layer), while EP distributes entire experts across GPUs (requiring all-to-all routing of tokens). These have different communication patterns and memory footprints.

SGLang's parallelism configuration: Knowledge of the --tp-size and --ep-size flags, how they interact, and how SGLang maps them to GPU groups. The assistant's reasoning about "2 EP groups of 4 GPUs each" reflects an understanding of SGLang's group-based parallelism model.

GPU memory budgeting: Understanding that 94.97 GiB is the total capacity of an RTX PRO 6000 Blackwell GPU, and that 93.6 GiB of PyTorch allocations plus overhead leaves essentially no headroom.

Systemd and service management: The assistant creates a systemd unit file, reloads the daemon, and starts the service—standard Linux service management that requires familiarity with systemd conventions.

CUDA and NCCL environment variables: The service file includes NCCL configuration (NCCL_IB_DISABLE=1, NCCL_P2P_LEVEL=5, NCCL_ALGO=Ring) that reflects an understanding of the PCIe-only interconnect (no InfiniBand) and the need to tune NCCL for this topology.

Output Knowledge Created

The message produces several concrete outputs:

A diagnostic conclusion: TP4 EP2 OOMs because the per-GPU memory footprint is too large, likely due to the combination of TP4 attention sharding and EP2 expert distribution interacting poorly with SGLang's memory allocator.
A new configuration: TP2 EP4, which the assistant hypothesizes will fit within the 95 GiB per-GPU budget.
A systemd unit file: /etc/systemd/system/sglang-k26-tp2ep4.service with the new configuration, including all necessary environment variables and launch arguments.
A deployed service: The service is started and (as subsequent messages show) becomes ready and operational.
An implicit lesson: The failure of TP4 EP2 and success of TP2 EP4 provides empirical data about SGLang's memory behavior with combined TP+EP configurations on Blackwell GPUs.

The Broader Significance

This message is significant beyond its immediate context for several reasons.

First, it demonstrates the iterative nature of distributed inference deployment. The assistant and user are not following a predetermined plan—they are exploring a configuration space, learning from failures, and adapting. The OOM is not a setback but information that guides the next attempt.

Second, it reveals the complexity of the TP/EP interaction. The naive assumption would be that TP4 EP2 and TP2 EP4 have similar memory footprints (both distribute 384 experts across 8 GPUs, both shard attention to some degree). The fact that one OOMs and the other works shows that the distribution is not simply additive—there are nonlinear interactions between TP degree, EP degree, and memory consumption.

Third, it highlights the importance of the reasoning process itself. The assistant's reasoning is not perfectly correct—the arithmetic about attention memory appears to be wrong—but it is productive. It generates a hypothesis, tests it, and the hypothesis works. In engineering, this is often more valuable than perfect theoretical analysis. The ability to form a plausible hypothesis, act on it, and learn from the result is the essence of debugging.

Fourth, this message sits at a critical juncture in the broader benchmarking campaign. The EP8 results (65 tok/s at C=1) had already validated the user's intuition that expert parallelism avoids PCIe bottlenecks. The failed TP4 EP2 and successful TP2 EP4 would eventually lead to EP4 (TP2 EP4 with different configuration), which achieved peak aggregate throughput of ~1531 tok/s—the best result of the entire parallelism exploration. The pivot in [msg 11517] is a stepping stone to that outcome.

Conclusion

Message [msg 11517] is a study in practical engineering reasoning under uncertainty. Faced with an OOM error on a complex distributed configuration, the assistant works through the memory distribution math, identifies a plausible root cause (excessive attention memory per GPU from TP4), proposes a remedy (reduce TP degree), and deploys it—all within a single message turn.

The reasoning is not flawless. The arithmetic about attention memory appears to compare against the wrong baseline. But the decision is sound, the configuration works, and the broader exploration continues. This is the reality of deploying large models on cutting-edge hardware: the theory is a guide, but the practice requires rapid iteration, pragmatic judgment, and the willingness to be wrong and correct course.

In the end, the message is a testament to the value of the reasoning trace itself. The assistant's internal monologue—complete with its self-corrections, its "No wait..." moment, and its imperfect but productive analysis—is far more valuable than the single line of output it produces. It is a window into how an AI system navigates the treacherous waters between theoretical understanding and practical deployment, making decisions that are good enough to move forward, even when they are not perfectly correct.