The Art of the Three-Word Experiment: Deconstructing "try tp4 ep2"

In the midst of an intense benchmarking session spanning multiple parallelism strategies for deploying the Kimi K2.6 mixture-of-experts (MoE) model across eight NVIDIA RTX PRO 6000 Blackwell GPUs, the user issues a message that is remarkable for its brevity and depth. The message, found at <msg id=11514>, consists of exactly three words:

try tp4 ep2

This is not a casual remark. It is a precisely targeted experimental directive that encapsulates hours of accumulated context, a sophisticated understanding of distributed inference tradeoffs, and a deliberate pivot in the exploration of the parallelism design space. To the uninitiated, "tp4 ep2" looks like jargon. To anyone who has followed the conversation, it represents a hypothesis about how to squeeze more performance from a complex hardware-software stack — a hypothesis that would immediately be tested, and would immediately fail, but whose failure would produce valuable insight.

The Context That Makes Three Words Meaningful

To understand why this message was written, one must reconstruct the state of the conversation at <msg id=11514>. The assistant had just completed a systematic benchmark of three pure parallelism strategies for running Kimi K2.6 on eight GPUs connected via PCIe (no NVLink interconnect). The results, presented in <msg id=11513>, were stark:

| Config | C=1 tok/s | C=32 | C=64 | Peak | |--------|-----------|------|------|------| | TP8 (pure tensor parallelism) | 26.3 | 577.8 | 693.1 | ~808 | | PP8 (pure pipeline parallelism) | 36.9 | 250.9 | 396.0 | ~396 | | EP8 (pure expert parallelism) | 65.3 | 546.0 | 795.0 | ~961 |

Expert parallelism had won decisively. EP8 delivered 2.5× the single-request throughput of TP8 (65 vs 26 tok/s) and scaled to nearly 1000 tok/s aggregate. The reason was clear: on PCIe, the All-to-All communication pattern used by expert parallelism for MoE routing was far cheaper than the AllReduce broadcast required by tensor parallelism for every layer. The assistant, having presented this data, asked an open-ended question: "Want me to use this config for the data generation, or push further with tuning?"

The user's response — "try tp4 ep2" — rejects both options. It says, in effect: don't settle on EP8 yet, and don't tune the same configuration. Instead, explore a different point in the design space.

The Hypothesis Embedded in Three Tokens

TP4 EP2 is a hybrid parallelism configuration. With eight GPUs total, it divides the hardware into two expert-parallel groups of four GPUs each (EP2), and within each group, applies 4-way tensor parallelism (TP4). The total parallelism is 4 × 2 = 8.

The user's reasoning, though unstated, is legible to anyone who understands the architecture. EP8 had worked well because it eliminated AllReduce on the MoE layers — the dominant computational cost in a model with 384 experts. But EP8 came with a cost: attention weights were fully replicated across all eight GPUs. Each GPU held the complete attention parameters, consuming memory that could otherwise be used for larger batch sizes or longer contexts. TP4 EP2 would shard the attention weights across four GPUs (each holding ¼ of the attention), reducing per-GPU memory for the attention layers, while still splitting the 384 experts into just two groups of 192 each.

The hypothesis was that this hybrid might offer the best of both worlds: reduced attention memory overhead from TP4, combined with the communication efficiency of expert parallelism for the MoE layers. Perhaps the All-to-All dispatch across only two EP groups would be fast enough on PCIe, while the TP4 AllReduce within each group of four GPUs would be cheaper than TP8's AllReduce across all eight. The user was probing whether the optimal configuration lies not at the extremes of pure parallelism, but somewhere in the hybrid middle.

What the Message Assumes — and Gets Wrong

The user made several assumptions in issuing this directive. First, that SGLang supports the TP4 EP2 combination. This was correct — the assistant had confirmed in <msg id=11508> and <msg id=11509> that --tp-size and --ep-size are both accepted parameters. Second, that the configuration would fit in GPU memory. This assumption proved incorrect.

The assistant's response in <msg id=11515> immediately deployed the configuration, creating a systemd service file with --tp-size 4 --ep-size 2 and starting the server. But within 15 seconds, the service failed with a CUDA out-of-memory error:

torch.OutOfMemoryError: CUDA out of memory. Tried to allocate 672.00 MiB. 
GPU 2 has a total capacity of 94.97 GiB of which 136.94 MiB is free. 
Including non-PyTorch memory, this process has 94.83 GiB memory in use.

The assistant's analysis in <msg id=11517> diagnosed the problem: TP4 meant each GPU held ¼ of the attention weights instead of ⅛ (as in EP8's TP1), doubling the attention memory footprint. Combined with EP2 (each EP group holding 192 experts, further sharded by TP4 to 48 experts per GPU — the same as EP8), the total per-GPU memory was actually higher than EP8. The 94.83 GiB consumed on GPU 2 left only 137 MiB free out of 94.97 GiB total — a fatal overcommit.

The user's implicit assumption that TP4 EP2 would reduce memory was wrong in the specific context of this model and these GPUs. The attention weights, while not the dominant parameter count in an MoE model, were large enough that doubling their per-GPU footprint pushed the system past the memory wall. This is a valuable negative result: it reveals that for Kimi K2.6 on 94 GB GPUs, the attention memory overhead of TP4 (vs TP1) is not free, and the optimal parallelism strategy must respect the hard constraint of per-GPU memory capacity.

The Knowledge Flow: Input and Output

The input knowledge required to parse "try tp4 ep2" is substantial. A reader must understand: what tensor parallelism and expert parallelism are; how they compose (TP4 × EP2 = 8 GPUs); why one might combine them for an MoE model; the PCIe bandwidth constraints that made EP8 outperform TP8; and the memory tradeoffs of attention replication vs sharding. Without this background, the message is opaque.

The output knowledge created by this message is equally significant. The immediate output was the discovery that TP4 EP2 OOMs on this hardware — a concrete constraint that narrows the feasible design space. But the broader output was a refined understanding of the memory-communication tradeoff surface. The failure led directly to the next experiment: TP2 EP4, which the assistant deployed in the same message (<msg id=11517>), reasoning that TP2 would reduce attention memory to ½ per GPU (still more than EP8's ⅛, but less than TP4's ¼) while EP4 would distribute experts across four groups. This iterative exploration, triggered by the user's three-word message, systematically maps the Pareto frontier of parallelism configurations.

The Thinking Process: Brevity as Expertise

The user's choice to write only "try tp4 ep2" — without explanation, justification, or even a question mark — reveals a specific mode of collaboration. This is not a novice asking for guidance or a manager issuing vague direction. It is an expert communicating with an expert assistant in a shared mental model. The user knows that the assistant will understand the reasoning, execute the experiment, and report results. The brevity is a signal of trust and efficiency: we both know why this is interesting; just run it.

This mode of interaction is characteristic of the most productive phases of the conversation. The assistant had just presented a comprehensive benchmark with a clear winner (EP8). A less sophisticated collaborator might have accepted that result and moved on. The user's instinct to probe the hybrid space — to ask "what about the combination you didn't test?" — demonstrates the kind of systematic exploration that distinguishes thorough optimization from surface-level tuning.

The message also reveals the user's mental model of the parallelism space. They are not exploring randomly. TP4 EP2 is a specific, principled choice: it tests whether reducing attention replication (via TP4) while keeping expert parallelism (via EP2) can outperform pure EP8. The fact that it OOMs is not a failure of reasoning — it is an empirical result that constrains future exploration. The user's hypothesis was worth testing, and the test produced clear, actionable data.

Conclusion

"try tp4 ep2" is a masterclass in concise, high-leverage communication in a technical collaboration. In three words, the user: rejected the assistant's proposal to settle on EP8; specified an alternative configuration with precise parameters; implicitly communicated a hypothesis about memory-communication tradeoffs; and initiated an experiment that would produce a valuable negative result. The message's brevity is not a lack of content but a concentration of it — every word carries the weight of the entire preceding conversation.

The failure of TP4 EP2 was not a dead end. It was a signpost that guided the exploration toward TP2 EP4, and eventually toward the deeper understanding that on PCIe-connected Blackwell GPUs, the optimal parallelism strategy for Kimi K2.6 is one that minimizes per-GPU memory while maximizing the benefits of expert parallelism's communication pattern. The user's three-word message, in its precision and timing, exemplifies how the most impactful interventions in a technical conversation are often the shortest ones.