The Hybrid Parallelism Gambit: Deploying TP4 EP2 for Kimi K2.6 on PCIe Blackwell GPUs

Introduction

In the high-stakes world of large language model deployment, few decisions carry as much weight as the choice of parallelism strategy. For Mixture-of-Experts (MoE) models like Kimi K2.6—a massive 384-expert architecture with Multi-head Latent Attention (MLA)—the parallelism configuration determines whether a cluster of eight NVIDIA RTX PRO 6000 Blackwell GPUs delivers 26 tokens per second or 961. The message at the center of this article, <msg id=11515>, captures a pivotal moment in a systematic exploration of these tradeoffs: the deployment of a TP4 EP2 hybrid parallelism configuration, bridging tensor parallelism and expert parallelism in a single inference server.

While the message itself is deceptively simple—a bash command that stops a running service, writes a systemd unit file, and starts a new one—it represents a carefully reasoned hypothesis about how to balance the competing communication and memory demands of a modern MoE transformer. The story of this message is the story of what happens when theory meets hardware constraints, and why the optimal parallelism strategy is rarely the purest one.

The Message: Deploying TP4 EP2

The subject message is the assistant's direct response to the user's succinct instruction: "try tp4 ep2" (<msg id=11514>). Without hesitation or visible reasoning, the assistant executes the command:

ssh -o ConnectTimeout=5 root@10.1.2.200 "
systemctl stop sglang-k26-ep8.service 2>/dev/null

cat > /etc/systemd/system/sglang-k26-tp4ep2.service << 'EOF'
[Unit]
Description=K2.6 TP4 EP2
After=network.target

[Service]
Type=simple
Environment=CUDA_VISIBLE_DEVICES=0,1,2,3,4,5,6,7
Environment=PATH=/usr/local/cuda-13.0/bin:/root/venv_sglang211/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
Environment=LD_LIBRARY_PATH=/root/venv_sglang211/lib/python3.12/site-packages/nvidia/cu13/lib:/root/venv_sglang211/lib/python3.12/site-packages/nvidia/cuda_runtime/lib:/usr/local/cuda-13.0/lib64
Environment=NCCL_IB_DISABLE=1
Environment=NCCL_P2P_LEVEL=5
Environment=NCCL_ALGO=Ring
Environment=OMP_NUM_THREADS=8
Environment=CUDA_HOME=/usr/local/cuda-13.0
ExecStart=/root/venv_sglang211/bin/python -m sglang.launch_server --model-path /root/models/Kimi-K2.6 --port 30001 --host 0.0.0.0 --tp-size 4 --ep-size 2 --mem-fraction-static 0.88 --context-length 32768 --max-running-requests 64 --attention-backend triton --trust-remote-code --grammar-backend none
Restart=no
StandardOutput=journal
StandardError=journal

[Install]
WantedBy=multi-user.target
EOF

systemctl daemon-reload
systemctl start sglang-k26-tp4ep2.service
echo 'Started TP4 EP2'
" 2>&1

The output is a single line: Started TP4 EP2. The message contains no analysis, no reasoning, no commentary—just action. This terseness is itself meaningful: the assistant has internalized the parallelism exploration context so thoroughly that it can execute the user's request without deliberation. The reasoning happened in the messages that preceded it, and the consequences would unfold in the messages that follow.

The Reasoning Behind Hybrid Parallelism

To understand why TP4 EP2 matters, we must trace the benchmarking arc that led to this moment. The assistant had just completed a systematic comparison of three pure parallelism strategies for Kimi K2.6 on 8× RTX PRO 6000 Blackwell GPUs connected via PCIe:

• TP8 (Tensor Parallelism): All 8 GPUs split every tensor operation via AllReduce. Single-request throughput: 26.3 tok/s. Peak aggregate: ~808 tok/s.
• PP8 (Pipeline Parallelism): The model is split into 8 stages, each on one GPU, with micro-batches flowing through. Single-request: 36.9 tok/s. Peak: ~396 tok/s.
• EP8 (Expert Parallelism): Each GPU holds 48 of the 384 experts locally, with All-to-All routing for tokens needing remote experts. Single-request: 65.3 tok/s. Peak: ~961 tok/s. EP8 was the unambiguous winner, achieving 2.5× the single-request throughput of TP8 and scaling to nearly 1000 tok/s aggregate. The reason was clear: on PCIe, All-to-All communication for expert routing is fundamentally cheaper than the AllReduce that TP requires for every single MoE layer. Each GPU processes its local experts independently and only exchanges token-level results, rather than broadcasting full activation tensors across all 8 GPUs. The user's request to "try tp4 ep2" (<msg id=11514>) was a deliberate probe into hybrid territory. The intuition was that combining tensor parallelism within expert-parallel groups might offer the best of both worlds: TP4 would split the attention weights across 4 GPUs (reducing memory per GPU for the attention mechanism), while EP2 would create two groups of 4 GPUs each, each group holding 192 experts. Within each group, the 192 experts would be further sharded by TP4, meaning each GPU would hold 48 experts—the same count as EP8, but with attention weights sharded across 4 GPUs instead of 8.

Assumptions Embedded in the Deployment

The TP4 EP2 configuration carries several implicit assumptions, some of which would prove incorrect:

Assumption 1: Memory fits. The assistant assumed that TP4 EP2 would use roughly the same memory per GPU as EP8. In EP8, each GPU holds 1/8 of the attention weights and 48 experts. In TP4 EP2, each GPU holds 1/4 of the attention weights (double EP8's share) and 48 experts (same count). The assumption was that the extra attention weight overhead would be manageable within the 95 GB per GPU budget.

Assumption 2: SGLang handles TP+EP combinations correctly. The assistant assumed that SGLang's implementation of combined tensor and expert parallelism would correctly distribute weights and manage communication between the two parallelism domains. This is a non-trivial engineering challenge—the attention layers use TP while the MoE layers use EP, and the two must coordinate seamlessly.

Assumption 3: The communication pattern is beneficial. The reasoning was that TP4 would reduce the AllReduce cost compared to TP8 (fewer GPUs to synchronize) while EP2 would keep expert routing local enough to avoid excessive All-to-All overhead. This assumed a smooth tradeoff curve between the two extremes.

Assumption 4: The PCIe topology supports the hybrid pattern. With NCCL_P2P_LEVEL=5 (NVLink-like P2P) and NCCL_IB_DISABLE=1 (no InfiniBand), the assistant assumed that the PCIe fabric could efficiently handle the mixed communication pattern of TP AllReduce within groups and EP All-to-All across groups.

What Happened Next: The OOM Reveal

The next message (<msg id=11516>) reveals the critical flaw in these assumptions. The service failed almost immediately:

[15s] FAILED
torch.OutOfMemoryError: CUDA out of memory. Tried to allocate 672.00 MiB.
GPU 2 has a total capacity of 94.97 GiB of which 136.94 MiB is free.
Including non-PyTorch memory, this process has 94.83 GiB memory in use.
Of the allocated memory 93.62 GiB is allocated by PyTorch...

GPU 2 had 136.94 MiB free out of 94.97 GiB—the card was nearly full. The TP4 EP2 configuration was consuming dramatically more memory per GPU than EP8 had. The assistant's analysis in the following message (<msg id=11517>) identified the root cause: with TP4 EP2, each GPU holds 1/4 of the attention weights (versus 1/8 in EP8), and the attention weights for Kimi K2.6's MLA are substantial enough that this doubling pushed the memory budget past the breaking point.

This failure reveals a deeper truth about hybrid parallelism: the memory savings from expert parallelism are partially offset by the replication costs of tensor parallelism within each EP group. The assistant quickly pivoted to TP2 EP4, which halves the attention shard to 1/8 per GPU (matching EP8) while keeping 96 experts per GPU—a configuration that would prove more viable.

Knowledge Required and Created

To fully understand this message, one needs:

• Understanding of parallelism taxonomies: The difference between tensor parallelism (splitting individual operations across GPUs), pipeline parallelism (splitting model layers across GPUs), and expert parallelism (distributing MoE experts across GPUs). Each has different communication patterns, memory footprints, and scaling behaviors.
• Knowledge of MoE architecture: Kimi K2.6 uses 384 experts with top-8 routing, meaning each token activates 8 experts per MoE layer. This routing pattern determines how expert parallelism distributes work.
• Familiarity with SGLang's configuration: The --tp-size, --ep-size, --mem-fraction-static, and related flags that control how the inference engine maps model weights to GPUs.
• PCIe vs NVLink awareness: The NCCL environment variables (NCCL_P2P_LEVEL=5, NCCL_ALGO=Ring) reflect a PCIe-only topology where NVLink is unavailable, fundamentally constraining communication bandwidth.
• System administration basics: systemd unit files, environment variable propagation, service lifecycle management. The message creates new knowledge by: 1. Testing a specific hybrid configuration that had not been evaluated before in this hardware/software stack. 2. Demonstrating the memory ceiling for TP+EP combinations on 95 GB GPUs, establishing that TP4 EP2 exceeds the budget while EP8 and (likely) TP2 EP4 do not. 3. Validating the user's intuition that hybrid parallelism is worth exploring, even when the first attempt fails—the failure itself is informative. 4. Creating an artifact (the systemd unit file and the running service) that can be inspected, benchmarked, and compared against other configurations.

The Thinking Process Visible in the Assistant's Reasoning

While the subject message itself contains no explicit reasoning, the assistant's thinking is visible in the surrounding messages. In <msg id=11508>, the assistant had laid out a detailed analysis of EP8's mechanics:

"With EP, each GPU holds 48 experts, and for each token 8 experts get selected. On average only 1 of those 8 is local to any given GPU, so tokens still need routing via All-to-All to reach their target experts. The key question is whether this All-to-All pattern is cheaper than AllReduce on PCIe—and for small batches it should be, since you're only sending tokens to the specific GPUs holding the needed experts rather than broadcasting across the entire system."

This reasoning reveals a sophisticated mental model of communication costs. The assistant understands that AllReduce (used by TP) broadcasts entire activation tensors to all GPUs, while All-to-All (used by EP) only routes token-level data to specific GPUs. On PCIe, where bandwidth is the bottleneck, minimizing the volume of data transferred per operation is paramount.

When the user asks for TP4 EP2, the assistant doesn't re-derive this analysis—it has already internalized it. The swift execution of the command reflects confidence that the hybrid approach is worth testing, even if the assistant doesn't know the exact memory outcome. The failure, when it comes, is not a surprise to be lamented but data to be incorporated into the mental model. The assistant's immediate pivot to TP2 EP4 in <msg id=11517> shows that the OOM result was instantly interpreted and a new hypothesis generated.

Conclusion

Message <msg id=11515> is a study in the economics of attention in AI engineering. The assistant's visible reasoning in surrounding messages shows a deep understanding of communication patterns, memory budgets, and parallelism tradeoffs. The message itself is pure action—the execution of a hypothesis without commentary. And the failure that follows is perhaps more instructive than success would have been: it reveals the precise memory boundary where hybrid parallelism breaks down on 95 GB PCIe GPUs.

For anyone deploying large MoE models, this message and its aftermath teach a crucial lesson: parallelism strategy is not a one-dimensional choice. The optimal configuration depends on the exact interplay of model architecture (attention vs. expert compute ratio), hardware topology (PCIe vs. NVLink), and memory budget. TP4 EP2 failed not because the concept is flawed, but because on this particular hardware with this particular model, the attention weight replication cost exceeded the available memory. On NVLink-connected hardware with faster interconnects, or on GPUs with larger memory, the same configuration might be optimal.

The exploration documented in this message—from TP8 through PP8, EP8, and the failed TP4 EP2—represents a systematic search of the parallelism design space. Each configuration tested reveals new information about the constraints and opportunities of the deployment target. And the assistant's ability to execute, learn, and pivot without hesitation is precisely the kind of adaptive intelligence that makes AI-assisted infrastructure engineering so powerful.