Benchmarking Expert Parallelism: The Turning Point for Kimi K2.6 on Blackwell PCIe

Introduction

In the high-stakes world of large language model deployment, the choice of parallelism strategy can mean the difference between a service that feels responsive and one that crawls. Message [msg 11512] captures a pivotal moment in an extended optimization campaign: the first benchmark of Expert Parallelism (EP8) for the Kimi K2.6 model running on 8× NVIDIA RTX PRO 6000 Blackwell GPUs connected via PCIe. The results were nothing short of transformative—single-request throughput jumped to 65 tok/s, nearly 2.5× the 26 tok/s achieved by Tensor Parallelism (TP8) and 1.8× the 36 tok/s of Pipeline Parallelism (PP8). This message is the empirical validation of a hypothesis that had been building across hours of debugging, configuration wrestling, and architectural reasoning: that for Mixture-of-Experts models on PCIe-bound multi-GPU systems, Expert Parallelism is the correct strategy.

The message itself is deceptively simple—a Python benchmark script and its output. But beneath the surface lies a rich story of hardware constraints, software architecture, and the kind of iterative engineering that defines modern AI infrastructure work. This article unpacks the reasoning, assumptions, decisions, and knowledge embedded in that single message.

The Strategic Context: Why EP8?

To understand why this message was written, we must rewind through the preceding conversation. The assistant had been systematically evaluating parallelism strategies for deploying Kimi K2.6—a massive MoE model with 384 experts—on a server equipped with 8× RTX PRO 6000 Blackwell GPUs. The critical constraint was that these GPUs were connected only via PCIe, not NVLink. This meant that any communication pattern requiring frequent AllReduce across all 8 GPUs would pay a severe bandwidth penalty.

The journey began with TP8 (Tensor Parallelism), where each GPU holds a shard of every layer and communicates via AllReduce at every step. TP8 achieved 26 tok/s at C=1 and peaked at ~808 tok/s aggregate—respectable but limited by PCIe bandwidth saturation on every AllReduce operation across the MoE layers. Next came PP8 (Pipeline Parallelism), which splits layers across GPUs so each GPU computes a different pipeline stage. PP8 improved single-request latency to 36 tok/s (no AllReduce within a stage) but suffered from pipeline bubbles at high concurrency, maxing out at ~396 tok/s.

It was at this point that the user issued a crisp directive in [msg 11507]: "Restart in EP8." The assistant's reasoning in [msg 11508] reveals the thought process: with Expert Parallelism, each of the 8 GPUs would hold a subset of the 384 experts (48 experts per GPU). For each token, only 8 experts are selected via routing, and on average only 1 of those 8 is local to any given GPU. The communication pattern shifts from AllReduce (broadcast to all GPUs) to All-to-All (send tokens only to the GPUs holding the needed experts). On PCIe, this is fundamentally cheaper—you're moving token data rather than full gradient-sized tensors across the bus.

The assistant's reasoning also considered attention: "Attention is replicated (or TP'd with a small TP size) across all GPUs under EP8." The actual deployment command in [msg 11510] used --tp-size 8 --ep-size 8, meaning attention still used TP8 while experts used EP8. This hybrid approach proved to be the winning combination.

The Technical Configuration

The EP8 service was launched with the following SGLang configuration:

--tp-size 8 --ep-size 8 --mem-fraction-static 0.88
--context-length 32768 --max-running-requests 64
--attention-backend triton --trust-remote-code

The --tp-size 8 parameter means the attention layers are sharded across all 8 GPUs using tensor parallelism. The --ep-size 8 parameter means the MoE experts are distributed across all 8 GPUs using expert parallelism. This hybrid configuration is significant: attention still pays the AllReduce cost, but since Kimi K2.6 uses Multi-head Latent Attention (MLA)—a highly compressed attention mechanism—the attention computation is small relative to the MoE feedforward layers. The AllReduce on attention becomes a minor cost compared to the massive savings from eliminating AllReduce on the expert computation.

The environment variables reveal careful tuning for PCIe: NCCL_IB_DISABLE=1 (no InfiniBand), NCCL_P2P_LEVEL=5 (use NVLink where available, but on PCIe-only cards this falls back to P2P over PCIe), and NCCL_ALGO=Ring (ring AllReduce, which is efficient for PCIe topologies). The CUDA_VISIBLE_DEVICES=0,1,2,3,4,5,6,7 ensures all 8 GPUs are visible.

The Benchmark Methodology

The Python script in the message is a well-structured benchmark harness. Let's examine its design choices:

Warmup strategy: Two quick calls with max_tokens=16 ensure the model is fully loaded and any JIT compilation or CUDA graph capture has completed before measurement begins. This is critical for reproducible results—the first inference call often includes one-time overheads.

Single-request measurement: Five diverse prompts test C=1 throughput: a code generation task (quicksort), a science explanation (relativity), a trivial math question, a creative task (haiku), and a code parsing task (JSON parser). The diversity ensures the measurement isn't biased by prompt-specific caching effects. The max_tokens=512 setting produces enough tokens for a stable throughput measurement while keeping each test quick (~8 seconds per prompt at 65 tok/s).

Concurrency sweep: The script tests concurrency levels from 1 to 256 with max_tokens=2048. The use of concurrent.futures.ThreadPoolExecutor with concurrent.futures.wait and a 600-second timeout is robust—it handles stragglers gracefully and reports failures separately. The if fails > 0: break guard stops the sweep early if the system starts dropping requests, preventing a cascade of timeouts.

Prompt cycling: The prompts[i%5] pattern distributes 5 different prompts across concurrent requests, avoiding the pathological case where all requests hit the exact same prompt and benefit from KV cache sharing (which would overstate throughput).

The Results: A Dramatic Improvement

The output tells a clear story:

=== K2.6 EP8 - C=1 (max=512) ===
  65.1 tok/s | 512 tok in 7.9s
  65.3 tok/s | 512 tok in 7.8s
  63.0 tok/s | 47 tok in 0.7s
  65.7 tok/s | 512 tok in 7.8s
  66.2 tok/s | 512 tok in 7.7s

The consistency is remarkable—five measurements cluster tightly between 63.0 and 66.2 tok/s, with the 63.0 outlier being the short "What is 2+2?" prompt that only generates 47 tokens (the measurement noise increases with fewer tokens). The core result is ~65 tok/s, a 2.5× improvement over TP8's 26 tok/s and a 1.8× improvement over PP8's 36 tok/s.

The concurrency sweep begins:

  C=  1: agg=   62.3 tok/s  wall=  32.2s  avg=2002tok/req
  C=  4: agg=   95.9 tok/s  wall=  37.4s  avg=897tok/req
  C=  8: agg=  197.9 tok/s  wall=  47.7s  avg=1180tok/req
  C= 16: agg=  341.7 tok/s  wall=  61.4s ...

The scaling is healthy: doubling concurrency from 4→8 nearly doubles throughput (95.9→197.9), and 8→16 gives a 1.73× increase (197.9→341.7). The system is not yet saturated at C=16. The message is cut off at this point (the output was truncated), but the trajectory suggests EP8 would continue scaling well beyond C=16.

Why EP8 Wins on PCIe

The dramatic improvement over TP8 and PP8 can be understood through the lens of communication cost. In TP8, every token requires AllReduce across all 8 GPUs for every layer—both attention and MoE. For a model with 60 MoE layers, that's 60 AllReduce operations per token, each moving tensors proportional to the hidden dimension. On PCIe Gen5 with ~64 GB/s per direction, these AllReduce operations quickly saturate the bus.

EP8 changes the equation fundamentally. The MoE layers use All-to-All communication instead of AllReduce. In All-to-All, each GPU sends token data only to the specific GPUs holding the experts those tokens need. The total data moved is proportional to the number of tokens that need to be routed, not the full hidden state of every token. For K2.6 with top-8 routing and 48 experts per GPU, roughly 1 of the 8 selected experts is local—meaning 7/8 of tokens need to be sent to other GPUs. But the key insight is that this is token data (a vector per token) rather than weight gradients (the full expert weight matrix). The communication volume is dramatically smaller.

Furthermore, on PCIe, All-to-All can be implemented as point-to-point transfers that utilize different PCIe lanes independently, while AllReduce requires coordinated communication across all devices. The Ring AllReduce algorithm, while efficient, still requires N-1 communication rounds per operation. All-to-All can be completed in a single round-trip per destination GPU.

Assumptions and Decisions

Several key assumptions underpin this benchmark:

Assumption 1: EP avoids PCIe AllReduce bottlenecks. This was the central hypothesis, and the results validate it decisively. The assistant assumed that the AllReduce on MoE layers was the primary bottleneck in TP8, and that replacing it with All-to-All would free up significant bandwidth.

Assumption 2: Attention TP8 cost is acceptable. The assistant chose --tp-size 8 --ep-size 8 rather than --tp-size 1 --ep-size 8 (which would replicate attention across GPUs). This assumes that the MLA attention mechanism is cheap enough that its AllReduce cost doesn't negate the EP gains. The results confirm this—65 tok/s with TP8 attention is far better than any alternative.

Assumption 3: The model fits in GPU memory with EP8. With --mem-fraction-static 0.88, the assistant assumed each GPU has enough memory for its 48 experts plus 1/8 of the attention weights. The service started successfully, validating this assumption.

One potential mistake: The benchmark uses max_tokens=2048 for the concurrency sweep but max_tokens=512 for C=1. This makes the C=1 result in the concurrency sweep (62.3 tok/s with 2002 tokens) slightly different from the dedicated C=1 test (65 tok/s with 512 tokens). The difference is likely due to the longer generation altering the KV cache memory pressure and potentially triggering different CUDA graph sizes. A more rigorous benchmark would use consistent max_tokens across all measurements.

Knowledge Created

This message creates several important pieces of knowledge:

  1. Empirical validation: EP8 achieves ~65 tok/s at C=1 on 8× Blackwell PCIe for Kimi K2.6, establishing a new single-request latency baseline.
  2. Scaling characteristics: The system scales nearly linearly from C=1 to C=16 (62→342 tok/s), suggesting good utilization of the 8 GPUs.
  3. Comparative ranking: The parallelism strategy hierarchy for this hardware is now clear: EP8 > PP8 > TP8 for single-request latency, and EP8 likely leads for aggregate throughput as well.
  4. Configuration template: The exact SGLang flags, NCCL environment variables, and CUDA toolkit version form a reproducible recipe for deploying MoE models on PCIe Blackwell systems.

Conclusion

Message [msg 11512] represents the culmination of a systematic optimization journey. The assistant had worked through CUDA toolkit conflicts, FlashInfer compatibility issues, and multiple parallelism strategies before arriving at this benchmark. The 65 tok/s result validated the hypothesis that Expert Parallelism is the correct approach for MoE models on PCIe-bound multi-GPU systems.

But this was not the end of the story. The concurrency sweep was cut off at C=16, and the full scaling curve remained unknown. The assistant would go on to complete the sweep, discover that EP8 reached ~1531 tok/s aggregate at high concurrency, and eventually deploy DFlash speculative decoding on top of EP8 for even greater throughput. This message was the turning point—the moment when the optimal strategy became clear, setting the stage for all subsequent optimization work.