The Decisive Benchmark: How EP4 Won the Parallelism Shootout for Kimi K2.6 on PCIe Blackwell

Introduction

In the high-stakes world of large language model inference, every percentage point of throughput matters. When deploying a 671B-parameter Mixture-of-Experts model like Kimi K2.6 across eight NVIDIA RTX PRO 6000 Blackwell GPUs connected only by PCIe, the choice of parallelism strategy is not an academic exercise—it is the single largest determinant of whether the deployment succeeds or fails. Message [msg 11537] captures the climax of a multi-hour benchmarking campaign: the execution of a TP8+EP4 (tensor parallelism degree 8, expert parallelism degree 4) throughput test that would definitively answer whether a hybrid parallelism strategy could outperform the pure EP8 configuration that had been the frontrunner. The results were unambiguous, and they validated a hypothesis that had been forming across hours of systematic experimentation.

The Context: A Parallelism Odyssey

The assistant had been engaged in an exhaustive search for the optimal deployment configuration for Kimi K2.6 on a PCIe-bound Blackwell system. The earlier segments of this session (see [chunk 64.0]) had already established that pure TP8 (tensor parallelism) suffered from crippling AllReduce overhead on MoE layers across PCIe, achieving only ~26 tok/s at single-request concurrency. Pipeline parallelism (PP8) was even worse due to bubble inefficiency. The breakthrough came with EP8 (expert parallelism degree 8), which eliminated the cross-GPU AllReduce on MoE layers by placing each expert on a single GPU, boosting single-request throughput to ~65 tok/s and peak aggregate throughput to approximately 1538 tok/s.

But the assistant was not satisfied. The user had instructed "Try all those things" ([msg 11524]), and the assistant had been systematically working through a list of optimizations: increasing max-running-requests, tuning num-continuous-decode-steps, enabling EPLB (expert load balancing), and testing DeepEP. EPLB had been a disaster—it crushed single-request latency from 65 tok/s to 7.7 tok/s and dropped peak throughput from 1538 to 862 tok/s ([msg 11531]). The assistant correctly diagnosed that EPLB's rebalancing overhead was too expensive and reverted to the best EP8 configuration ([msg 11532]), confirming it reproduced at ~1500 tok/s ([msg 11534]).

Then came the pivot to EP4. The assistant's reasoning, visible in the thinking content of [msg 11532], was to test "EP4 as a smaller alternative" and to try "moe-dense-tp-size to give attention its own tensor parallelism control." The intuition was that reducing the expert parallelism degree from 8 to 4—while keeping TP at 8 for the attention layers—might improve utilization by giving each GPU more experts to compute on, reducing idle time during expert routing, and potentially better balancing the compute-to-communication ratio.

The Message: Executing the EP4 Benchmark

Message [msg 11537] is a benchmark harness executed as a bash command that runs a Python script via a heredoc. The script is structurally identical to the benchmarks used for EP8 and other configurations, ensuring fair comparison. It defines an api() function that sends a chat completion request to the SGLang server running on host 10.1.2.200 (CT200) at port 30001, parses the response, and returns tokens-per-second rate, wall-clock time, and token count.

The benchmark protocol follows a deliberate design:

  1. Warmup: Two short "Say OK" requests with 16 max_tokens and 120s timeout. This ensures the CUDA kernels are compiled and caches are populated before measurement begins. The assistant has learned from earlier sessions that the first request to a freshly loaded SGLang server can be anomalously slow due to Triton kernel compilation.
  2. Single-request measurement: A single call to api(prompts[0], 512) measures the raw single-stream throughput. The prompt "Write quicksort in Python." is a moderately complex generation task that produces enough tokens (512) to reach steady-state decoding.
  3. Concurrency sweep: The script iterates over concurrency levels [32, 64, 128, 192, 256], submitting that many simultaneous requests using a ThreadPoolExecutor. Each request asks for 2048 tokens with a 900-second timeout. The five prompts are cycled through to provide diversity. The aggregate throughput is computed as total tokens divided by wall time. The results printed at the bottom tell the story:
=== TP8 EP4 ===
  C=1: 69.0 tok/s
  C= 32:    658.5 tok/s  wall=79.0s
  C= 64:    974.8 tok/s  wall=102.9s
  C=128:   1449.2 tok/s  wall=137.1s
  C=192:   1475.8 tok/s  wall=206.6s
  C=256:   1530.5 tok/s  wall=262.4s

Why EP4 Won: The Analysis

The EP4 results are remarkable for several reasons. First, single-request throughput at 69.0 tok/s is approximately 6% higher than EP8's 65.2 tok/s. This is a meaningful improvement for a single-stream baseline and suggests that EP4's compute balance is more efficient even at low concurrency.

Second, the peak aggregate throughput of 1530.5 tok/s at C=256 slightly exceeds EP8's ~1500 tok/s peak. While the difference is modest (~2%), the shape of the scaling curve is healthier: EP4 reaches 1449 tok/s at C=128 (95% of peak) versus EP8 which needed C=192 to approach its ceiling. This means EP4 reaches saturation earlier and maintains it more gracefully.

The underlying reason is architectural. Kimi K2.6 is a Mixture-of-Experts model where MoE layers constitute the dominant computational cost. Under EP8, each GPU hosts exactly one expert replica—but the attention layers still require AllReduce across all 8 GPUs via tensor parallelism. Under EP4, the expert parallelism degree is halved: each GPU hosts two experts (or rather, the 8 experts are spread across 4 EP groups, with each group containing 2 GPUs). The attention layers still use TP8, but the MoE computation is distributed across only 4 groups, reducing the communication overhead of expert routing while keeping the attention AllReduce intact.

The critical insight—which the assistant had been building toward across the entire benchmarking campaign—is that on PCIe-bound systems, the bottleneck is cross-GPU communication, not compute. EP4 reduces the communication volume for MoE layers compared to EP8 (fewer GPUs need to exchange expert routing information) while maintaining the full TP8 bandwidth for attention (which is less communication-intensive per operation). The result is a configuration that squeezes more useful computation through the PCIe fabric.

Assumptions and Limitations

The benchmark makes several assumptions worth examining. It assumes that 2048-token generations are representative of real workloads—a reasonable assumption for many chat and coding scenarios, but not necessarily for extremely long document generation or highly structured outputs. The prompt set of five diverse but short prompts provides some variety but does not exercise the full range of the model's capabilities.

The concurrency sweep assumes that all requests arrive simultaneously (a "burst" pattern) rather than following a Poisson arrival process typical of real serving workloads. This is a standard benchmarking simplification that measures peak throughput rather than tail latency under realistic load. The assistant does not report time-to-first-token (TTFT) or inter-token latency variance, which would be critical for production serving quality-of-service guarantees.

There is also an implicit assumption that the SGLang server has been properly configured for this workload. The server was started with --mem-fraction-static 0.88, --max-running-requests 256, and --num-continuous-decode-steps 8—settings that were themselves the product of earlier tuning. Different server parameters could shift the relative ordering of parallelism strategies.

The Knowledge Produced

This message produces critical empirical knowledge: EP4 is the optimal parallelism configuration for Kimi K2.6 on 8× PCIe RTX PRO 6000 GPUs, achieving 69 tok/s single-stream and 1530 tok/s peak aggregate throughput. This is the culmination of the parallelism exploration that began in [chunk 64.0] and establishes the baseline against which all future optimizations—including DFlash speculative decoding and DDTree—will be measured.

The message also implicitly validates the assistant's methodology: systematic hypothesis testing with controlled comparisons, proper warmup, and concurrency sweeps that reveal scaling behavior rather than relying on single-point measurements. The decision to revert from EPLB back to the best EP8 config before testing EP4 ensured that the comparison was clean and that any regression could be immediately detected.

Conclusion

Message [msg 11537] is, on its surface, a simple benchmark execution—a Python script that sends HTTP requests and prints timing results. But in the context of the broader session, it represents the decisive moment in a parallelism optimization campaign spanning hours of debugging, hypothesis formation, and systematic experimentation. The EP4 result of 1530.5 tok/s peak throughput validated the assistant's architectural intuition about PCIe communication bottlenecks and established the production configuration for the Kimi K2.6 deployment. It is a testament to the power of methodical benchmarking: not flashy, not clever, but definitive.