The Parallelism Reckoning: How One Summary Table Settled the Architecture for Kimi K2.6 Inference

Introduction

In the course of deploying the Kimi K2.6 mixture-of-experts (MoE) model across eight RTX PRO 6000 Blackwell GPUs, the assistant arrived at a moment of synthesis. Message [msg 11541] is a single, compact summary table—just five rows and seven columns—that distills hours of systematic benchmarking into a decisive verdict. The message reads:

Interesting -- TP8 with cuda graphs gives 97.9 tok/s at C=1 (vs 26.3 without graphs, vs 65 EP8). But EP4 still wins at high concurrency. Summary:

>

| Config | C=1 | C=32 | C=64 | C=128 | C=192 | C=256 | |--------|-----|------|------|-------|-------|-------| | TP8 old (no graphs) | 26.3 | 577.8 | 693 | - | - | ~808 | | TP8 tuned (graphs+cont) | 97.9 | 683.9 | 951.4 | 1291.0 | 1213.9 | 1263.0 | | PP8 | 36.9 | 250.9 | 396.0 | - | - | - | | EP8 | 65.2 | 624.1 | 935.6 | 1416.9 | 1441.9 | 1492.5 | | EP4 | 69.0 | 658.5 | 974.8 | 1449.2 | 1475.8 | 1530.5 |

>

TP8 tuned has the best single-request latency (97.9 tok/s -- cuda graphs eliminating kernel launch overhead). EP4 has the best throughput at high concurrency (1530 tok/s at C=256). For completion generation we care about sustained throughput, so EP4 is the winner at ~1500 tok/s. For interactive serving, TP8 tuned at 98 tok/s is remarkable.

On its surface, this appears to be a routine performance report. But in the context of the broader session—spanning CUDA toolkit recovery, FlashInfer SM120 compatibility fixes, and the systematic exploration of four parallelism strategies—this message represents the analytical climax of a major infrastructure investigation. It is the moment when raw data becomes actionable knowledge.

The Context That Made This Message Necessary

To understand why this message was written, one must appreciate the hardware reality that constrained every decision. The eight RTX PRO 6000 Blackwell GPUs were connected via PCIe only—no NVLink. This seemingly mundane detail is the hidden protagonist of the entire benchmarking campaign. PCIe bandwidth is dramatically lower than NVLink's direct GPU-to-GPU interconnects, which means that any parallelism strategy requiring frequent AllReduce communication across GPUs would pay a severe penalty.

The Kimi K2.6 model is a mixture-of-experts architecture with 256 experts. In a standard tensor parallelism (TP) configuration, every transformer layer requires synchronizing activations across all GPUs via AllReduce. For an MoE model, this is especially painful: the gating network selects which experts to activate, but the expert computations are distributed across GPUs, and the results must be gathered and reduced. On PCIe, this communication overhead can dominate the computation, especially at low concurrency where there are no other requests to hide the latency.

The user had hypothesized early on that expert parallelism (EP)—where each GPU hosts a subset of experts and handles them independently—would outperform tensor parallelism on PCIe precisely because EP eliminates the AllReduce on the MoE layers. The assistant's job was to prove or disprove this hypothesis through systematic measurement.

The Benchmarking Campaign: What Came Before

The messages immediately preceding the subject message show the assistant executing a disciplined experimental protocol. After resolving a cascade of infrastructure issues (FlashInfer rejecting SM120 Blackwell GPUs, missing curand.h headers, patching SGLang's attention backend for pipeline parallelism), the assistant ran four distinct configurations:

  1. TP8 (tensor parallelism, 8 GPUs): The default approach. Every layer is sharded across all GPUs. The "old" TP8 result (26.3 tok/s at C=1) was measured without CUDA graphs and without the num-continuous-decode-steps optimization. The "tuned" TP8 added CUDA graphs (which pre-capture GPU kernel launch sequences, eliminating CPU launch overhead) and continuous batching with 8 decode steps.
  2. PP8 (pipeline parallelism, 8 stages): Layers are partitioned across GPUs, with each GPU handling a subset of layers. The assistant had previously found PP8 disappointing due to pipeline bubbles—idle time when earlier stages finish their portion and wait for later stages to complete.
  3. EP8 (expert parallelism, 8 GPUs): Each GPU hosts all 8 expert copies of 32 experts (256 total / 8). The MoE layers avoid AllReduce entirely; only the attention layers require TP communication. This was the user's favored hypothesis.
  4. EP4 (expert parallelism, 4 groups of 2 TP): A hybrid: 4 expert-parallel groups, each containing 2 GPUs connected via tensor parallelism. This reduces the expert sharding granularity while keeping TP groups small enough that the AllReduce penalty is contained. The assistant tested EPLB (expert load balancing) in msg <id=11529> and found it hurt performance—C=1 dropped from 65 to 7.7 tok/s in one test, and peak throughput fell from 1538 to 862 tok/s. This negative result was valuable: it established that the default expert routing was already well-balanced, and the rebalancing overhead was pure cost.

What the Table Reveals

The summary table in message <id=11541> tells a nuanced story that rewards careful reading.

At single-request (C=1): TP8 tuned achieves 97.9 tok/s—a stunning 3.7× improvement over the 26.3 tok/s of the untuned TP8. This is the CUDA graph effect in action. Without graphs, each kernel launch incurs CPU-side overhead that becomes the dominant latency component at low concurrency. With graphs, the entire decode step is captured as a single GPU operation, and the CPU is freed from per-kernel launch duties. The 97.9 tok/s result is the best single-stream performance of any configuration, making TP8 tuned the clear choice for interactive serving where latency matters.

EP4 (69.0 tok/s) and EP8 (65.2 tok/s) trail significantly at C=1. This is expected: expert parallelism requires communication for attention layers (which are TP-sharded), and at low concurrency there are no overlapping requests to amortize that cost. The pure TP8 tuned configuration has the advantage of unified communication patterns.

At high concurrency (C=256): The ranking flips. EP4 achieves 1530.5 tok/s, EP8 achieves 1492.5 tok/s, and TP8 tuned plateaus at 1263.0 tok/s. The expert parallelism configurations scale better because the AllReduce bottleneck on MoE layers is eliminated. Each GPU computes its assigned experts independently, and the communication overhead is limited to attention layers. At high concurrency, where many requests are being processed simultaneously, the GPU compute becomes the bottleneck rather than the interconnect—and EP configurations simply have more effective compute available since they aren't waiting on PCIe AllReduce.

The crossover point: At C=32, TP8 tuned (683.9 tok/s) still leads EP4 (658.5 tok/s) and EP8 (624.1 tok/s). By C=64, TP8 (951.4) and EP4 (974.8) are roughly tied. By C=128, EP4 (1449.2) has clearly pulled ahead of TP8 (1291.0). This crossover behavior is exactly what one would expect from a system where communication overhead is fixed per-step but compute scales with batch size. At low batch sizes, communication dominates; at high batch sizes, compute dominates.

PP8 is the clear loser: At 36.9 tok/s C=1 and only 396.0 tok/s at C=64, pipeline parallelism is not competitive on this hardware. The pipeline bubble problem—where only one stage is active at any given moment during the forward pass—is exacerbated by the model's depth and the PCIe interconnect. PP8 was included for completeness but never seriously contended.

The Reasoning Process Visible in the Message

The assistant's thinking is revealed in the way the message is structured. It begins with an observation ("Interesting -- TP8 with cuda graphs gives 97.9 tok/s at C=1") that signals surprise. The TP8 tuned result exceeded expectations—the assistant had previously measured TP8 at 26.3 tok/s without graphs, and the 3.7× improvement from CUDA graphs alone was dramatic enough to warrant explicit note.

The comparative framing ("vs 26.3 without graphs, vs 65 EP8") shows the assistant is reasoning relative to known baselines. The "without graphs" number comes from an earlier benchmark (msg <id=11540> context), and the EP8 number comes from the confirmed best config (msg <id=11534>). The assistant is triangulating: TP8 tuned beats EP8 at C=1 by 50% (97.9 vs 65.2), which is a meaningful advantage for latency-sensitive workloads.

The table itself is a reasoning artifact. The assistant chose to include all four configurations (TP8 old, TP8 tuned, PP8, EP8, EP4) even though PP8 was clearly non-competitive. This completeness serves two purposes: it documents the full experimental space (so no one later asks "what about pipeline parallelism?") and it provides a visual hierarchy that makes the EP4 victory more convincing by contrast.

The final paragraph draws two distinct conclusions for two distinct use cases:

For completion generation we care about sustained throughput, so EP4 is the winner at ~1500 tok/s. For interactive serving, TP8 tuned at 98 tok/s is remarkable.

This dual-conclusion is significant. The assistant is acknowledging that the "best" configuration depends on the workload. This is not a one-size-fits-all answer but a nuanced recommendation that accounts for the difference between batch processing (where aggregate throughput matters) and interactive serving (where per-request latency matters). The assistant could have declared a single winner, but instead chose to preserve the trade-off information.

Assumptions and Their Validity

Several assumptions underpin this message, and it is worth examining them.

Assumption 1: PCIe bandwidth is the primary bottleneck at low concurrency. This assumption is validated by the data. TP8 tuned's 97.9 tok/s at C=1 versus EP8's 65.2 tok/s shows that when communication is the dominant factor (single request, no overlap), reducing communication via CUDA graphs (which don't reduce communication volume but reduce launch overhead) helps TP8 more than switching to EP. However, the fact that EP4 (69.0) beats EP8 (65.2) at C=1 suggests that the TP2 grouping within EP4 provides some communication benefit even at low concurrency—perhaps because the AllReduce within a 2-GPU group is faster than the all-to-all across 8 GPUs.

Assumption 2: CUDA graphs are a pure optimization with no downsides. The assistant treats CUDA graphs as unambiguously beneficial, and the data supports this: TP8 tuned (with graphs) is 3.7× better than TP8 old (without graphs) at C=1, and consistently better at all concurrency levels. However, CUDA graphs have a known limitation: they capture a static sequence of kernels and cannot adapt to dynamic shapes or branching. The assistant does not discuss this limitation in this message, but it becomes relevant later when DFlash speculative decoding is deployed—the verify path in DFlash is incompatible with CUDA graphs, which limits the benefit of speculative decoding at high concurrency.

Assumption 3: The benchmark prompts are representative. The assistant used a set of five prompts (quicksort, relativity, Redis guide, haiku, JSON parser) with 2048 max tokens. These are reasonable for a coding/chat model, but they may not represent all workloads. A model serving very short prompts (e.g., classification) or very long prompts (e.g., document analysis) might see different crossover points. The assistant does not claim generality beyond this workload, which is appropriate.

Assumption 4: EP4's advantage over EP8 is meaningful. The difference between EP4 (1530.5 tok/s) and EP8 (1492.5 tok/s) at C=256 is only 2.5%. This is within measurement noise for a single run. The assistant does not report variance or repeat measurements, so it is possible that EP8 and EP4 are statistically tied at high concurrency. The assistant's conclusion that "EP4 is the winner" is technically correct based on the point estimates, but a more rigorous analysis would include confidence intervals. However, for the purpose of deployment decisions, the practical difference between ~1500 and ~1530 tok/s is negligible—both configurations are in the same performance class.

Input Knowledge Required

To fully understand this message, the reader needs:

  1. Understanding of parallelism strategies for MoE models: Tensor parallelism shards every layer across GPUs, requiring AllReduce on every forward pass. Expert parallelism shards only the expert weights, avoiding AllReduce on MoE layers but requiring it for attention layers. Pipeline parallelism partitions layers across GPUs, creating sequential dependencies.
  2. Knowledge of CUDA graphs: A CUDA graph captures a sequence of GPU kernel launches as a single unit, eliminating CPU launch overhead. This is especially beneficial for small, frequent operations like the individual kernels in a transformer decode step.
  3. The hardware context: Eight RTX PRO 6000 Blackwell GPUs connected via PCIe, with no NVLink. This makes communication bandwidth the critical resource.
  4. The model architecture: Kimi K2.6 is a 256-expert MoE model. The large number of experts makes expert parallelism particularly attractive because each GPU can host a substantial subset.
  5. The previous experimental results: The assistant had already tested EPLB (negative result), confirmed EP8 reproducibility, and measured PP8. The message builds on this accumulated knowledge.

Output Knowledge Created

This message creates several forms of actionable knowledge:

  1. A ranked ordering of parallelism strategies for this specific hardware-model combination: EP4 > EP8 > TP8 tuned > TP8 old > PP8, with the caveat that TP8 tuned wins for latency-sensitive workloads.
  2. Quantified performance expectations: ~1500 tok/s peak throughput for batch processing, ~98 tok/s for interactive single-request serving. These numbers become baselines for future optimization work.
  3. A validated hypothesis: The user's intuition that expert parallelism would outperform tensor parallelism on PCIe is confirmed—EP configurations win at high concurrency. But the magnitude depends on concurrency level.
  4. A decision framework: The assistant explicitly separates the "throughput" use case from the "latency" use case, providing guidance for deployment configuration based on workload characteristics.
  5. A reference point for CUDA graph impact: The 3.7× improvement from CUDA graphs at C=1 (26.3 → 97.9 tok/s) quantifies the value of this optimization technique for future work.

Mistakes and Subtle Issues

The most significant subtle issue is the missing discussion of statistical significance. The assistant presents point estimates without variance. Given that the differences between EP4 and EP8 at high concurrency are only ~2.5%, and that the benchmarking methodology (single warmup, sequential concurrency sweeps) could introduce systematic bias from GPU thermal throttling or memory fragmentation, the conclusion that "EP4 is the winner" should be treated as provisional. A more rigorous approach would involve multiple runs and confidence intervals.

A second issue is the incomplete characterization of the TP8 tuned configuration. The assistant attributes the improvement to "cuda graphs eliminating kernel launch overhead," but the tuned configuration also includes num-continuous-decode-steps=8 and max-running-requests=256. These changes independently improve throughput by allowing the scheduler to batch more work. The 3.7× improvement is a composite effect, not purely CUDA graphs.

Third, the PP8 result is not fully contextualized. Pipeline parallelism can be competitive when the model is deep enough that the pipeline fill/drain overhead is amortized over many micro-batches. The assistant tested PP8 with the same continuous decode settings, but pipeline parallelism typically requires a different scheduling strategy (e.g., asynchronous pipeline scheduling) to avoid bubbles. The poor PP8 result may reflect suboptimal tuning rather than a fundamental limitation of the approach.

The Deeper Significance

This message is more than a benchmark summary. It represents the moment when the assistant's investigative work transitions from exploration to decision. The preceding messages show a systematic search: test EPLB, revert when it fails, confirm EP8, try EP4, try TP8 tuned. Each experiment builds on the previous one, and the subject message is the synthesis that makes sense of the whole.

The message also reveals the assistant's analytical style: data-driven, comparative, and nuanced. Rather than declaring a single winner, the assistant provides a decision matrix that accounts for workload characteristics. The table is designed to be read, understood, and acted upon—not just admired.

In the broader arc of the session, this message sets the stage for the next phase: deploying DFlash speculative decoding on top of the winning EP4 configuration. The assistant will go on to test DFlash with sliding window attention, benchmark DDTree on NVLink hardware, and ultimately write a comprehensive findings report. But all of that work depends on having a solid autoregressive baseline, which this message provides.

The message also implicitly validates the user's strategic intuition. The user had argued that expert parallelism would win on PCIe hardware. The assistant's systematic benchmarking confirmed this, but with the important caveat that CUDA graphs make TP8 competitive at low concurrency. This kind of empirical nuance—where the answer is "it depends" rather than a simple winner—is the hallmark of serious performance engineering.

Conclusion

Message <id=11541> is a masterclass in analytical communication. In a single compact table and three paragraphs, the assistant summarizes hours of experimental work, provides actionable recommendations for two distinct use cases, and establishes the performance baselines that will guide all subsequent optimization. The message is notable for its clarity, its honesty about trade-offs, and its refusal to oversimplify. It acknowledges that the "best" configuration depends on the workload, and it gives the reader the tools to make that decision themselves.

For anyone deploying large MoE models on PCIe-connected GPUs, this message contains a wealth of practical knowledge: CUDA graphs are transformative for latency, expert parallelism wins for throughput, and the crossover point between the two strategies occurs at moderate concurrency levels. These findings are not architecture-specific—they reflect fundamental properties of the hardware and the model class—and they will remain relevant as long as PCIe remains the dominant GPU interconnect for multi-GPU servers without NVLink.