The Fair Baseline: Establishing TP8 with CUDA Graphs for Kimi K2.6 on PCIe Blackwell

In the midst of an intensive benchmarking campaign for Kimi K2.6 on an 8× RTX PRO 6000 (Blackwell) machine, the assistant reached a critical methodological turning point. Message <msg id=11538> is deceptively brief — a single sentence of analysis followed by a bash command to deploy a new service configuration. But this message represents the moment the assistant recognized a gap in their experimental design and moved to close it, establishing what they explicitly called "the fair TP8 baseline we never had." Understanding why this mattered, and what reasoning led to this moment, requires tracing the benchmarking logic that preceded it.

The Benchmarking Arc: From EP8 to EP4 to the Missing Baseline

The context leading up to this message reveals a systematic parallelism optimization campaign. The assistant had been testing Kimi K2.6 — a Mixture-of-Experts (MoE) model — across multiple parallelism strategies on a PCIe-only Blackwell machine (no NVLink between GPUs). The key insight driving this work was that MoE models suffer from a specific bottleneck on PCIe interconnects: tensor parallelism (TP) requires AllReduce operations across all GPUs for every layer, including the large expert weight matrices. Expert parallelism (EP) avoids this by routing each token to only the GPUs holding its assigned experts, dramatically reducing communication volume.

The assistant's earlier benchmarks confirmed this intuition. EP8 (8 expert shards) achieved ~1500 tok/s peak aggregate throughput, while a brief experiment with EPLB (expert load balancing) proved counterproductive — it introduced rebalancing overhead that crushed low-concurrency latency and dropped peak throughput to 862 tok/s. The assistant then tested EP4 (4 expert shards with TP2 groups), which proved "slightly better across the board" at ~1531 tok/s peak.

But here's the critical gap: the assistant had never actually run a properly tuned TP8 (pure tensor parallelism, no expert parallelism) baseline. Earlier TP8 tests had been run without CUDA graphs enabled and without the NCCL tuning that might help mitigate the PCIe bottleneck. Without this baseline, the assistant couldn't quantify how much EP was actually helping — they knew EP was better, but not by how much, and not whether the gap could be narrowed with better TP tuning.

The Reasoning: What "Fair" Means in This Context

The assistant's opening line — "EP4 is slightly better across the board. Now TP8 with CUDA 13 and cuda graphs (the fair TP8 baseline we never had)" — reveals several layers of reasoning.

First, the acknowledgment that EP4 is "slightly better" is itself a meaningful finding. EP4 uses 4 expert shards with TP2 groups (tensor parallelism within each expert group), meaning it still incurs some AllReduce overhead but less than TP8. The fact that EP4 outperforms EP8 suggests that the optimal EP configuration isn't the one that minimizes communication the most — there's a tradeoff between communication efficiency and compute utilization, and EP4 hits the sweet spot.

Second, the phrase "the fair TP8 baseline we never had" is an admission of an experimental design flaw. Earlier in the segment, TP8 had been tested but without the benefit of CUDA graphs (which reduce kernel launch overhead by capturing and replaying GPU operations) and without the aggressive NCCL tuning that might improve PCIe bandwidth utilization. The assistant is now correcting this by creating a TP8 configuration that matches the tuning applied to the EP configurations — same --num-continuous-decode-steps 8, same --max-running-requests 256, same --mem-fraction-static 0.88, but now with CUDA graphs implicitly enabled (they're the default in SGLang when using the triton attention backend with CUDA 13).

The NCCL Tuning: A Hypothesis About PCIe Bottlenecks

The most technically interesting aspect of this message is the NCCL environment variables in the TP8 service configuration. These are absent from the EP configurations, revealing that the assistant is making specific hypotheses about how to optimize TP8 for PCIe:

• NCCL_MIN_NCHANNELS=8 and NCCL_MAX_NCHANNELS=16: These control the number of NCCL communication channels. More channels can improve bandwidth utilization on multi-GPU systems by enabling more concurrent communication streams. The assistant is betting that TP8's AllReduce overhead can be mitigated by aggressive channel parallelism.
• NCCL_PROTO=LL: This selects the Low Latency protocol for NCCL, which uses shared memory for small message sizes. This is an interesting choice — LL protocol trades off bandwidth for latency, which might help with the frequent but relatively small AllReduce operations in the attention layers.
• NCCL_BUFFSIZE=16777216 (16 MB): This sets the NCCL buffer size, which affects how data is segmented for communication. Larger buffers can improve throughput for large transfers but may increase latency.
• NCCL_NTHREADS=512: This allocates more CPU threads for NCCL communication, potentially reducing CPU-side bottlenecks in the communication pipeline. These parameters represent a sophisticated understanding of NCCL internals. The assistant is essentially trying to squeeze every bit of performance out of the PCIe interconnect to see how close TP8 can get to the EP configurations. The implicit question being asked: "If we tune NCCL aggressively, can TP8 close the gap with EP on PCIe?"

The Broader Methodological Significance

This message exemplifies a crucial scientific practice: establishing proper baselines before making comparative claims. The assistant had been running EP configurations and observing their performance, but without a comparable TP8 baseline, the magnitude of EP's advantage was unknown. The "fair" qualifier is important — it acknowledges that earlier TP8 runs weren't comparable because they lacked the same tuning knobs (CUDA graphs, NCCL tuning, continuous decode steps).

The message also reveals the assistant's mental model of the problem. They understand that:

1. TP8 is expected to be slower on PCIe due to AllReduce overhead on MoE layers
2. But the degree of slowdown matters for deployment decisions
3. A properly tuned TP8 baseline is needed to quantify EP's benefit
4. NCCL tuning might narrow the gap, making the comparison more informative

What Follows: The Results

While the subject message only contains the deployment command, the subsequent messages (visible in the broader segment context) reveal the outcome. The TP8-tuned service achieved ~98 tok/s at C=1 and ~1291 tok/s peak aggregate throughput. This confirmed the assistant's hypothesis — TP8 was indeed slower than EP4 (69 tok/s at C=1, ~1531 tok/s peak) and EP8 (65 tok/s at C=1, ~1500 tok/s peak), but the gap was smaller than one might expect. The NCCL tuning helped, but not enough to overcome the fundamental AllReduce overhead on MoE layers.

This result validated the user's earlier intuition that expert parallelism avoids PCIe AllReduce bottlenecks — EP configurations won across the board. But the TP8 baseline also served another purpose: it provided a reference point for the subsequent DFlash speculative decoding experiments, where the assistant would later deploy K2.6 with DFlash on EP8 and measure the speedup over the autoregressive baseline.

Input Knowledge Required

To fully understand this message, one needs:

• Familiarity with parallelism strategies for large language models (TP vs EP vs PP)
• Understanding of how MoE architectures create unique communication patterns
• Knowledge of NCCL tuning parameters and their effects on PCIe communication
• Awareness of CUDA graphs and their role in reducing kernel launch overhead
• Context about the hardware platform: 8× RTX PRO 6000 Blackwell GPUs connected via PCIe (no NVLink)

Output Knowledge Created

This message produces:

• A deployed TP8 service configuration with aggressive NCCL tuning
• A controlled experimental condition for comparing TP8 vs EP4 vs EP8
• Documentation (in the service file) of the NCCL parameters tested
• A methodological correction: the fair TP8 baseline that was missing from earlier experiments

Conclusion

Message <msg id=11538> is a small but methodologically significant moment in a larger benchmarking campaign. It represents the assistant recognizing a gap in experimental design and taking action to close it, all while articulating the reasoning clearly. The message is structured as a single analytical sentence followed by an execution command — the thinking is compressed into that opening line, but it contains a wealth of implied reasoning about experimental methodology, hardware bottlenecks, and the importance of fair comparisons. In the broader arc of the session, this TP8 baseline would prove essential for quantifying the benefits of expert parallelism and, later, speculative decoding with DFlash.