Pivoting Under Pressure: How a Failed Expert Parallelism Experiment Led to a Breakthrough on NVLink

Introduction

In the high-stakes world of large language model inference optimization, the path to peak performance is rarely a straight line. Message [msg 11792] captures a pivotal moment in an extended optimization session for Kimi K2.6, a Mixture-of-Experts (MoE) model running on NVIDIA B300 SXM6 GPUs with NVLink interconnect. The assistant had been systematically exploring parallelism strategies—tensor parallelism (TP), pipeline parallelism (PP), and expert parallelism (EP)—to maximize throughput for speculative decoding with the DDTree algorithm. What makes this message remarkable is not a triumphant breakthrough, but rather the disciplined reasoning process by which the assistant absorbed three consecutive failures, re-evaluated the problem from first principles, and pivoted to a simpler configuration that worked immediately.

The message sits at a critical inflection point. The user had observed that the GPUs were drawing only 300–400W out of a 1100W thermal budget ([msg 11785]), suggesting severe underutilization. Their intuition was that expert parallelism combined with NVLink's multicast capabilities (NVLS) could better saturate the hardware. The assistant had already attempted three variations of this idea—EP8+NVLS+allreduce-fusion (CUBLAS error), EP8+NVLS alone (CUBLAS error in MLA attention), and EP8 alone (NoneType error during CUDA graph capture)—all of which failed at startup. This message is the moment the assistant steps back, analyzes why the failures occurred, and charts a new course.

The Reasoning Process: A Window Into Diagnostic Thinking

The message opens with a substantial Agent Reasoning block that reveals the assistant's cognitive process in real time. This is not a polished retrospective but a raw, stream-of-consciousness diagnostic that mirrors how an experienced engineer would think through a complex systems problem.

The assistant begins by connecting the dots between the two failure modes:

So EP8 without NVLS triggers a different failure—a NoneType error on the cuda graph side, which mirrors the original DDTree issue. This suggests EP8 combined with DDTree and cuda graphs is hitting a None buffer somewhere, while EP8 with NVLS fails earlier in the MLA absorb step with a CUBLAS error, likely due to memory constraints.

This is a crucial diagnostic insight. The assistant recognizes that the two failure modes are different — they occur at different stages of the startup sequence and likely have different root causes. The EP8+NVLS combination fails during the "dummy run" (CUDA graph capture) in the MLA attention's batched GEMM, while EP8 alone fails later with a null pointer. This rules out a single simple bug and suggests the EP+DDTree combination is fundamentally unstable in this SGLang build.

But then the assistant does something more valuable: it zooms out entirely.

Let me zoom out though. The user's low power observation and EP+NVLS suggestion are worth reconsidering against what we actually know: the workload is HBM-bandwidth-bound at 100% utilization and 360W.

This "zoom out" is the intellectual core of the message. The assistant reframes the problem from "how do we make EP+NVLS work?" to "what is the actual bottleneck, and does EP address it?" This reframing is what separates effective debugging from chasing symptoms.

The HBM-Bandwidth Analysis

The assistant then delivers a concise but deep analysis of the workload's bottleneck characteristics:

For INT4 MoE at decode, the real bottleneck is reading expert weights from HBM. EP and TP distribute this differently—with TP8, each GPU reads 1/8 of each selected expert across all 8 GPUs, while EP8 has each GPU holding 48 full experts and reading only the weights for experts its tokens route to. At low batch sizes this creates load imbalance, but at higher batch (C=64+) when all experts activate, both approaches end up reading the full expert set once total, just with different communication patterns.

This analysis reveals a critical insight: at high concurrency, the total HBM traffic is roughly the same regardless of parallelism strategy. The difference lies in the communication pattern — TP8 requires AllReduce across all 8 GPUs after each MoE layer to combine the partial expert results, while EP8 requires an all-to-all communication to route tokens to the correct GPU before each MoE layer. On NVLink, AllReduce is relatively cheap (NVLink provides high-bandwidth, low-latency GPU-to-GPU communication), so TP8's communication overhead is minimal. EP8's advantage on PCIe systems (where AllReduce is expensive) largely disappears.

The assistant also offers a more mundane explanation for the low power draw:

The low power is probably just that INT4 GEMM doesn't fully saturate the tensor cores and the batch size isn't large enough to drive full utilization.

This is a grounded, practical assessment. INT4 matrix multiplications are memory-bound, not compute-bound — the tensor cores can process the data faster than HBM can supply it. The 360W power draw at 100% GPU utilization is consistent with a memory-bound workload where the compute units are stalled waiting for data.

The Decision: Pivot, Don't Perseverate

The most consequential decision in this message is captured in a single sentence:

More importantly, TP8+DDTree already works well on B300 (285 tok/s at C=1, 3795 at C=64, 2.15x spec speedup), and EP didn't even win on the PCIe box for DDTree—only for autoregressive.

This is a masterclass in opportunity-cost-aware decision-making. The assistant explicitly weighs the marginal benefit of fixing EP (uncertain, possibly zero) against the cost of continued debugging (time, attention, complexity). The working TP8+DDTree configuration already achieves a 2.15× speedup over the autoregressive baseline. Even if EP could be made to work, the improvement over TP8 on NVLink is likely small because NVLink already makes TP8 collectives cheap.

The assistant then formulates a concrete next step:

Rather than chase EP bugs, I should test whether NVLS helps the working TP8 config and push concurrency higher to see if throughput keeps scaling.

This is a beautifully framed hypothesis. NVLS (NVLink Shared Memory / multicast) can accelerate the AllReduce operations that TP8 requires, potentially reducing the communication overhead further. Testing NVLS on the working TP8 configuration isolates the variable and avoids the confounding bugs introduced by EP.

The Execution: Clean Pivot

The bash command that follows is clean and minimal:

# TP8 + NVLS only on the working DDTree service
cp /etc/systemd/system/sglang-k26-ddtree.service /etc/systemd/system/sglang-k26-ddtree-nvls.service
sed -i 's|--tp-size 8 |--tp-size 8 --enable-nccl-nvls |' /etc/systemd/system/sglang-k26-ddtree-nvls.service
sed -i 's|Description=.*|Description=K2.6 TP8 NVLS + DDTree (B300)|' /etc/systemd/system/sglang-k26-ddtree-nvls.service
systemctl daemon-reload && systemctl start sglang-k26-ddtree-nvls.service

The assistant creates a new systemd service file by copying the known-working DDTree service and adding only the --enable-nccl-nvls flag. This is a deliberate strategy of minimal modification — change one thing at a time, test, and evaluate. The readiness check loop is also clean, with a 50-iteration timeout and explicit failure detection.

The result: [165s] TP8+NVLS READY. The service starts successfully after about 2 minutes and 45 seconds of weight loading. This is the first successful startup after three consecutive failures, validating the assistant's pivot decision.

Assumptions and Their Validity

Several assumptions underpin the reasoning in this message:

  1. The workload is HBM-bandwidth-bound. This is supported by the 100% GPU utilization at only 360W power draw. The assumption is reasonable and consistent with known characteristics of INT4 MoE inference.
  2. EP's advantage shrinks on NVLink. This follows from the architectural understanding that NVLink reduces AllReduce costs, which is TP8's main weakness. The assumption is sound but unverified — the EP8+DDTree combination never worked on this machine, so we cannot directly compare.
  3. EP+DDTree bugs are a deep rabbit hole. The assistant has observed two distinct failure modes across three configurations. This assumption is conservative and justified by the evidence.
  4. NVLS is compatible with TP8+DDTree. This is the hypothesis being tested, and it turns out to be correct. One potential blind spot: the assistant assumes that the EP bugs are in SGLang's DDTree integration rather than in the configuration or environment. It's possible that a different combination of flags, a different SGLang build, or a different startup sequence would make EP work. However, given the time pressure and the existence of a working alternative, the decision to pivot is defensible.

Input Knowledge Required

To fully understand this message, the reader needs:

Output Knowledge Created

This message produces several forms of knowledge:

  1. A confirmed working configuration: TP8+NVLS+DDTree on B300 with CUDA graphs, verified by a successful service startup.
  2. A documented failure mode: EP8+DDTree with CUDA graphs is unstable in this SGLang build, producing either CUBLAS errors (with NVLS) or NoneType errors (without NVLS).
  3. A bottleneck analysis: The workload is HBM-bandwidth-bound, not compute-bound, which constrains the expected benefit of any optimization.
  4. A decision framework: A template for when to persevere with a complex optimization versus pivot to a simpler working alternative, based on opportunity cost and marginal benefit.

Conclusion

Message [msg 11792] is a case study in disciplined engineering under uncertainty. The assistant faced three consecutive failures of a promising optimization path, resisted the temptation to continue debugging in circles, and instead re-examined the fundamental assumptions underlying the approach. By recognizing that the workload's bottleneck (HBM bandwidth) was unlikely to be addressed by expert parallelism on NVLink hardware, and that the working TP8 configuration already delivered substantial speedups, the assistant made a clean pivot that yielded immediate results.

The message also illustrates the value of "zooming out" — stepping back from the immediate debugging task to reconsider the problem at a higher level of abstraction. This cognitive move, combined with a clear articulation of tradeoffs and a minimal-change execution strategy, transformed a series of failures into a successful outcome. For anyone working on ML inference optimization, this message offers a model of how to think about parallelism choices, bottleneck analysis, and the art of knowing when to pivot.