The HBM-Bound Hypothesis: Diagnosing GPU Utilization and Pivoting to Expert Parallelism on NVLink

Introduction

In the high-stakes world of large language model inference optimization, few moments are as revealing as the first time you see a GPU running at 100% utilization while drawing only one-third of its rated power. This paradoxical signal—full occupancy paired with anemic power draw—tells an unambiguous story: the hardware is not compute-bound but memory-bandwidth-bound. The GPUs are waiting, stalled on loading weights from High Bandwidth Memory (HBM) into the compute units, and no amount of parallel decode scheduling will fix a fundamental memory-wall bottleneck.

This is precisely the situation the assistant confronts in message [msg 11788], a pivotal turning point in a multi-day optimization campaign to deploy Kimi K2.6 with DFlash speculative decoding across two radically different hardware platforms: a PCIe-connected 8× RTX PRO 6000 system and an 8× B300 SXM6 machine with NVLink interconnects. The message captures the moment when raw benchmarking data crystallizes into architectural insight, and that insight drives a strategic pivot from tensor parallelism (TP) to expert parallelism (EP) combined with NVLink multicast (NVLS) collectives.

This article examines [msg 11788] in depth: the reasoning that led to its decisions, the assumptions embedded in its analysis, the knowledge it required and produced, and the subtle interplay between measurement, interpretation, and action that defines effective ML infrastructure engineering.

The Scene: What Came Before

To understand [msg 11788], we must first understand the arc of the preceding conversation. The assistant had been engaged in a sprawling deployment effort spanning multiple machines, CUDA toolkit versions, and parallelism strategies. The work began in [msg 11776] with a simple health check: "up?" The service was active but not responding—stuck in CUDA graph capture initialization. A cascade of issues followed: missing flash_attn.cute on the B300 machine (solved by --skip-server-warmup), a model download via aria2 at 575 MiB/s, and the discovery that the B300's sm_103 architecture required special handling for Triton JIT compilation.

By [msg 11783], the assistant had successfully deployed Kimi K2.6 with DDTree speculative decoding on the B300 machine and run a comprehensive benchmark matrix. The results were striking: DDTree achieved 285 tok/s at concurrency 1 (C=1) versus 133 tok/s for the autoregressive baseline—a 2.15× speedup. At higher concurrency, however, the picture grew more complex. At C=64, DDTree reached 3795 tok/s aggregate, but the advantage over the baseline (3870 tok/s) had vanished, and DDTree was actually slightly slower.

Then came the user's observation in [msg 11785]: "Only 300-400W power use out of 1100W, maybe worth trying EP now that we have nvlink with multicast support? Cuda graphs working?" This single comment reframed the entire problem. The GPUs were thermally and electrically underutilized despite showing 100% utilization in nvidia-smi. The user suspected—correctly—that the bottleneck was not compute but communication and memory bandwidth.

The assistant confirmed in [msg 11786] that CUDA graphs were indeed active (51 decode batches with graphs, 80 prefills without—normal behavior), and discovered that SGLang supported NVLS flags including --enable-nccl-nvls and --enable-flashinfer-allreduce-fusion. The autoregressive baseline measured in [msg 11787] confirmed 132.7 tok/s at C=1 and 3870 tok/s at C=64, with power draw hovering around 360W per GPU—precisely one-third of the 1100W thermal design power.

This is the state of knowledge entering [msg 11788].

The Message: Reasoning and Decisions

[msg 11788] opens with an extended reasoning block that synthesizes the available data into a coherent mental model. The assistant writes:

Key insight confirmed: GPUs show 100% util but only 360W/1100W — this is HBM-bandwidth-bound (INT4 MoE expert weight loading), not compute-bound. DDTree gives 2.15x at C=1 (285 vs 133 auto). NVLS helps collectives, but EP could help by keeping experts resident.

This paragraph is dense with inference. The assistant is connecting three observations: (1) GPU utilization is pegged at 100%, (2) power draw is only 360W out of 1100W, and (3) the model is an INT4-quantized Mixture-of-Experts architecture. The conclusion that the system is HBM-bandwidth-bound follows from the nature of MoE inference: each token must be routed to a subset of experts, and those experts' weights must be loaded from HBM into the compute units. When weights are quantized to INT4, the compute-to-data ratio shifts dramatically—dequantization and matrix multiplication are fast relative to the memory transfer time. The GPU compute units spend most of their cycles waiting for data.

The reasoning then considers two architectural paths. Tensor parallelism (TP) splits each layer's weights across all 8 GPUs, meaning every token requires an AllReduce operation to synchronize partial results. On PCIe, this AllReduce is cripplingly slow. On NVLink, it is much faster, but the fundamental issue remains: each GPU still loads only a fraction of the expert weights per token, and the collective communication overhead grows with batch size.

Expert parallelism (EP), by contrast, places complete experts on each GPU and routes tokens via all-to-all communication. Each GPU keeps its assigned experts resident in HBM, eliminating redundant weight loading across devices. The tradeoff is that EP introduces communication at the token-routing layer rather than at the weight-gradient layer. With NVLink's NVLS (NVIDIA Live System) multicast support, this all-to-all communication can be accelerated significantly.

The assistant's decision tree at this point is worth examining. The options considered include:

  1. Push concurrency higher on the existing TP8 configuration to see if throughput scales and power rises. If the system is simply under-loaded at C=64, higher concurrency might saturate compute and raise power draw.
  2. Try EP8 + NVLS to keep experts resident and reduce HBM bandwidth pressure.
  3. Try NVLS collectives alone on TP8 to accelerate AllReduce without changing the parallelism strategy. The assistant chooses option 2, but with a twist: it also increases max-running-requests from 128 to 256, hedging against the possibility that the system is under-loaded. This is a pragmatic decision that tests two hypotheses simultaneously: that EP reduces memory bandwidth pressure, and that higher concurrency improves throughput.

The Implementation: Deploying EP8 + NVLS

The bash command in [msg 11788] is remarkably concise for the complexity of the operation. It performs the following steps:

  1. Stops the autoregressive baseline service (sglang-k26-auto.service).
  2. Copies the DDTree service definition to a new variant (sglang-k26-ddtree-ep.service).
  3. Injects three new flags into the ExecStart line: --ep-size 8, --enable-nccl-nvls, and --enable-flashinfer-allreduce-fusion.
  4. Increases --max-running-requests from 128 to 256.
  5. Updates the service description for clarity.
  6. Reloads systemd and starts the new service.
  7. Polls for readiness with a 15-second interval, checking for failure or successful generation. The choice to copy rather than modify the existing service file is a deliberate safety measure: it preserves the working TP8 configuration as a fallback. The assistant also prints the modified ExecStart line for verification, showing:
ExecStart=/root/venv_sglang211/bin/python -m sglang.launch_server --model-path /root/models/Kimi-K2.6 --port 30001 --host 0.0.0.0 --tp-size 8 --ep-size 8 --enable-nccl-nvls --enable-flashinfer-allreduce-fusion --mem-fraction-static 0.88 --context-length 32768 --max-running-requests 256 --page-size 1 --num-continuous-decode-steps 8 --attention-backend triton --trust-remote-code --grammar-backend none --skip-server-warmup --speculative-algorithm DDTREE --speculative-draft-model-path /root/models/K...

The combination of --tp-size 8 and --ep-size 8 is notable. In SGLang's architecture, TP and EP are not mutually exclusive; they can be combined. With TP8+EP8, each GPU holds 1/8 of each layer's weights (from TP) and 1/8 of the experts (from EP). The NVLS flag accelerates the all-to-all communication that EP requires, while the flashinfer allreduce fusion flag optimizes the AllReduce operations that TP still performs on non-MoE layers.

Assumptions Embedded in the Decision

The assistant's reasoning in [msg 11788] rests on several assumptions, some explicit and some implicit.

Assumption 1: The bottleneck is HBM bandwidth, not compute. This is the central hypothesis. The evidence—100% GPU utilization with 360W power draw—strongly supports it, but there are alternative explanations. The low power draw could also indicate that the GPU is power-capped, thermally throttled, or that the workload's compute-to-memory ratio is such that the cores are busy but not doing heavy arithmetic (e.g., they are stalled on memory operations, which nvidia-smi counts as utilization). The assistant implicitly assumes that "utilization" in nvidia-smi reflects active compute rather than memory-stalled cycles, which is a known subtlety of GPU utilization metrics.

Assumption 2: EP reduces HBM bandwidth pressure. This is true in theory: if each GPU keeps its assigned experts resident, tokens are routed to the correct GPU rather than loading all expert weights on all GPUs. However, the all-to-all communication that EP introduces consumes NVLink bandwidth and adds latency. The assistant assumes that NVLS multicast makes this communication cheap enough that the net effect is positive. This is a testable hypothesis, not a certainty.

Assumption 3: Higher concurrency (256 max-running-requests) will improve throughput. The assistant is implicitly assuming that the system is under-loaded at C=64. If the bottleneck is truly HBM bandwidth, higher concurrency might not help—it would just increase queueing delay without improving throughput. The assistant hedges by testing this alongside the EP change.

Assumption 4: The service will start successfully with these flags. This is not guaranteed. The combination of EP8, NVLS, and DDTree speculative decoding may expose bugs in SGLang's handling of these features together. The polling loop with failure detection is a necessary safety net.

Input Knowledge Required

To fully understand [msg 11788], the reader needs knowledge spanning several domains:

GPU architecture and power modeling. The relationship between GPU utilization, power draw, and workload type is foundational. A GPU drawing 360W out of 1100W while showing 100% utilization is in a memory-bandwidth-bound regime—the cores are active but stalled on memory loads, which consumes less power than compute-heavy operations like FP16 matrix multiplication.

MoE inference characteristics. Mixture-of-Experts models like Kimi K2.6 have sparse feedforward layers where each token activates only a subset of experts. This creates a unique memory access pattern: the expert weights must be loaded from HBM for each token, but the compute per weight is low (especially with INT4 quantization). The memory wall is the dominant factor.

Parallelism strategies for LLM inference. Tensor parallelism splits individual operations across GPUs, requiring AllReduce after every layer. Expert parallelism splits experts across GPUs, requiring all-to-all communication for token routing. Pipeline parallelism splits layers across GPUs, creating pipeline bubbles. Each strategy has different communication patterns and hardware requirements.

NVLink and NVLS. NVLink is NVIDIA's high-speed GPU-to-GPU interconnect, providing much higher bandwidth than PCIe. NVLS (NVIDIA Live System) is a multicast capability that allows one GPU to send data to multiple GPUs simultaneously, accelerating collective operations like AllReduce and all-to-all.

SGLang server configuration. The flags --tp-size, --ep-size, --enable-nccl-nvls, --enable-flashinfer-allreduce-fusion, and --speculative-algorithm DDTREE are specific to SGLang's server. Understanding their semantics requires familiarity with SGLang's architecture and command-line interface.

DDTree speculative decoding. DDTree is a speculative decoding algorithm that uses a small draft model to propose multiple candidate tokens in parallel, then verifies them against the target model. The budget and topk parameters control the tree structure and verification cost.

Output Knowledge Created

[msg 11788] produces several forms of knowledge:

A testable hypothesis about EP8+NVLS on B300 hardware. The message sets up an experiment whose results will either confirm or refute the hypothesis that expert parallelism with NVLS multicast improves throughput for DDTree speculative decoding on HBM-bound MoE models.

A reusable service configuration. The sglang-k26-ddtree-ep.service file becomes a documented configuration that can be compared against the TP8 baseline. The assistant prints the full ExecStart line, creating an audit trail.

A methodology for diagnosing bottlenecks. The reasoning in [msg 11788] demonstrates a systematic approach: measure utilization and power, identify the bottleneck regime, hypothesize a solution, design an experiment, and execute. This methodology is transferable to other inference optimization problems.

A baseline for comparison. The autoregressive baseline measurements (132.7 tok/s at C=1, 3870 tok/s at C=64) and the DDTree TP8 measurements (285 tok/s at C=1, 3795 tok/s at C=64) provide the reference points against which the EP8+NVLS results will be evaluated.

A documented decision point. The reasoning captures why the assistant chose EP8+NVLS over other options (higher concurrency alone, NVLS on TP8 without EP). This documentation is valuable for future debugging: if the EP8+NVLS experiment fails, the assistant can revisit the decision tree.

The Thinking Process: A Closer Look

The reasoning in [msg 11788] reveals a sophisticated mental model of the inference stack. The assistant does not simply react to the power draw observation; it connects it to the specific characteristics of INT4 MoE models, the communication patterns of different parallelism strategies, and the capabilities of NVLink hardware.

The structure of the reasoning is worth examining:

  1. Observation: GPUs at 100% util, 360W/1100W.
  2. Interpretation: HBM-bandwidth-bound, not compute-bound.
  3. Implication: DDTree's 2.15× speedup at C=1 is impressive, but the advantage vanishes at C=64 because the verification overhead adds memory traffic without improving the fundamental bottleneck.
  4. Hypothesis: EP could help by keeping experts resident, reducing redundant HBM loads.
  5. Action: Deploy EP8 + NVLS + higher concurrency.
  6. Verification plan: The polling loop and subsequent benchmarking will test the hypothesis. The assistant also implicitly rejects an alternative hypothesis: that higher concurrency alone would solve the problem. By bundling the EP change with increased max-running-requests, the assistant creates a compound experiment that cannot distinguish between the two effects. This is a methodological weakness—if throughput improves, it will be unclear whether EP or higher concurrency (or both) caused the improvement. However, in the context of a fast-paced deployment effort where time is limited, this pragmatic tradeoff is reasonable. The assistant can always run follow-up experiments to isolate the effects.

Potential Mistakes and Incorrect Assumptions

Several aspects of [msg 11788] warrant critical examination.

The conflation of GPU utilization with compute activity. As noted earlier, nvidia-smi reports utilization as the percentage of time during which at least one kernel was executing. A GPU can show 100% utilization while being completely memory-bound—the cores are active but stalled on memory loads, consuming less power and doing less useful work than the utilization percentage suggests. The assistant correctly interprets this as HBM-bound behavior, but the reasoning would be stronger if it acknowledged this subtlety explicitly.

The compound experiment design. Testing EP and higher concurrency simultaneously means the results will be ambiguous. If throughput improves, was it EP, higher concurrency, or the interaction between them? A cleaner experiment would test one variable at a time. The assistant's defense would be that time is limited and the service restart takes ~6 minutes, making sequential experiments costly.

The assumption that NVLS is beneficial for EP. NVLS multicast accelerates collective operations where one GPU sends data to many. In EP's all-to-all communication, each GPU sends tokens to specific other GPUs—a permutation, not a broadcast. NVLS may not provide the same benefit for all-to-all as it does for AllReduce. The assistant includes --enable-flashinfer-allreduce-fusion alongside NVLS, suggesting some awareness that different communication patterns need different optimizations.

The risk of flag incompatibility. The combination of --ep-size 8, --enable-nccl-nvls, and --enable-flashinfer-allreduce-fusion with DDTree speculative decoding may not have been tested by the SGLang team. The polling loop's failure detection is essential, but the assistant does not appear to have checked whether these flags are compatible with the DDTree algorithm specifically.

Conclusion

[msg 11788] captures a moment of architectural insight in an ML inference optimization campaign. The assistant synthesizes observations about GPU utilization, power draw, and model architecture into a coherent hypothesis about the bottleneck, then acts on that hypothesis by deploying a new parallelism configuration. The message is a case study in diagnostic reasoning under uncertainty: the assistant must interpret ambiguous signals, weigh competing hypotheses, and make pragmatic decisions under time pressure.

The decision to pivot from tensor parallelism to expert parallelism with NVLS multicast is not obviously correct—it rests on assumptions about HBM bandwidth, communication patterns, and software compatibility that could prove wrong. But the reasoning is transparent, the experiment is well-defined, and the verification plan is in place. Whether the EP8+NVLS configuration succeeds or fails, the knowledge gained will be valuable.

In the broader context of the conversation, [msg 11788] represents a shift from measurement to action. The assistant has spent several messages gathering data—benchmarking, measuring power, checking CUDA graph status. Now it uses that data to make a strategic decision. This pattern—measure, interpret, hypothesize, experiment—is the essence of systems optimization, and [msg 11788] exemplifies it beautifully.

The message also highlights the importance of domain knowledge in ML infrastructure. Understanding why a GPU draws 360W instead of 1100W requires knowledge of GPU architecture, memory hierarchies, and workload characteristics. Connecting that observation to the choice between TP and EP requires understanding MoE inference, communication collectives, and interconnect topology. [msg 11788] is a testament to the depth of expertise required to optimize large-scale model deployment.