Scaling Speculative Decoding to the Cluster: Expert Parallelism on B300/GB300

"Now, what if we have a B300 / GB300 cluster with e.g. 16/36/72 GPUs to spread around the experts? HGX IB vs NVL72?"

This single sentence, posed by the user at message index 12044, marks a dramatic shift in scope. After dozens of messages spent meticulously building a native C/C++/CUDA DDTree inference engine, diagnosing throughput regressions, and estimating single-stream performance on a single B300 GPU, the user now pivots to the cluster level. The question is brief—barely twenty words—but it carries enormous weight. It asks the assistant to reason about scaling a speculative decoding system across 16, 36, or 72 GPUs, to compare two radically different interconnect topologies (HGX InfiniBand versus NVL72), and to consider how expert parallelism (EP) changes the performance calculus when the drafter and target model are distributed across multiple nodes.

The Reasoning and Motivation Behind the Question

The user's question did not emerge from a vacuum. It is the natural culmination of a long chain of analysis that began with single-GPU performance characterization on the PRO 6000 (an 8× RTX PRO 6000 Blackwell workstation), moved through detailed B300 throughput estimation, and now arrives at the frontier question: what happens when we scale beyond one GPU?

In the immediately preceding messages ([msg 12039] through [msg 12043]), the assistant had built a calibrated throughput estimator (estimate_b300.py) that projected single-stream B300 performance at roughly 303 tok/s with the current block-8 drafter, and potentially 815–1090 tok/s with a stronger drafter and a native engine. These numbers are impressive, but they describe a single GPU serving a single request stream. The user's question implicitly recognizes that real-world deployment—serving many concurrent users, or processing very large models—requires distributing the workload across many GPUs.

The phrase "to spread around the experts" is the critical clue to the user's mental model. The Kimi K2.6 model, like other DeepSeek-variant architectures, uses a Mixture-of-Experts (MoE) layer with 384 experts (including shared experts). In a single-GPU setup, all expert weights must reside in the GPU's HBM, and each forward pass activates only a subset of experts via a routing mechanism. But when you have 16, 36, or 72 GPUs, you can distribute the experts across GPUs so that each GPU hosts only a fraction of the total expert parameters. This is expert parallelism (EP), and it is the dominant scaling strategy for MoE models because it reduces per-GPU memory pressure and, in theory, allows the aggregate bandwidth of all GPUs to be brought to bear on the expert-weight streaming problem.

The user is therefore asking: given that we now understand single-GPU performance, how does the picture change when we distribute experts across a cluster? And which cluster topology—standard HGX with InfiniBand, or the tightly coupled NVL72—gives better results?

The Two Topologies: HGX IB vs NVL72

To understand the question, one must understand what these two topologies represent.

HGX IB refers to a standard NVIDIA HGX baseboard (typically housing 8 GPUs) connected via InfiniBand to other HGX nodes. In a 16-GPU cluster, that would be two HGX nodes linked by IB; in a 36-GPU cluster, roughly 4–5 nodes; in a 72-GPU cluster, 9 nodes. InfiniBand provides high-bandwidth inter-node communication (typically 400–800 Gb/s per link for modern NDR IB), but it is a network fabric—communication between GPUs on different nodes requires traversing the NIC, the IB switch fabric, and the target NIC. Latency is measured in microseconds rather than the sub-microsecond latency of NVLink, and bandwidth is shared across the fabric.

NVL72 is NVIDIA's flagship NVLink-switched system, where 72 GPUs are connected in a single NVLink domain. NVLink provides direct GPU-to-GPU communication at very high bandwidth (900 GB/s bidirectional per GPU on B300, via NVLink 6 or similar) and extremely low latency. All 72 GPUs can communicate as if they were in a single giant server, without traversing any network fabric. The trade-off is that NVL72 is a single monolithic system—you cannot easily scale beyond 72 GPUs without breaking the NVLink domain.

The user's question implicitly asks: for expert parallelism with speculative decoding, which topology wins? The answer depends on the communication pattern of expert-parallel MoE inference, and specifically on how the AllReduce of the MoE output interacts with the DDTree speculative decode loop.

Input Knowledge Required

To fully grasp the user's question, one needs substantial background knowledge across several domains:

  1. MoE architecture: Understanding that the Kimi K2.6 model has 384 experts (including shared experts), that each forward pass routes tokens to a subset of experts (top-k routing), and that expert parallelism distributes the expert weight matrices across GPUs. The key cost is the AllReduce needed to combine expert outputs after the routed computation.
  2. Speculative decoding with DDTree: The DDTree (Draft-Draft Tree) mechanism builds a tree of candidate token sequences using a smaller draft model, then verifies all candidates in parallel with the target model. The "verify forward pass" processes the tree's tokens through all 61 layers of the target model, including the MoE layers. Under expert parallelism, this verify forward pass requires AllReduce communication at every MoE layer—not just at the end.
  3. Communication topology performance: The difference in bandwidth, latency, and bisection bandwidth between NVLink (within NVL72) and InfiniBand (across HGX nodes). NVLink provides ~900 GB/s bidirectional per GPU; a single NDR InfiniBand link provides ~50 GB/s (400 Gb/s) unidirectional. The ratio is roughly 18:1 in favor of NVLink for raw bandwidth.
  4. The Amdahl effect from the previous analysis: The assistant had just established that on B300, the MoE forward pass is extremely fast (~1.5–2 ms for M=9) and that overhead dominates. Scaling to expert parallelism introduces more overhead in the form of AllReduce communication, which could easily overwhelm the MoE compute savings if the interconnect is not fast enough.
  5. The native engine's capabilities: The assistant had built a native C/C++/CUDA engine that owns the verify-attention kernel, the tree-building kernel, and the tree-accept kernel. This engine is designed to minimize Python overhead and launch latency. The user's question implicitly asks whether this engine can be extended to handle the distributed communication patterns of expert parallelism.

Assumptions Embedded in the Question

The user makes several implicit assumptions:

First, that expert parallelism is the right scaling strategy for this workload. This is a reasonable assumption for MoE models—EP is the standard approach in production systems like vLLM and SGLang. However, the user does not consider tensor parallelism (TP) as an alternative for the cluster case. On B300 with NVLink, TP-8 was the preferred strategy within a single node (as documented in the B300 analysis). The question of whether to use TP within nodes and EP across nodes, or pure EP across all GPUs, is left implicit.

Second, that 16/36/72 GPUs are the relevant scales. These numbers are not arbitrary: 16 GPUs corresponds to two HGX nodes (2 × 8), 36 GPUs corresponds to roughly 4.5 HGX nodes (or half an NVL72), and 72 GPUs is a full NVL72. The user is probing the scaling envelope from small cluster to full NVL72.

Third, that the DDTree speculative decoding loop can be efficiently distributed. The verify forward pass under expert parallelism requires an AllReduce at every MoE layer (61 layers in Kimi K2.6). If each AllReduce takes even 10 microseconds on InfiniBand (which is optimistic for small messages), the cumulative overhead across 61 layers is ~0.6 ms per verify step. On NVL72 with NVLink, the same AllReduce might take 1–2 microseconds, adding only ~0.06–0.12 ms. This difference could be decisive.

Fourth, that the drafter model can also be distributed or replicated. The user does not specify whether the drafter (a smaller 1.5B-parameter model) would also be expert-parallel or simply replicated on each node. The drafter is small enough that replication is feasible, but the communication pattern of the drafter's forward pass also matters.

What the Question Does Not Say

The question is notably silent on several important dimensions:

The Thinking Process the Question Invites

The user's question does not contain reasoning text—it is a direct query. But it invites a specific kind of analytical reasoning that the assistant had already demonstrated in the B300 estimator. The expected response would:

  1. Decompose the verify step under EP: Model the per-layer cost as MoE_compute + AllReduce_communication, where the AllReduce cost depends on the interconnect topology (NVLink vs IB).
  2. Scale the MoE compute: With N expert-parallel GPUs, each GPU processes 1/N of the active expert weights. The MoE GEMM time shrinks accordingly, but the AllReduce cost is added at every layer.
  3. Compare the two topologies: For NVL72, NVLink's high bandwidth and low latency make the AllReduce cost nearly negligible, so the MoE compute scales almost linearly with GPU count. For HGX IB, the AllReduce cost across nodes is significant and may dominate at small batch sizes.
  4. Identify the crossover point: At some number of GPUs and some batch size, the AllReduce overhead on InfiniBand exceeds the MoE compute savings, and adding more GPUs hurts throughput. This is the classic strong-scaling Amdahl effect applied to distributed MoE.
  5. Consider the DDTree-specific twist: The verify forward pass processes M tokens (budget+1) through all layers. Under EP, the AllReduce must happen at every MoE layer, not just at the final output. For a tree of 33 tokens (budget=32) through 61 layers, that is 61 AllReduces per verify step. The cumulative cost of 61 small-message AllReduces on InfiniBand could easily exceed the MoE compute time, making expert parallelism counterproductive for speculative decoding.

Output Knowledge Created by This Question

Although the question itself is short, it generates a rich analytical response that creates significant new knowledge:

Mistakes and Potential Pitfalls

The user's framing, while insightful, contains some potential pitfalls:

Overlooking the batch-size dependence: Expert parallelism is most effective at large batch sizes where the MoE compute per GPU is substantial and the AllReduce cost is amortized. At C=1 (single stream), the MoE compute per GPU is tiny even at M=33, and the AllReduce overhead may dominate. The user's question does not specify C, which is a critical parameter.

Assuming uniform expert distribution: The user says "spread around the experts" but does not consider that some experts may be "hot" (frequently routed to) while others are "cold." Load imbalance across expert-parallel GPUs can cause straggler effects that reduce throughput. This is a well-known challenge in MoE serving.

Ignoring the shared expert: Kimi K2.6 has a shared expert that is always active. Under expert parallelism, the shared expert must either be replicated on every GPU (defeating part of the memory savings) or communicated across GPUs (adding overhead). The user does not mention this.

The NVL72 monolithic constraint: While NVL72 provides excellent performance, it is a single system. If the user's workload grows beyond 72 GPUs, they cannot incrementally add GPUs without breaking the NVLink domain. HGX IB, by contrast, scales to hundreds or thousands of GPUs. The user's question implicitly assumes that 72 GPUs is sufficient, but a production deployment might need more.

Conclusion

The user's question at message 12044 is a masterful pivot from micro-level optimization to macro-level architecture. It takes the hard-won knowledge of single-GPU DDTree performance—the MoE GEMM latencies, the overhead decomposition, the Amdahl effects—and asks what happens when we multiply the hardware by an order of magnitude. The question is deceptively simple, but answering it requires a deep synthesis of MoE architecture, interconnect topology, speculative decoding mechanics, and distributed systems scaling laws. It is the kind of question that separates a narrow optimization exercise from a genuine production deployment strategy.

The answer, when it comes, will likely reveal that NVL72's NVLink fabric is transformative for expert-parallel speculative decoding—not because it makes the MoE compute faster, but because it makes the AllReduce overhead small enough that the Amdahl ceiling moves far out. On HGX IB, the cumulative AllReduce cost across 61 MoE layers may cap throughput well below what the raw GPU compute could deliver. The user's question thus sets the stage for the next major analytical effort: building a distributed scaling model that accounts for interconnect topology, batch size, and the unique communication patterns of DDTree verification.