The NVLink Breakthrough: Deploying DDTree Speculative Decoding on B300 and the Road to a Custom Inference Stack

Introduction

In the high-stakes world of large language model inference, hardware topology is destiny. For months, the team deploying Kimi K2.6—a 590-billion-parameter Mixture-of-Experts model—had been fighting against the constraints of PCIe-connected Blackwell GPUs. The PCIe bus, with its limited bandwidth and high-latency cross-GPU communication, imposed a hard ceiling on speculative decoding performance. Every AllReduce operation on MoE layers became a bottleneck, and the promising DDTree (Draft-Draft Tree) algorithm, which evaluates multiple candidate token paths in parallel, struggled to realize its theoretical advantages.

Then everything changed. A new machine arrived: an 8× B300 SXM6 system with NVLink interconnect, 275 GB of HBM per GPU, and the sm_103 compute architecture. This article examines the pivotal deployment and benchmarking campaign that followed—a campaign that would validate the DDTree approach with a 2.15× speedup over the autoregressive baseline, uncover a critical CUDA graph bug specific to sm_103, confirm the HBM-bandwidth-bound nature of the workload, and culminate in a comprehensive findings report that charts the course for a custom C/C++/CUDA inference stack.

The Deployment: From PCIe to NVLink

The transition from the PCIe-connected RTX PRO 6000 cluster to the NVLink-connected B300 SXM6 machine was not merely a hardware swap—it was a fundamental shift in the performance landscape. The assistant's first task was to replicate the working environment on the new machine. This involved streaming the CT200 virtual environment (which had been carefully tuned with CUDA 13.0, SGLang nightly builds, and all the custom patches accumulated over weeks of debugging) and downloading the 590 GB model via aria2 at an impressive 575 MiB/s.

The deployment was not without friction. The new machine's sm_103 architecture required a Triton JIT compilation fix—the Python.h header was missing from the build environment, a common issue when deploying across different CUDA toolkit versions. The assistant also had to skip the vision tower warmup, which required flash_attn.cute—a kernel library that wasn't available on the B300. These were minor speed bumps compared to the weeks of CUDA toolkit and FlashInfer compatibility battles that had plagued the PCIe deployment, but they underscore the reality that no two GPU clusters are identical, and every deployment requires environment-specific adjustments.

Once the service launched, the assistant ran a comprehensive benchmark suite. The results were stunning.

The NVLink Advantage: 2.15× Speedup

With DDTree configured at budget=8, NVLS (NVLink Shared Memory) enabled, and CUDA graphs active, the B300 system achieved 303 tok/s at concurrency 1—a 2.15× speedup over the 133 tok/s autoregressive baseline. This was the validation the team had been seeking. On the PCIe system, DDTree had struggled to break even with the baseline at low concurrency, achieving only ~50 tok/s at budget=32 due to the AllReduce overhead dominating the compute budget. On NVLink, where cross-GPU communication bandwidth is an order of magnitude higher, the tree verification cost was amortized efficiently, and the speculative decoding gains were fully realized.

The scaling behavior was equally impressive. At concurrency 128, DDTree reached 4723 tok/s—nearly 3.5× the single-stream throughput. The NVLink interconnect allowed the continuous batching scheduler to keep all 8 GPUs saturated with work, and the CUDA graph replay eliminated the Python-level dispatch overhead that had been a significant bottleneck on the PCIe system.

A particularly noteworthy finding was the impact of NVLS. The assistant benchmarked DDTree with and without NVLink Shared Memory and found that NVLS provided a free ~5% boost across all configurations. This is the kind of optimization that requires no code changes—it's a hardware-level feature that transparently accelerates collective operations when GPUs are connected via NVLink.

The NVLink topology also flipped the parallelism calculus. On PCIe, expert parallelism (EP) had been the clear winner because it avoided AllReduce on MoE layers. On NVLink, where AllReduce is cheap, TP8 (tensor parallelism across 8 GPUs) became the superior choice for the speculative decoding workload. This is a critical lesson: parallelism strategy is not a fixed property of a model architecture—it must be reevaluated for each hardware platform.

The Larger Budget Hypothesis: Confirmed and Blocked

The user had hypothesized that larger DDTree budgets would better utilize the B300's spare compute capacity. The B300 GPUs, with their 275 GB of HBM and sm_103 architecture, have substantial compute throughput that might go underutilized if the tree is too small. The assistant tested this by running DDTree with budget=16 in eager mode (without CUDA graphs).

The results confirmed the hypothesis. Budget=16 lifted the average acceptance to 5.3–6.4 tokens per step, compared to budget=8's 4.48 tokens. The tree was exploring more paths and finding longer valid continuations. All coding tests passed (5/5 correctness), proving that the DDTree logic was sound and the larger budget was genuinely beneficial.

But there was a catch. When the assistant tried to use CUDA graphs with budget=16, the service crashed with an illegal memory access. The same crash occurred with budget=12. Budget=32 produced garbage output instead of a crash. The common thread: any budget > 8 caused failures during the tree-verify graph capture on sm_103.

This was a critical discovery. The CUDA graph infrastructure in SGLang, which provides approximately a 3.8× speedup over eager mode by capturing and replaying GPU kernel launch sequences, had a fundamental incompatibility with the DDTree verify path when the tree exceeded a certain size. The crash was not a simple bug—it was an architectural tension between CUDA graphs' requirement for fixed tensor shapes and DDTree's variable-size tree structure.

The assistant diagnosed the issue through a series of targeted experiments. The error signature—illegal memory accesses for budget=12 and budget=16, garbage output for budget=32—pointed to a memory layout problem during graph capture. The tree verification kernel, which processes all tree nodes simultaneously using a custom attention mask, was producing tensor shapes that the CUDA graph runner couldn't handle. The graph capture phase (which uses dummy inputs during warmup) somehow succeeded with compatible shapes, but the replay phase (which uses real draft outputs) produced different shapes, causing the crash.

The practical consequence was stark: with CUDA graphs, budget=8 achieved 303 tok/s (2.15× baseline). Without graphs (eager mode), budget=16 achieved only ~80 tok/s—lower than the b8+graphs configuration, despite the higher acceptance rate. The 3.8× graph speedup was essential to realizing DDTree's benefits, and it only worked reliably at budget=8.

The HBM-Bound Bottleneck: Compute Is Idle

One of the most revealing findings from the B300 benchmarking was the power and utilization analysis. The assistant monitored GPU utilization and power draw during DDTree inference and found that the GPUs were running at 100% utilization but only 360–460 W out of a 1100 W thermal design power (TDP). This is a textbook signature of an HBM-bandwidth-bound workload: the compute units are fully occupied (100% util) but they're spending most of their time waiting for data to arrive from HBM, not actually computing. The low power draw confirms that the compute pipelines are stalled on memory accesses.

This finding has profound implications. It means that the B300's compute capacity is genuinely idle during DDTree inference—the bottleneck is moving data between HBM and the compute units, not the compute itself. This validates the entire speculative decoding approach: if the system were compute-bound, adding more draft tokens to verify would increase latency proportionally. But since it's bandwidth-bound, the marginal cost of verifying additional tree nodes is low—the data has to be fetched from HBM anyway, and the extra compute is essentially free.

The HBM-bound analysis also explains why NVLS provided a free 5% boost: NVLink Shared Memory allows GPUs to share data without going through HBM, effectively increasing the effective bandwidth of the system. And it explains why TP8 won over EP on NVLink: the AllReduce operations that TP requires are bandwidth-efficient on NVLink, while EP's All-to-All communication pattern doesn't benefit as much from the shared memory.

Most importantly, the HBM-bound diagnosis directly motivates the next phase of the project: a custom C/C++/CUDA inference stack. If the bottleneck is memory bandwidth, then the optimization targets are clear: fuse operations to reduce memory traffic, eliminate Python overhead that adds latency between kernel launches, and build custom kernels that maximize HBM utilization. The assistant's findings report would explicitly cite this analysis as the foundation for the custom stack roadmap.

The Findings Report: A Roadmap for the Custom Stack

With the B300 benchmarking complete and the key insights documented, the assistant downloaded all benchmark artifacts and wrote DDTREE_FINDINGS_REPORT.md—a comprehensive document that synthesizes everything learned across both hardware platforms. The report is structured to directly support the next phase of the project: building a custom C/C++/CUDA inference stack.

The report covers:

Algorithm mechanics: A clear explanation of how DDTree works, including the tree construction (best-first search over the draft model's top-k predictions), the tree verification (batched forward pass with custom attention mask), and the commit logic (following the longest verified path).
The three SGLang bug fixes: Documentation of the CUDA graph sizing fix (the 8 vs 33 tensor mismatch), the None dtype diagnostic, and the tree verification path corrections. These fixes represent the accumulated debugging effort and serve as a reference for anyone maintaining the SGLang speculative decoding integration.
Cross-platform performance comparison: A side-by-side analysis of PCIe vs NVLink results. The PCIe system achieved ~50 tok/s at C=1 with DDTree (budget=32), while the NVLink system reached 303 tok/s (budget=8, with graphs). The comparison quantifies exactly how much performance the PCIe interconnect was leaving on the table.
The HBM-bound bottleneck analysis: The power and utilization data that proves the workload is memory-bandwidth-bound, not compute-bound. This is the key insight that justifies the custom stack approach.
A prioritized roadmap for the custom inference stack: The report lays out a concrete plan targeting the PRO6000 box first (since it's the more constrained platform and would benefit most from optimization). The roadmap includes: - Eliminate Python overhead: Replace the SGLang Python orchestration with a C/C++ runtime that directly manages GPU kernel launches, eliminating the 3.8× overhead that CUDA graphs currently mask. - Fuse dequantization with MoE GEMM: The current pipeline decompresses INT4 weights to FP16 before the GEMM operation, which doubles memory traffic. Fusing these operations would cut HBM bandwidth consumption by nearly half. - Build a custom tree-attention MLA verify kernel: The current DDTree verify path uses SGLang's attention backend, which is optimized for standard causal attention but not for tree-structured attention masks. A custom kernel could eliminate the sm_103 graph instability by design. - Move tree construction to the GPU: The current tree construction runs on CPU using Python's heapq, creating a serialization bottleneck at high concurrency. A GPU-based tree builder would eliminate this bottleneck and enable larger budgets. - Concrete performance targets: The report sets targets of 150–170 tok/s with acceptance of ~4.5 tokens per step—numbers that would represent a transformative improvement over the current PCIe baseline.

The Broader Significance

This chunk of the conversation represents a watershed moment in the project. The B300 deployment validated that DDTree speculative decoding can deliver substantial speedups (2.15×) when the hardware interconnect doesn't bottleneck the algorithm. The sm_103 CUDA graph bug, while frustrating, revealed a fundamental architectural tension that cannot be patched away—it requires a new approach. And the HBM-bound analysis provided the scientific foundation for that new approach.

The findings report is not just a retrospective document—it's a strategic blueprint. The decision to target the PRO6000 box first for the custom stack is telling: the more constrained platform has more to gain from optimization, and success there would prove the approach before scaling to the B300. The concrete performance targets (150–170 tok/s, accept ~4.5) give the engineering team clear, measurable goals.

For anyone studying ML inference optimization, this chunk offers a masterclass in the full lifecycle of performance engineering: from environment setup and dependency resolution, through benchmarking and bottleneck analysis, to strategic planning and roadmap creation. The assistant's methodical approach—deploy, benchmark, diagnose, document, plan—is a template for how to navigate the complex, multi-layered challenges of deploying cutting-edge models on novel hardware.

Conclusion

The NVLink breakthrough was exactly what the project needed: a clear demonstration that DDTree speculative decoding works, combined with a precise understanding of what's blocking further gains. The 2.15× speedup at budget=8 proves the algorithm's value. The CUDA graph bug at budget>8 defines the next engineering challenge. The HBM-bound analysis provides the theoretical framework for solving it.

The DDTREE_FINDINGS_REPORT.md document that emerged from this work is more than a report—it's a declaration of intent. The custom C/C++/CUDA inference stack it describes would bypass the limitations of the current SGLang-based approach entirely, replacing Python orchestration with a lean C++ runtime, fusing memory-bound operations, and building purpose-built kernels for tree-attention verification. The sm_103 graph instability that blocked larger budgets on the B300 would be irrelevant because the custom stack wouldn't use CUDA graphs at all—it would eliminate the Python overhead that makes graphs necessary.

In the end, the crash that seemed like a setback was actually a revelation. It told the team exactly where the architectural boundaries lay and what would be required to cross them. The B300 deployment proved that the destination is worth the journey.