The B300 Verdict: Validating Bigger Trees, Hitting the sm_103 Wall

Introduction

In the high-stakes world of speculative decoding for large language models, few moments are as satisfying—or as instructive—as when a carefully crafted hypothesis meets hard benchmark data. The assistant's message at index 11812 delivers exactly that: a concise, data-packed summary of deploying Kimi K2.6 with DDTree (Draft-Tree) speculative decoding on an 8× NVIDIA B300 SXM6 NVLink machine, and the bittersweet discovery that while the core idea is sound, a hardware-specific CUDA graph bug blocks the path to full realization. This message is the capstone of an intense investigation spanning multiple GPU architectures, parallelism strategies, and debugging rabbit holes, and it distills weeks of engineering into a few paragraphs that simultaneously celebrate a 2.15× speedup and lay out the roadmap for the next phase.

The Message in Context

To understand why this message matters, one must appreciate the journey that led to it. The broader conversation (segment 64 of the session) had been a marathon of infrastructure recovery, parallelism optimization, and speculative decoding deployment. The assistant had already:

The Message: A Data-Rich Culmination

The subject message is the assistant's final summary after completing the B300 deployment and benchmarking. It is structured as a compact report with five key sections: deployment methodology, benchmark results, validation of the budget hypothesis, power analysis, and future levers. Let us examine each in turn.

Deployment on B300 NVLink

The assistant reports deploying K2.6 + DDTree on 8× B300 SXM6 by streaming the working CT200 virtual environment directly (patches included), installing python3-dev, using --skip-server-warmup (because K2.6's vision tower requires flash_attn.cute which text-only serving never touches), and downloading moonshotai/Kimi-K2.6 via aria2 with 16 connections and 16 segments at approximately 575 MiB/s—three times faster than Hugging Face's hf_transfer downloader. The 590 GB model was downloaded in about 17 minutes.

This deployment section reveals several important assumptions and decisions. First, the assistant chose to stream the entire virtual environment from the CT200 machine rather than rebuilding it from scratch on B300, which saved enormous time but required that all patches and fixes were already baked into that environment. Second, the --skip-server-warmup flag was a pragmatic workaround: the vision tower warmup would have crashed due to the missing flash_attn.cute dependency, but since the deployment was text-only, the warmup was unnecessary. Third, the use of aria2 over hf_transfer reflects a practical optimization—the assistant recognized that the Hugging Face downloader was bottlenecked and switched to a more aggressive multi-connection tool.

Benchmark Results: The Numbers That Matter

The assistant presents a clean results table for TP8 (tensor parallelism across 8 GPUs) with greedy decoding:

| Config | C=1 | C=64 | C=128 | Accept | Coding | |---|---|---|---|---|---| | Autoregressive | 133 | 3870 | — | — | — | | DDTree b8 + graphs | 285 | 3795 | — | 4.48 | 5/5 | | DDTree b8 + NVLS + graphs | 303 | 4006 | 4723 | 4.36 | — |

The headline result is a 2.15× speedup over autoregressive at concurrency 1 (303 vs 133 tok/s), approximately double the PCIe box's single-stream throughput, and roughly four times the aggregate throughput. NVLink makes TP8 the clear winner—the expert parallelism advantage that dominated on PCIe disappears when GPUs can communicate at 900 GB/s instead of being bottlenecked by PCIe bandwidth. NVLS (NVLink SHARP, a hardware-accelerated collective communication primitive) provides a free ~5% boost.

These numbers are significant not just for their absolute values but for what they reveal about the architecture. The B300's NVLink interconnect fundamentally changes the parallelism calculus: on PCIe, EP8 won because it avoided AllReduce on MoE layers across the slow PCIe bus. On NVLink, TP8 is optimal because the interconnect is fast enough that tensor parallelism's communication overhead is negligible, and it avoids the pipeline bubbles and load imbalance that plague expert parallelism.

Validating the User's Hypothesis

The assistant directly addresses the user's budget question with a clear verdict: "you were right, with a caveat." The core finding is that bigger budgets do lift acceptance: budget=16 in eager mode achieves an acceptance length of 5.3–6.4 tokens per step, compared to budget=8's 4.48, and passes all 5 coding tests. The DDTree logic is correct, and the wider tree genuinely helps the draft model propose more tokens that the target model accepts.

But the caveat is a showstopper: CUDA graphs crash for any budget > 8 on sm_103. Budget=12 and budget=16 produce illegal memory access errors; budget=32 produces garbage output (acceptance collapses to 1.0, meaning every draft token is rejected). The assistant correctly diagnoses this as an sm_103-specific CUDA graph/Triton kernel bug in the tree-verify path at query length greater than 9, not a flaw in the DDTree algorithm itself. Eager mode works correctly but loses the graph speedup (b16 eager achieves only 111 tok/s vs b8+graphs at 285 tok/s at C=1), so net throughput does not beat b8+graphs.

There is also a secondary limitation: maxreq × (budget + 1) > ~1200 hits a CUBLAS limit in the MLA (Multi-head Latent Attention) absorb GEMM on sm_103. This means that at high concurrency with large budgets, the matrix multiplication kernel itself runs into architectural constraints.

This finding is particularly valuable because it separates the algorithmic question from the engineering question. The user's intuition about larger trees improving acceptance was correct—the DDTree algorithm benefits from wider search. The blocker is purely an implementation issue: the CUDA graph capture mechanism, which records GPU kernel launches for replay, does not correctly handle the tree-verify kernel at larger sequence lengths on the sm_103 architecture. This is a deep, architecture-specific bug that would require significant engineering effort to fix.

Power Analysis: Confirming HBM-Bound Workload

The assistant reports a critical power measurement: 100% GPU utilization but only 360–460 Watts out of a 1100 Watt thermal design power. This confirms the user's read that the workload is HBM-bandwidth-bound—the GPUs are fully busy but drawing far less power than their maximum because they are waiting on memory, not computing. The INT4 weight-only MoE decode is memory-bound: the GPUs have plenty of spare floating-point compute capacity (FLOPs) that goes unused because the bottleneck is reading weights and activations from HBM.

This is exactly why larger DDTree budgets should be nearly free here. The tree verification step involves additional compute—running the target model's forward pass on multiple candidate continuations—but if the GPU has spare compute capacity that is currently idle due to memory bandwidth limitations, the extra work can be overlapped with the existing memory waits. The fact that the GPU is at 100% utilization but only 40% of its power ceiling means there is genuine headroom for additional computation without increasing the memory bottleneck.

Two Levers Forward

The assistant concludes with a clear-eyed assessment of the path forward, identifying two levers to exploit the spare compute:

  1. Fix the sm_103 CUDA graph tree-verify kernel for q_len > 9: This would unlock budget=16 with graph speed, estimated to reach ~370 tok/s at C=1 (a ~30% improvement over b8+graphs' 285 tok/s). The estimate is based on the ratio of acceptance lengths: if b16 achieves accept 5.8 vs b8's 4.48, and the verify cost is similar (which the compute-rich B300 should absorb), throughput scales proportionally.
  2. Build a deeper-block drafter (block_size > 8): This is described as "the more fundamental win." The current drafter has a depth of 7 (block_size=8 with 6 draft layers), which limits the maximum number of tokens that can be accepted per step. A deeper-block drafter would raise this ceiling, potentially allowing even larger speedups regardless of the budget limitation. The first lever is a targeted engineering fix; the second is a model architecture improvement. The assistant implicitly prioritizes the second as more impactful, but the first is more immediately actionable within the existing infrastructure.

Assumptions and Knowledge Required

To fully understand this message, one needs substantial background knowledge:

Output Knowledge Created

This message creates several pieces of actionable knowledge:

  1. Quantified B300 DDTree performance: Concrete throughput numbers for autoregressive, b8+graphs, and b8+NVLS+graphs configurations across multiple concurrency levels, with acceptance lengths and coding correctness scores.
  2. Validation of the budget hypothesis: Experimental proof that larger budgets improve acceptance rates (5.3–6.4 vs 4.48), with the caveat that CUDA graphs break at budget > 8 on sm_103.
  3. Identification of a specific, reproducible bug: The sm_103 CUDA graph tree-verify kernel crash at q_len > 9, with distinct failure modes (illegal memory access at b12/b16, garbage output at b32).
  4. Power characterization: Confirmation that the workload is HBM-bandwidth-bound (100% util, 360–460 W / 1100 W), with spare compute capacity available.
  5. Parallelism architecture guidance: Evidence that NVLink makes TP8 optimal, eliminating the EP advantage seen on PCIe, and that NVLS provides a free ~5% boost.
  6. A prioritized roadmap: Two concrete levers for improvement, with estimated upside (~370 tok/s for the CUDA graph fix, unknown but potentially larger for deeper-block drafter).
  7. A complete reproduction package: All findings, result JSONs, service unit files, and manifests saved to /data/dflash/k26-ddtree-repro/, ensuring reproducibility.

Mistakes and Incorrect Assumptions

The message is remarkably accurate, but a few points warrant scrutiny:

The estimate of ~370 tok/s for b16+graphs assumes that the verify cost scales proportionally with acceptance length and that the compute-rich B300 absorbs the extra work. This is a reasonable first-order approximation, but it ignores that larger budgets increase the verify cost per step (more candidate tokens to verify), which could reduce the net gain. The actual throughput improvement might be closer to 20–25% than the estimated 30%.

The assistant also implicitly assumes that fixing the sm_103 CUDA graph bug is feasible within the existing SGLang/Triton framework. This may be optimistic—the bug could be in NVIDIA's CUDA graph capture implementation for sm_103, which would be outside the team's control. Alternatively, it could require changes to Triton's kernel code generation that are non-trivial.

The characterization of a "deeper-block drafter" as "the more fundamental win" is correct in principle but underspecified. A deeper-block drafter would require retraining the draft model with a larger block_size, which is a significant investment. The assistant does not address whether such a model exists, whether the training infrastructure is available, or what the expected acceptance ceiling would be.

The Thinking Process

The reasoning visible in the preceding messages (11805–11811) reveals a methodical, hypothesis-driven approach. The assistant systematically:

  1. Formulated a hypothesis: Larger budgets should improve acceptance rates because the tree covers more candidate continuations.
  2. Designed experiments: Tested b8, b12, b16, and b32 with and without CUDA graphs.
  3. Isolated variables: When b16+graphs crashed, the assistant tested b16 without graphs to separate the algorithmic question from the implementation bug.
  4. Narrowed the bug: Tested b12 as an intermediate point to determine the exact threshold (budget > 8, i.e., q_len > 9).
  5. Checked buffer sizing: Examined the cuda_graph_custom_mask buffer allocation in SGLang's Triton attention backend to rule out a simple memory overflow.
  6. Characterized the failure modes: Noted that b12/b16 produce illegal memory access while b32 produces garbage, suggesting a kernel sensitivity to query length rather than a buffer sizing issue.
  7. Made a pragmatic call: Reverted to the stable b8+NVLS+graphs configuration and documented the findings rather than continuing to chase the sm_103 bug. This is textbook debugging methodology: form a hypothesis, test it, isolate variables, narrow the search space, and make a pragmatic decision when the root cause is too deep to fix quickly.

Conclusion

The assistant's message at index 11812 is a masterclass in concise, data-driven technical communication. It delivers a clear verdict on the user's hypothesis (correct, with a caveat), provides actionable benchmark data, identifies a specific engineering blocker, and lays out a prioritized roadmap. The B300 deployment validated that DDTree speculative decoding can achieve 2.15× speedup over autoregressive on NVLink-connected hardware, that larger budgets genuinely improve acceptance rates, and that the sm_103 CUDA graph bug is the critical bottleneck preventing further gains. The message closes a major investigation cycle while opening the next one, setting the stage for either fixing the graph bug or pursuing a deeper-block drafter—or, as the assistant's subsequent work would show, pivoting to building a custom C/C++/CUDA inference stack to bypass these limitations entirely.