Chunk 64.2

The assistant deployed the K2.6 + DDTree stack on a new 8× B300 SXM6 machine (NVLink, 275 GB each, sm_103) by streaming the working CT200 virtual environment and downloading the 590 GB model via aria2 at 575 MiB/s. After fixing a Triton JIT compilation issue (missing `Python.h`) and skipping the vision tower warmup (which required unavailable `flash_attn.cute`), the service launched successfully. Benchmarking confirmed NVLink's massive advantage: DDTree with budget=8, NVLS, and CUDA graphs achieved 303 tok/s at C=1 (2.15× over the 133 tok/s autoregressive baseline) and scaled to 4723 tok/s at C=128, with 5/5 coding correctness. NVLS provided a free ~5% boost, and NVLink made TP8 the clear winner over EP for this workload. The user's hypothesis that larger budgets would better utilize the B300's spare compute was confirmed: budget=16 in eager mode lifted acceptance to 5.3–6.4 tokens per step (vs b8's 4.48) and passed all coding tests, proving the DDTree logic is sound. However, a critical sm_103-specific CUDA graph bug blocked realizing this gain—any budget > 8 caused illegal memory accesses (b12/b16) or garbage output (b32) during the tree-verify graph capture, while eager mode lost the 3.8× graph speedup, resulting in lower net throughput than b8+graphs. The workload was confirmed to be HBM-bandwidth-bound (100% GPU util but only 360–460 W / 1100 W), meaning the compute is genuinely idle and waiting on memory, validating the potential of optimized kernels. The assistant downloaded all B300 benchmark artifacts and wrote a comprehensive DDTree findings report (`DDTREE_FINDINGS_REPORT.md`) structured to directly support the next phase. The report covers the algorithm mechanics, the three SGLang bug fixes, cross-platform performance comparisons (PCIe vs NVLink), the HBM-bound bottleneck analysis, and a prioritized roadmap for a custom C/C++/CUDA inference stack targeting the PRO6000 box first. This stack aims to eliminate Python overhead, fuse dequantization with MoE GEMM, build a custom tree-attention MLA verify kernel to bypass the sm_103 graph instability, and move tree construction to the GPU, with concrete performance targets to beat (150–170 tok/s, accept ~4.5).

The NVLink Breakthrough: Deploying DDTree Speculative Decoding on B300 and the Road to a Custom Inference Stack 2178 words

Message Articles