Segment 64

The assistant resolved CUDA/FlashInfer SM120 compatibility issues on Blackwell GPUs, benchmarked four parallelism strategies (TP8, PP8, EP8, EP4) for Kimi K2.6 on PCIe-connected PRO6000 GPUs, finding EP8 best for single-stream throughput (65 tok/s) and EP4 for peak aggregate (~1531 tok/s). Deployed DFlash speculative decoding with sliding window attention, achieving 86 tok/s at C=1 and 4/5 coding correctness after fixing a code extraction bug. Then deployed the same stack on an 8× B300 SXM6 NVLink machine, where DDTree with budget=8 reached 303 tok/s at C=1 (2.15× over baseline) and scaled to 4723 tok/s at C=128, with 5/5 coding correctness. Larger budgets (b16) improved acceptance but were blocked by a CUDA graph bug on sm_103. The assistant documented all findings in DDTREE_FINDINGS_REPORT.md, including cross-platform comparisons, bottleneck analysis, and a roadmap for a custom C/C++/CUDA inference stack.

The DDTree Optimization Campaign: Deploying Kimi K2.6 with Speculative Decoding Across PCIe and NVLink Blackwell GPUs 4359 words

Chunks

• The Optimization Gauntlet: From CUDA Toolkit Recovery to Parallelism Benchmarking and DFlash Deployment for Kimi K2.6
  2193 words
• The DDTree CUDA Graph Fix: Unlocking Speculative Decoding at 109 tok/s on PCIe Blackwell
  2134 words
• The NVLink Breakthrough: Deploying DDTree Speculative Decoding on B300 and the Road to a Custom Inference Stack
  2178 words
• From Greedy to Sampling: The Architecture of Choice in DDTree Speculative Decoding
  2622 words
• From Config Sweep to Reproduction Package: The DDTree Optimization Campaign for Kimi K2.6
  1701 words
• From PCIe to NVLink: The DDTree Deployment Journey Across Two GPU Worlds
  2644 words