From PCIe to NVLink: The DDTree Deployment Journey Across Two GPU Worlds

Introduction

In the high-stakes world of large language model inference optimization, deploying a 590-billion-parameter Mixture-of-Experts model like Kimi K2.6 with speculative decoding across heterogeneous GPU hardware is a journey fraught with infrastructure pitfalls, architectural surprises, and hard-won insights. This article synthesizes a remarkable multi-week engineering campaign documented across 33 messages in an opencode coding session, spanning two radically different hardware platforms—an 8× RTX PRO 6000 Blackwell system connected via PCIe, and an 8× B300 SXM6 system connected via NVLink—and culminating in a strategic pivot to build a custom C/C++/CUDA inference stack.

The arc of this work follows a clear trajectory: from infrastructure recovery and parallelism benchmarking, through speculative decoding deployment and systematic debugging of a critical architecture-specific CUDA graph bug, to the synthesis of findings in a comprehensive report and the decision to build a purpose-built inference engine. Each phase produced insights that shaped the next, and the entire journey exemplifies the disciplined engineering required to push inference performance to its limits.

Part I: Laying the Foundation — Infrastructure and Parallelism

The campaign began not with benchmarking but with infrastructure recovery. The assistant faced a cascade of CUDA toolkit compatibility issues on the Blackwell GPUs: FlashInfer rejected SM120, curand.h headers were missing, and Triton JIT compilation failed. These were resolved by cleanly installing the full CUDA 13.0 toolkit alongside the existing CUDA 12.8, then patching SGLang's Triton attention backend to support pipeline parallelism with non-zero start layers [1]. This unblocked all downstream work.

With the infrastructure stable, the assistant systematically benchmarked four parallelism strategies across the 8× RTX PRO 6000 PCIe-connected system: TP8 (pure tensor parallelism), PP8 (pipeline parallelism), EP8 (expert parallelism), and EP4 (expert parallelism with TP2 groups). The results validated a key architectural insight: on PCIe-connected hardware, expert parallelism dramatically outperforms tensor parallelism because it eliminates AllReduce communication on MoE layers across the slow PCIe bus [1]. EP8 improved single-request throughput from 26 to 65 tok/s, and EP4 reached a peak aggregate throughput of approximately 1531 tok/s at high concurrency. PP8 was disappointing due to pipeline bubbles, while TP8 with CUDA graphs hit 98 tok/s at C=1 but plateaued at ~1291 tok/s. The user's intuition that expert parallelism avoids PCIe AllReduce bottlenecks was empirically confirmed.

Part II: DFlash Speculative Decoding — First Deployment

With the parallelism strategy established, the assistant deployed DFlash speculative decoding using a DDTree (Draft-and-Diverge Tree) drafter. The drafter model, SubSir/Kimi-K2.6-DFlash-tmp-long, was a 6.5 GB, 6-layer transformer with block_size=8, trained as a drop-in replacement for the target model's early layers. The assistant enabled sliding window attention for the drafter with --speculative-dflash-draft-window-size 2048, which clamped the draft KV cache to the last 2048 tokens—matching the drafter's training window [1].

A comprehensive benchmark harness (bench_ddtree_matrix.py) was built to test across context lengths (60, 1024, 4096, 8192) and concurrency levels (1, 8, 32, 64), including a coding correctness evaluation. The first benchmark with budget=8, topk=4, and window=2048 achieved approximately 170 tok/s at C=1 with short context—the best single-stream result yet on the PCIe system. However, the initial coding evaluation showed 0/5 passes due to a code extraction bug: the thinking model outputs reasoning before a response marker, and the regex was grabbing an incomplete code block from the thinking section. After fixing the extraction to split on response and take the final code block, coding evaluation showed 4/5 passes at 162 tok/s (the single failure was a harness truncation edge case) [1].

The assistant then attempted a full config sweep across budget/topk/window combinations, but all failed to start due to a race condition in the service restart readiness check—the old process answered /v1/models briefly before being killed, while the new process was still loading weights for approximately 6 minutes. This debugging set the stage for the more robust orchestration patterns used later.

Part III: The B300 NVLink Machine — A New World

The campaign then shifted to a new 8× B300 SXM6 machine with NVLink interconnects, 275 GB of HBM per GPU, and sm_103 architecture. The assistant deployed the K2.6 + DDTree stack by streaming the working CT200 virtual environment and downloading the 590 GB model via aria2 at 575 MiB/s—three times faster than Hugging Face's hf_transfer downloader [2]. After fixing a Triton JIT compilation issue (missing Python.h) and skipping the vision tower warmup (which required unavailable flash_attn.cute), the service launched successfully.

The benchmark results on NVLink were transformative. DDTree with budget=8, NVLS (NVLink SHARP, a hardware-accelerated collective communication primitive), and CUDA graphs achieved 303 tok/s at concurrency 1—a 2.15× speedup over the 133 tok/s autoregressive baseline [2]. At high concurrency (C=128), throughput scaled to 4723 tok/s with 5/5 coding correctness. NVLS alone provided a free ~5% boost by enabling direct GPU-to-GPU communication over NVLink. The NVLink interconnect fundamentally changed the parallelism calculus: on PCIe, EP8 won because it avoided AllReduce across the slow bus; on NVLink, TP8 became the clear winner because the interconnect was fast enough that tensor parallelism's communication overhead was negligible [2][3].

The power analysis was equally revealing. The GPUs showed 100% utilization but only 360–460 W out of a 1100 W thermal budget—a classic signature of an HBM-bandwidth-bound workload. The INT4 weight-only MoE decode was memory-bound, not compute-bound: the tensor cores were idle, waiting on memory. This meant there was abundant spare compute capacity that larger speculative decoding trees could potentially exploit [3][4].

Part IV: The Budget Ceiling — Pushing Against CUDA Graph Limits

The user's sharp observation in message 11795 cut to the heart of the matter: "b8 is really low, will cause us to underutilise compute and best case equal sequential accept len, no? Can we push 16/32/64?" [4]. This question set off an intensive investigation into whether larger DDTree budgets could better utilize the B300's spare compute.

The assistant designed a budget sweep experiment, testing budget=16, 32, and 64 with topk=8. But every attempt to scale beyond budget=8 on the B300 failed in progressively more spectacular ways. Budget=16 crashed with cudaErrorIllegalAddress. Budget=32 appeared to run but produced garbled output—acceptance length stuck at exactly 1.0 (meaning no draft tokens were accepted) and the generated code was syntactically invalid, failing all 5 coding evaluations. Budget=64 crashed with CUBLAS errors [5][6].

The breakthrough came when the assistant formulated a critical hypothesis: perhaps the failures weren't fundamental kernel bugs at all, but artifacts of CUDA graph capture—a performance optimization that records GPU operations for replay, eliminating CPU launch overhead. By disabling CUDA graphs and testing budget=16 in eager mode, the assistant produced a definitive result: 5/5 coding correctness with acceptance length jumping to 5.3–6.4 tokens per step, compared to budget=8's 4.48 [7]. The DDTree logic was sound. The bug was specifically in CUDA graph capture on sm_103 at query lengths greater than 9.

This finding validated the user's hypothesis—larger budgets genuinely improved acceptance rates—while simultaneously revealing a critical infrastructure blocker. The assistant systematically investigated the root cause, examining the cuda_graph_custom_mask buffer sizing in SGLang's Triton attention backend [8][9], testing intermediate budgets like budget=12 (which also crashed) [10], and ultimately concluding that the bug was a deep, architecture-specific interaction between Triton-compiled attention kernels and CUDA graph capture on sm_103 [11]. The workload was confirmed to be HBM-bandwidth-bound, meaning the compute units were genuinely idle and the extra verification work from larger trees would be nearly free—if the graph bug could be fixed or bypassed [10][11].

Part V: The Strategic Pivot — From SGLang to Custom Inference

The user's response to this impasse was decisive and forward-looking. In message 11813, the user instructed: "Download artifacts from the B300 machine, write a report of ddtree findings so far, next we will most likely try to implement a C/C++/CUDA inference stack to maximally optimise ddtree inference specific to Kimi and the machine we're running on. First we target the pro6000 box." [12]

This marked a fundamental shift in strategy. Instead of continuing to fight SGLang's Python/Triton/CUDA-graph stack—debugging the sm_103 graph bug, patching Triton kernels, and working within the framework's constraints—the team would build something entirely new: a purpose-built inference engine designed for one model (Kimi K2.6) on one hardware configuration (the PRO6000 box first, then B300). The guiding principle was specialization over generality [12][13].

The assistant executed the user's instruction with methodical precision. It downloaded all B300 benchmark artifacts (six JSON files capturing autoregressive, b8t4, NVLS, b32, and b12 configurations) and journalctl excerpts showing acceptance metrics and CUDA graph behavior [14]. It then wrote the comprehensive DDTREE_FINDINGS_REPORT.md—a 14,229-byte document covering seven major sections: the model architecture, DDTree algorithm mechanics, three SGLang bug fixes discovered during deployment, cross-platform performance comparisons (PCIe vs NVLink), the HBM-bandwidth-bound bottleneck analysis, prioritized requirements for the custom C/C++/CUDA stack, and open items for future work [15][16].

The report identified five ranked optimization priorities for the custom stack:

  1. Own the decode loop — eliminate Python overhead and CUDA graph fragility
  2. Fused Marlin INT4 dequant + MoE GEMM — reduce memory traffic by fusing operations
  3. Custom tree-attention MLA verify kernel — bypass the sm_103 graph instability entirely
  4. GPU-side tree build — move tree construction from CPU heapq to GPU
  5. Fused drafter forward — integrate the drafter into the custom pipeline The concrete performance targets were to beat 150–170 tok/s at C=1 with an acceptance rate of approximately 4.5 tokens per step, with token-exact greedy and distribution-exact temperature correctness [16].

Part VI: Closing the Loop — Preservation and Handoff

The final phase of the investigation involved preserving all knowledge for the next phase. The assistant committed the B300 artifacts and findings report to the git repository with a detailed commit message that served as a miniature executive summary: "B300 DDTree b8+NVLS 303 tok/s C=1 (2.15x), 4723 C=128, coding 5/5; bigger budget lifts accept (b16 eager 5.3-6.4) but graphs crash >b8 on sm_103; HBM-bound." [17]

The commit itself required debugging—a path resolution issue caused the initial git add to fail silently [18]—but was ultimately successful after the assistant corrected the file paths [19]. The reproduction package at /data/dflash/k26-ddtree-repro/ was updated with a SHA256 manifest, and the B300 production service was verified healthy on its best stable configuration: budget=8, NVLS enabled, CUDA graphs enabled, maxreq=128, sliding window=2048 [20].

The user's final message was empty—an XML wrapper with no content [21]. This silence was itself meaningful: it signaled tacit approval, continuity, and readiness to proceed. The exploratory phase was complete. The engineering phase was about to begin.

Key Findings and Their Implications

Several findings from this campaign have broad implications for LLM inference optimization:

1. Interconnect topology dictates parallelism strategy. On PCIe, expert parallelism wins because it avoids AllReduce on MoE layers across the slow bus. On NVLink, tensor parallelism is optimal because the interconnect is fast enough that communication overhead is negligible. NVLS provides a free ~5% boost on NVLink [2][3].

2. Larger DDTree budgets improve acceptance rates. Budget=16 in eager mode achieved 5.3–6.4 tokens per step versus budget=8's 4.48, confirming that wider trees genuinely help the drafter propose more acceptable tokens. The block_size=8 depth cap (maximum depth 7) means larger budgets add width rather than depth, with diminishing returns [7][11].

3. CUDA graph capture is architecture-sensitive. On sm_103, CUDA graphs crash for any DDTree budget > 8 (query length > 9) during the tree-verify path. This manifests as illegal memory accesses at b12/b16 and garbage output at b32. Eager mode works correctly but loses the 3.8× graph speedup [10][11].

4. The workload is HBM-bandwidth-bound. 100% GPU utilization but only 33–42% power draw indicates the compute units are idle, waiting on memory. This means speculative decoding's extra compute is nearly free, validating the entire approach [3][4].

5. Python overhead is a massive tax. The 3.8× speedup from CUDA graphs over eager mode shows how much overhead the Python serving stack introduces. A C/C++/CUDA implementation that eliminates this overhead could approach graph-captured performance without the fragility [16].

Conclusion

The DDTree deployment journey across PCIe and NVLink GPU platforms represents a masterclass in systematic inference optimization. The assistant navigated infrastructure recovery, parallelism benchmarking, speculative decoding deployment, and architecture-specific kernel debugging with disciplined methodology—forming hypotheses, designing experiments, isolating variables, and making pragmatic decisions when root causes proved too deep for quick fixes.

The final verdict is both a validation and a challenge. The DDTree algorithm works correctly and delivers real speedups: 2.15× over autoregressive on NVLink hardware, with larger budgets demonstrably improving acceptance rates. But the sm_103 CUDA graph bug blocks the path to full realization, and the Python overhead of the existing serving stack limits throughput. The strategic pivot to a custom C/C++/CUDA inference stack is the natural response to these constraints—a decision to build a precision instrument for one model on one hardware platform, rather than continuing to fight the limitations of a general-purpose framework.

The artifacts preserved in the findings report and the reproduction package ensure that the hard-won knowledge from this campaign will inform the next phase. The silence of the user's final message is the silence of satisfaction—and of anticipation for what comes next.## References

[1] "Pivoting Under Pressure: How a Failed Expert Parallelism Experiment Led to a Breakthrough on NVLink" — Article on the EP8/TP8 parallelism pivot on B300.

[2] "The Pivot That Paid Off: Benchmarking NVLS-Enhanced DDTree on B300" — Article on the NVLS benchmark results (303 tok/s, 4723 tok/s).

[3] "Pushing Against the Ceiling: When Scaling an Inference Service Reveals Hidden Limits" — Article on the maxreq=256 crash and power analysis.

[4] "The Budget Ceiling: A User's Challenge to Underutilized Compute in Speculative Decoding" — Article on the user's question about larger budgets.

[5] "The Budget Ceiling: Diagnosing a CUBLAS Crash on the Path to Bigger Speculative Trees" — Article on the CUBLAS failure at maxreq=256.

[6] "Diagnosing the CUBLAS Ceiling: How One Message Unlocked the Path to Larger Speculative Decoding Budgets on B300" — Article on diagnosing the cuBLAS dimension limit.

[7] "The CUDA Graph Hypothesis: How Disabling a Feature Uncovered the Real Bottleneck in Speculative Decoding" — Article on the critical experiment disabling CUDA graphs.

[8] "The Critical Diagnostic: Isolating a CUDA Graph Capture Bug in DDTree Speculative Decoding on Blackwell B300" — Article on isolating the graph capture bug.

[9] "The Anatomy of a Surgical Debugging Probe: Tracing a CUDA Graph Buffer Bug in SGLang's DDTree Implementation" — Article on the source code probe.

[10] "The Budget Ceiling: Diagnosing CUDA Graph Instability for DDTree Speculative Decoding on Blackwell B300" — Article on the final diagnosis of the sm_103 graph bug.

[11] "The Final Verdict: When CUDA Graphs Break on sm_103" — Article on the conclusive finding that any budget > 8 crashes CUDA graphs.

[12] "The Strategic Pivot: From SGLang Debugging to a Custom C/C++/CUDA Inference Stack" — Article on the user's directive to build a custom stack.

[13] "The Pivot Point: From Benchmarking to Custom Inference — Analyzing a Transitional Message in the DDTree Deployment Journey" — Article on the transition between phases.

[14] "The Data Collection Bridge: Gathering Evidence Before Synthesis" — Article on downloading B300 artifacts.

[15] "The Moment of Synthesis: Writing the B300 Findings Report" — Article on writing the findings report.

[16] "The Culmination of Discovery: Synthesizing DDTree Findings for the Next Frontier" — Article on the final synthesis and roadmap.

[17] "The Commit That Closed a Chapter: How a Git Command Captured Weeks of DDTree Discovery" — Article on the git commit.

[18] "The Art of Self-Correction: Debugging a Git Commit in an AI-Assisted ML Workflow" — Article on debugging the git commit path issue.

[19] "The Commit That Almost Wasn't: Preserving Knowledge at a Pivot Point" — Article on the failed git commit and recovery.

[20] "The Final Verification: Closing the Loop on DDTree Deployment" — Article on the final manifest and service health check.

[21] "The Silence Between Movements: An Empty Message as a Transition Point in ML Infrastructure Development" — Article on the user's empty closing message.