The Strategic Pivot: From SGLang Debugging to a Custom C/C++/CUDA Inference Stack

"Download artifacts from the B300 machine, write a report of ddtree findings so far, next we will most likely try to implement a C/C++/CUDA inference stack to maximally optimise ddtree inference specific to Kimi and the machine we're running on. First we target the pro6000 box"

At first glance, message <msg id=11813> appears to be a simple task assignment: the user asks the assistant to download benchmark artifacts, write a report, and then pivot to building a custom inference stack. But this brief instruction is actually a strategic inflection point in a months-long engineering campaign. It represents the moment when the team makes a deliberate decision to abandon the incremental debugging of an existing framework (SGLang) and instead commit to building something entirely new and purpose-built. This message is the hinge between two phases of the project: the exploration and benchmarking phase, and the implementation phase. To understand its full weight, one must trace the chain of events that led to this decision.

The Context: A Monumental Benchmarking Campaign

The message arrives at the conclusion of an extraordinarily thorough investigation into speculative decoding for the Kimi K2.6 model using the DDTree (Draft-Tree) algorithm. The preceding segment ([msg 11573] through [msg 11812]) documents a multi-week effort spanning two hardware platforms, four parallelism strategies, countless SGLang patches, and dozens of benchmark configurations.

On the PCIe-connected PRO6000 box (8× RTX PRO 6000 Blackwell GPUs), the assistant systematically benchmarked TP8 (tensor parallelism), PP8 (pipeline parallelism), EP8 (expert parallelism), and EP4 (expert parallelism with TP2 groups). The results validated the user's intuition that expert parallelism avoids PCIe AllReduce bottlenecks on MoE layers: EP8 dramatically improved single-request throughput from 26 to 65 tok/s, and EP4 reached peak aggregate throughput of ~1531 tok/s at high concurrency. PP8 was disappointing due to pipeline bubbles, while TP8 with CUDA graphs hit 98 tok/s at C=1 but plateaued at ~1291 tok/s.

On the NVLink-connected B300 SXM6 machine (8× B300, 275 GB each, sm_103 architecture), the results were even more dramatic. DDTree with budget=8, NVLS (NVIDIA Link-Local Scratch), and CUDA graphs achieved 303 tok/s at C=1 — a 2.15× speedup over the 133 tok/s autoregressive baseline — and scaled to an astonishing 4723 tok/s at C=128 with 5/5 coding correctness. NVLS provided a free ~5% boost, and NVLink made TP8 the clear winner over EP for this workload.

But the most important finding was also the most frustrating. The user's hypothesis that larger budgets would better utilize the B300's spare compute was confirmed: budget=16 in eager mode lifted acceptance to 5.3–6.4 tokens per step (vs b8's 4.48) and passed all coding tests. The DDTree logic was sound. However, a critical sm_103-specific CUDA graph bug blocked realizing this gain: any budget > 8 caused illegal memory accesses (b12/b16) or garbage output (b32) during the tree-verify graph capture, while eager mode lost the 3.8× graph speedup, resulting in lower net throughput than b8+graphs.

The assistant spent considerable effort diagnosing this bug — tracing through buffer sizing in init_cuda_graph_state, examining the replay path in init_forward_metadata_replay_cuda_graph, testing intermediate budgets like b12 — but each attempt confirmed the same pattern: the bug was a deep interaction between Triton kernels, CUDA graph capture, and the sm_103 architecture at query lengths greater than 9. As the assistant noted in <msg id=11808>, "This looks like a genuine sm_103 cuda-graph kernel bug—possibly in the triton decode attention kernel with custom masking at q_len=17/33 under graph capture, or a cublas issue in the MLA absorb."

Why This Message Was Written: The Strategic Calculus

The user's message is a response to this impasse. The assistant's previous message ([msg 11812]) laid out the situation clearly: the DDTree algorithm works, larger budgets improve acceptance, but CUDA graphs break on sm_103 for any budget > 8. The assistant identified two levers: (1) fix the sm_103 CUDA graph tree-verify kernel, or (2) build a deeper-block drafter. But both of these are incremental improvements within the existing SGLang framework.

The user's message rejects this incremental path. Instead of continuing to fight SGLang's Python/Triton/CUDA-graph stack, the user proposes building something entirely new: a custom C/C++/CUDA inference stack "to maximally optimise ddtree inference specific to Kimi and the machine we're running on." The emphasis on "specific to" is crucial — this is a deliberate choice of specialization over generality. SGLang is a general-purpose inference engine that must support dozens of model architectures, attention variants, and deployment scenarios. By building a custom stack for a single model (Kimi K2.6) on a single hardware configuration (8× PRO6000 or 8× B300), the team can eliminate every layer of abstraction, every unnecessary indirection, and every Python overhead that SGLang imposes.

The decision to "first target the pro6000 box" is also strategic. The PRO6000 machine is PCIe-connected, which means its bottlenecks are different from the B300's NVLink fabric. The custom stack can be designed to address the specific PCIe AllReduce bottleneck that EP8 partially solved, but with much more aggressive optimizations: fusing dequantization with MoE GEMM, eliminating Python overhead from the scheduling loop, building a custom tree-attention MLA verify kernel that bypasses the sm_103 graph instability entirely, and moving tree construction from the CPU to the GPU.

How Decisions Were Made

The decision visible in this message is the culmination of a long chain of empirical evidence. Every step was data-driven:

  1. The parallelism benchmarks on PRO6000 established that EP configurations win on PCIe, while TP8 wins on NVLink. This directly informs the architecture of the custom stack: it must support both parallelism strategies, or be designed so that the parallelism layer is swappable.
  2. The CUDA graph debugging on B300 established that the sm_103 architecture has a fundamental incompatibility with Triton-generated CUDA graphs at larger tree budgets. This is not a bug that can be fixed with a configuration tweak — it requires writing custom CUDA kernels that bypass the Triton compilation pipeline entirely.
  3. The power analysis (100% GPU util but only 360–460 W out of 1100 W) confirmed that the workload is HBM-bandwidth-bound, meaning the compute units are genuinely idle while waiting on memory. This validates the entire premise of speculative decoding for this model: if there's spare compute, the tree verification step can be essentially free as long as it doesn't increase memory traffic.
  4. The acceptance rate data (4.48 tokens per step at b8, 5.3–6.4 at b16) quantifies the potential upside of larger trees. The user's earlier hypothesis that "bigger trees would better utilize the B300's spare compute" was confirmed, but the implementation path was blocked by the framework. Each of these findings contributed to the conclusion that the framework itself is the bottleneck. The decision to build a custom stack is not a gamble — it is a calculated response to specific, measured limitations.

Assumptions Embedded in the Message

The user makes several assumptions, some explicit and some implicit:

Explicit assumptions:

Input Knowledge Required

To fully understand this message, one must possess a substantial body of knowledge accumulated over the preceding weeks:

  1. DDTree algorithm mechanics: Understanding how draft-tree speculative decoding works — the tree structure, the verification pass, the acceptance criterion, and how budget and top-k parameters control the tradeoff between draft quality and verification cost.
  2. SGLang architecture: Knowledge of SGLang's Python-based scheduler, its Triton attention backends, its CUDA graph capture mechanism, and how speculative decoding is implemented as a "draft runner" that interacts with the main model.
  3. Hardware platform characteristics: Understanding the difference between PCIe-connected GPUs (where inter-GPU communication goes over the PCIe bus, creating AllReduce bottlenecks) and NVLink-connected GPUs (where direct GPU-to-GPU links provide much higher bandwidth). Understanding the sm_103 architecture of the B300 and its differences from the sm_120 architecture of the Blackwell PRO6000.
  4. MoE inference bottlenecks: Knowledge of how Mixture-of-Experts models like Kimi K2.6 are inference-bound — the weight-only INT4 quantization, the HBM-bandwidth-limited decode phase, and the AllReduce overhead on MoE layers.
  5. The benchmarking campaign: Familiarity with the specific results — the EP8 vs TP8 tradeoffs, the CUDA graph speedup factors, the acceptance rate improvements with larger budgets, and the power utilization data.
  6. The sm_103 CUDA graph bug: Understanding the specific failure mode — illegal memory access at budget≥16 during CUDA graph capture, correct behavior in eager mode, and the suspicion that it's related to query length exceeding 9 in the tree-verify path. Without this context, the user's message reads as a simple task assignment. With it, it reads as a strategic document.

Output Knowledge Created

This message creates several forms of knowledge and direction:

  1. A strategic roadmap: The project now has a clear sequence: (a) download B300 artifacts, (b) write a comprehensive findings report, (c) build a custom C/C++/CUDA inference stack targeting the PRO6000 first.
  2. A design philosophy: The phrase "maximally optimise ddtree inference specific to Kimi and the machine we're running on" establishes the guiding principle — specialization over generality. Every design decision should be evaluated against this criterion.
  3. A priority order: PRO6000 first, B300 second. This is a pragmatic choice — the PRO6000 box is likely more accessible, has fewer architecture-specific bugs (no sm_103 CUDA graph issue), and represents the primary deployment target.
  4. A preservation mandate: The instruction to "download artifacts from the B300 machine" ensures that the hard-won benchmarking data is not lost when the focus shifts. The B300 results serve as a reference point and a validation target for the custom stack.
  5. A termination signal: The implicit message is that SGLang debugging is done. No more time should be spent fixing the sm_103 CUDA graph bug, tuning configuration parameters, or patching Triton kernels. The framework approach has been evaluated and found wanting.

Mistakes and Potential Pitfalls

While the strategic pivot is well-reasoned, it carries risks:

  1. Engineering cost: Building a custom inference stack from scratch is a massive undertaking. SGLang represents years of engineering effort by a team of specialists. The custom stack must replicate the core functionality — model loading, weight quantization, attention computation, MoE routing, KV cache management, scheduling, and speculative decoding — before it can even begin to optimize. The assumption that a small team can outperform a mature framework is optimistic.
  2. The PRO6000-first decision: The most impressive results came from the B300 machine (303 tok/s, 2.15× speedup). The PRO6000 box achieved more modest results (86 tok/s, 1.3× speedup). By targeting the slower platform first, the team may be optimizing for the wrong constraints. The PCIe bottleneck on PRO6000 is fundamentally different from the NVLink-rich B300 environment, and optimizations that work for one may not transfer to the other.
  3. Abandoning SGLang prematurely: The sm_103 CUDA graph bug, while frustrating, may be fixable with sufficient effort. The assistant's investigation suggested it might be a buffer sizing issue or a Triton kernel interaction — both potentially solvable. By pivoting to a custom stack, the team is accepting that this bug is unfixable within the framework, which may not be true.
  4. The "specific to Kimi" constraint: Building a stack that only works for one model creates a single point of failure. If the model changes (e.g., Kimi K3.0 is released with a different architecture), the custom stack may need a complete rewrite. A more general approach would amortize the engineering investment across multiple models.

The Thinking Process Visible in the Message

The user's thinking is remarkably compressed into this short message. The structure reveals the logical flow:

  1. Preserve first ("Download artifacts from the B300 machine"): Before pivoting, secure the data. The B300 benchmarks represent irreplaceable empirical evidence about DDTree performance on NVLink hardware. If the custom stack eventually targets B300, these results will be the baseline to beat.
  2. Document second ("write a report of ddtree findings so far"): Consolidate knowledge before moving on. The report serves multiple purposes: it forces the team to articulate what they've learned, it provides a reference document for the custom stack design, and it captures the reasoning behind the strategic pivot.
  3. Build third ("next we will most likely try to implement a C/C++/CUDA inference stack"): The "most likely" qualifier is interesting — it suggests the user is open to alternative paths if the report reveals something unexpected. But the direction is clear.
  4. Specialize ("to maximally optimise ddtree inference specific to Kimi and the machine we're running on"): This is the core design principle. The stack will not be general-purpose; it will be a precision instrument for one model on one hardware configuration.
  5. Sequence ("First we target the pro6000 box"): The final piece of the plan establishes priority. The PRO6000 is the first target, implying the B300 is second — or that the lessons learned on PRO6000 will inform the B300 implementation. The message is notable for what it does not say. There is no hand-wringing about the CUDA graph bug, no lament about the limitations of SGLang, no debate about whether the custom stack is the right choice. The user has processed all of that internally and arrived at a clear decision. The message is purely directional — it tells the assistant what to do next, not why.

Conclusion

Message <msg id=11813> is a quiet turning point in a complex engineering project. It marks the transition from exploration to construction, from debugging to creation, from working within constraints to building a custom solution. The user's decision to build a purpose-built C/C++/CUDA inference stack is a vote of confidence in the DDTree algorithm and a recognition that the existing framework has reached its limits. The PRO6000 box, with its PCIe bottlenecks and proven EP8 performance, becomes the first target — a platform where the custom stack can demonstrate its value before tackling the more exotic B300 NVLink environment.

The message also embodies a philosophy that recurs throughout engineering: sometimes the fastest path forward is not to fix what's broken, but to build something new that bypasses the broken parts entirely. The sm_103 CUDA graph bug may never be fixed in SGLang, but that no longer matters — the team is building a stack that doesn't need it.