The Architecture of Knowledge: How a Comprehensive Status Message Bridges Two Phases of DDTree Inference Optimization

Introduction

In the middle of an intense, multi-week optimization campaign for speculative decoding on NVIDIA Blackwell GPUs, there comes a moment when the work shifts from exploration to construction. Message 11825 in this opencode session is that moment. It is not a message of action—no tool calls are issued, no code is written, no commands are executed. Instead, it is a message of synthesis: a comprehensive project status document that captures the complete state of the DDTree speculative decoding effort, distills the lessons learned from dozens of debugging and benchmarking rounds, and lays out the blueprint for the next major phase: building a custom C/C++/CUDA inference stack from scratch.

This message is remarkable not for what it does, but for what it knows. It is a knowledge artifact—a structured repository of engineering insight that compresses weeks of trial, error, measurement, and discovery into a form that can directly inform the next generation of the system. To understand this message is to understand the entire trajectory of the project: where it has been, where it stands, and where it is going.

The Strategic Context: A Project at a Crossroads

The message opens with a clear statement of purpose: "Optimize K2.6 + DFlash/DDTree speculative decoding (cuda graphs + temperature + sweeps) on PCIe RTX PRO 6000 (SM120) and NVLink B300 (SM103); next: build a C/C++/CUDA inference stack to maximally optimize DDTree for Kimi, targeting the PRO 6000 box first; write a DDTree findings report."

This single sentence captures the dual nature of the message. It is simultaneously a summary of completed work (the optimization of K2.6 + DFlash/DDTree across two hardware platforms) and a declaration of future intent (building a custom C/CUDA stack). The message exists at the inflection point between these two phases, and its primary function is to ensure that the knowledge gained in the first phase is not lost as the project enters the second.

The user had previously instructed the assistant to "write a report of ddtree findings so far, next we will most likely try to implement a C/C++/CUDA inference stack to maximally optimise ddtree inference specific to Kimi and the machine we're running on" ([msg 11813]). This message is the assistant's response to that directive—but rather than producing a separate report file (which it also does, in the form of DDTREE_FINDINGS_REPORT.md), it first produces this comprehensive status message that serves as the working knowledge base from which the report and the subsequent implementation will be built.

The Architecture of the Message: Structured Knowledge Engineering

The message is organized into seven clearly labeled sections: Goal, Constraints & Preferences, Progress, Key Decisions, Next Steps, Critical Context, and Relevant Files. This structure is not arbitrary—it reflects a deliberate approach to knowledge management that is essential for a project of this complexity.

The Goal section establishes the dual mandate: optimization work that is already complete, and the new C/CUDA stack that is about to begin. The Constraints & Preferences section captures the hardware and software environment with surgical precision—host IPs, GPU counts and topologies, required flags (--attention-backend triton, --page-size 1), CUDA toolkit versions, and platform-specific quirks (the B300's need for python3-dev and --skip-server-warmup). This section alone represents dozens of hours of debugging distilled into a few paragraphs.

The Progress section is the heart of the message, organized into Done, In Progress, and Blocked subsections. This triage reflects a mature engineering discipline: acknowledging what has been achieved, what is still being worked on, and what is preventing further progress. The Key Decisions section documents the engineering judgments that were made along the way—decisions that might otherwise be lost or need to be rediscovered. The Next Steps section provides the actionable roadmap. And the Critical Context section provides the deep technical background needed to understand the entire effort.

This structure is itself a form of output knowledge: it establishes a template for how complex engineering projects should document their state. Any engineer reading this message can quickly orient themselves, understand what has been tried, what worked, what didn't, and what the path forward looks like.

The Three Bug Fixes: A Debugging Journey

One of the most valuable contributions of this message is the documentation of three SGLang bugs that were fixed during the optimization campaign. These fixes are listed in the "Done" section under Progress, and they represent the critical path that had to be cleared before DDTree could work correctly:

  1. cuda_graph_runner.py: The DDTree target-verify graph was incorrectly sized to block_size instead of ddtree_budget+1. This meant that when the budget differed from the block size (which is the normal case), the CUDA graph captured the wrong number of tokens, leading to silent corruption.
  2. triton_backend.py: The target backend was setting num_draft_tokens = ddtree_budget+1, but the custom-mask logic was not correctly handling this when budget+1 ≠ block_size. This was "THE reason wide trees 'hurt'"—the corruption was not algorithmic but a simple sizing bug in the Triton kernel integration.
  3. ddtree_utils.py + dflash_info.py + dflash_worker.py: Temperature-based sampling for DDTree was implemented via tree rejection sampling, reusing the tree_speculative_sampling_target_only sgl_kernel with a first-child/next-sibling encoding, node_logws, and a greedy-vs-sampling branch with shared commit logic. These three fixes are not just bug reports—they are a map of the SGLang codebase's DDTree integration. They tell a future implementer exactly where the fragile interfaces are: the CUDA graph sizing, the Triton custom-mask pathway, and the sampling logic. For the upcoming C/CUDA stack, these are the exact areas that need to be reimplemented correctly from the ground up. The message also notes the dramatic impact of the CUDA graph fix: "cuda graphs base 26→98 tok/s C=1 (3.8×)." This single number—a 3.8× speedup from eliminating Python launch overhead—is perhaps the most compelling argument for the custom C/CUDA stack. If simply removing Python overhead from the existing code gives 3.8×, what could a ground-up C/CUDA implementation achieve?

The Performance Landscape: Two Platforms, One Algorithm

The message captures a comprehensive set of benchmark results across both hardware platforms, providing a clear picture of DDTree's performance characteristics:

PRO 6000 (PCIe Blackwell, SM120): DDTree with budget=8, topk=4, and CUDA graphs achieves 150-170 tokens/second at concurrency 1, with an acceptance rate of approximately 4.5 tokens per step and coding accuracy of 5/5. The budget sweep shows that larger budgets (b12, b16) improve acceptance at long context (acceptance rising to 4.91), and block_size=16 pushes acceptance to 5.19. EP4/EP8 configurations peak at approximately 1500 tokens/second aggregate.

B300 (NVLink, SM103): The NVLink platform achieves substantially higher performance. Autoregressive baseline is 133 tokens/second at C=1. DDTree with budget=8, NVLS, and CUDA graphs reaches 303 tokens/second at C=1—a 2.15× speedup over autoregressive. At higher concurrency, the system achieves 4006 tokens/second at C=64 and 4723 at C=128. Acceptance rate is 4.36-4.48, and coding accuracy is 5/5.

The message also documents a critical finding about larger budgets: "budget=16 EAGER: accept 5.3-6.4, coding 5/5—confirms bigger trees lift acceptance." This is important because it validates the DDTree algorithm's correctness—wider trees do improve acceptance—but the CUDA graph implementation cannot handle them on SM103 due to a separate bug.

The power measurements are equally revealing: "Power 360-460W/1100W under load (HBM-bandwidth-bound)." The GPUs are drawing only 33-42% of their thermal design power, yet achieving 100% utilization. This is the classic signature of a memory-bandwidth-bound workload: the compute units are idle, waiting for data from HBM. This observation is the foundational insight for the entire C/CUDA stack effort.

The Bottleneck Analysis: HBM Bandwidth as the Central Constraint

The message's identification of the HBM bandwidth bottleneck is perhaps its single most important analytical contribution. The observation that the GPUs are "100% util but only ~33-42% power" under load reveals that the INT4 weight-only MoE decode is fundamentally memory-bound, not compute-bound. The tensor cores are underutilized because the weights arrive from HBM too slowly to keep them fed.

This has profound implications for the C/CUDA stack design. If the workload is HBM-bound, then:

Blockers and Their Implications

The "Blocked" section of the message identifies three specific barriers to further progress:

  1. sm_103 CUDA graph bug: Any DDTree budget greater than 8 causes CUDA graph execution to crash on the B300's SM103 GPUs. Budgets of 12 or 16 produce cudaErrorIllegalAddress, while budget 32 produces garbage output (acceptance rate of 1.0, coding 0/5). Only budget 8 (with q_len=9) works correctly. This blocks the exploitation of the B300's spare compute capacity with wider trees.
  2. MLA absorb CUBLAS limit (sm_103): When maxreq × (budget+1) > ~1200, the batched GEMM in the MLA absorb kernel fails with CUBLAS_STATUS_EXECUTION_FAILED. This imposes an upper bound on the combination of batch size and tree width.
  3. Acceptance ceiling from drafter depth: The drafter's block_size=8 caps the tree depth at 7. Increasing the budget only adds width, not depth. The real lever for improving acceptance is a deeper-block drafter. These blockers are not just problems—they are design constraints for the C/CUDA stack. The custom implementation must handle the tree-verify attention without relying on the buggy CUDA graph mechanism, must avoid the CUBLAS limit through proper kernel sizing, and must be designed to accommodate a future deeper-block drafter.

Key Decisions and Their Rationale

The message documents several key engineering decisions that were made during the optimization campaign:

TP8 on NVLink, EP on PCIe: "TP8 is correct on NVLink B300 (EP's PCIe advantage gone); EP wins only on PCIe." This decision reflects a deep understanding of the hardware topology. On the NVLink-connected B300, inter-GPU communication is fast enough that tensor parallelism (TP) is optimal. On the PCIe-connected PRO 6000, expert parallelism (EP) wins because it avoids the PCIe bandwidth bottleneck. This is the kind of platform-specific knowledge that is essential for optimal deployment.

Recommended default configuration: DDTREE block_size=8 budget=8 topk_cap=4 draft-window-size=2048 with CUDA graphs (and NVLS on B300), with maxreq=128. This represents the best stable configuration that works across both platforms.

Draft window size: 2048 tokens is identified as the right default because it is "free at ≤4k, bounds long-ctx draft KV." This balances the need for sufficient draft context against the memory cost of storing draft key-value cache.

DDTree requires pure-attention target: "K2.6 qualifies; Qwen3.6 GDN-hybrid suffers state leakage." This is a critical architectural constraint—DDTree's correctness depends on the target model having no stateful components beyond attention. The Kimi K2.6 model's pure-attention architecture makes it an ideal target.

Critical Context: The Model Architecture

The "Critical Context" section provides a detailed technical description of the Kimi K2.6 model and the DDTree speculative decoding flow. This is essential input knowledge for anyone who wants to understand the optimization effort:

Assumptions and Their Validity

The message rests on several assumptions that are worth examining:

  1. The drafter is undertrained: The drafter model (SubSir/Kimi-K2.6-DFlash-tmp-long) is described as "~undertrained, 'tmp-long'." This assumption is supported by the relatively low acceptance rates (4.36-4.48 for budget=8) and the observation that acceptance drops on hard reasoning text. If this assumption is wrong and the drafter is actually well-trained, the acceptance ceiling would be higher than expected.
  2. DDTree is exact for pure-attention models: The message states that DDTree "requires pure-attention target" and that K2.6 qualifies. This is a correctness assumption—if the model has hidden stateful components beyond attention, the speculative decoding could produce incorrect results.
  3. The HBM bandwidth bottleneck is the primary constraint: The power measurements support this, but it's worth noting that this conclusion is based on aggregate GPU behavior. Individual kernels may have different characteristics.
  4. The C/CUDA stack will yield substantial improvements: This is the central bet of the next phase. The 3.8× CUDA graph speedup provides strong evidence, but the actual improvement depends on the quality of the implementation.
  5. The sm_103 CUDA graph bug is a kernel issue, not an algorithm issue: The message notes that eager mode works correctly for larger budgets, confirming that the algorithm is sound and the bug is in the CUDA graph capture mechanism.

The Road to Custom CUDA: How This Message Feeds the Next Phase

The "Next Steps" section explicitly lays out the plan for the C/CUDA inference stack, and the "Critical Context" and "Relevant Files" sections provide the detailed technical knowledge needed to execute that plan. The message identifies four specific optimization targets:

  1. Kill Python/launch overhead: The 3.8× CUDA graph speedup shows what's possible when Python overhead is eliminated. A native C/CUDA implementation can go further by owning the entire decode loop.
  2. Fused INT4 Marlin dequant+MoE GEMM: The current implementation separates dequantization from the GEMM, wasting memory bandwidth. Fusing them would reduce traffic and improve throughput.
  3. Custom tree-attention MLA verify kernel: The current implementation relies on SGLang's Triton attention backend with a custom mask. A dedicated CUDA kernel with the visibility mask baked in would eliminate the q_len>9 instability and the CUBLAS limit.
  4. GPU-side tree build: The current CPU heapq-based tree construction is a scaling concern. Moving it to the GPU would eliminate host-device synchronization and enable larger, more complex trees. The message also provides concrete targets: "150-170 tok/s @ C=1, accept ~4.5" on the PRO 6000, and the correctness anchors of "token-exact greedy, distribution-exact temperature."

Conclusion: The Message as a Knowledge Artifact

Message 11825 is not where the work happens—it is where the work is understood. It is a moment of synthesis in a complex engineering project, where the lessons of dozens of debugging sessions, benchmark runs, and configuration sweeps are distilled into a coherent picture that can guide the next phase of development.

For the C/CUDA inference stack that will follow, this message serves as both a requirements document and a design specification. It answers the essential questions: What are we building? Why are we building it? What constraints must it satisfy? What pitfalls must it avoid? What performance targets must it meet?

The message's structured format—Goal, Constraints, Progress, Decisions, Next Steps, Context, Files—is itself a contribution to the project's engineering discipline. It establishes a template for how complex technical knowledge should be captured and communicated. Any engineer joining the project could read this message and quickly understand the state of the art, the open problems, and the path forward.

In the end, this message is a testament to the value of documentation in engineering. The code that will be written in the next phase—the CUDA kernels, the C++ inference loop, the fused operations—will be the tangible output of the project. But the knowledge captured in this message is what makes that code possible. It is the map that guides the construction, the accumulated wisdom that prevents the next engineer from repeating the same mistakes, and the shared understanding that enables a complex system to be built by distributed effort.

The message is, in short, the architecture of knowledge—and that architecture is as important as the architecture of the software it describes.