The Commit That Brought Temperature to DDTree: Bridging Greedy and Sampling in Speculative Decoding

In the world of large language model inference, speculative decoding has emerged as a powerful technique for accelerating generation without sacrificing quality. The core idea is elegant: a lightweight "draft" model proposes candidate tokens, and a larger "target" model verifies them in parallel. But the devil, as always, lives in the implementation details. One particularly devilish detail was that the Diverse Dynamic Tree (DDTree) variant of speculative decoding—a sophisticated approach that builds a best-first tree of draft candidates—only supported greedy verification. Temperature-based sampling, essential for creative and diverse generation, was simply unavailable.

Message [msg 11660] in this coding session marks the precise moment that limitation was lifted. It is the commit message and deployment trigger for a substantial refactoring effort spanning three files and 271 lines of new code. This message, seemingly just a git command and a log line, is actually the culmination of a carefully reasoned engineering journey—one that reveals deep thinking about kernel interfaces, tree representations, and the architecture of modern inference engines.

The Message in Full

Let us examine the exact content of the message:

[assistant] Now deploy to CT200's site-packages and restart with DDTree + temperature. Let me commit first, then deploy: [bash] cd /home/theuser/glm-kimi-sm120-rtx6000bw git add -A ct200_sglang_working/speculative/ddtree_utils.py ct200_sglang_working/speculative/dflash_info.py ct200_sglang_working/speculative/dflash_worker.py git commit -m "Add DDTree temperature support via tree-structured rejection sampling

>

Reuses the existing tree_speculative_sampling_target_only sgl_kernel (same one EAGLE and linear DFlash use). Converts the DDTree best-first tree into EAGLE's first-child/next-sibling encoding (compile_ddtree_retrieve), siblings ordered by cumulative draft logprob. verify() now branches: greedy uses follow_verified_tree, non-greedy computes target_probs (temp+top_k/top_p) and runs the kernel, then shares the same commit/KV/hidden path. Worker builds retrieve buffers and only blocks non-greedy when the kernel is unavailable." 2>&1 | tail -3 [master 4c5b24f] Add DDTree temperature support via tree-structured rejection sampling 3 files changed, 271 insertions(+), 16 deletions(-)

The message is deceptively short. A git add, a git commit, and a three-line summary of the result. But the commit message itself is a compressed technical essay, and the surrounding context reveals the full weight of what was accomplished.

Why This Message Was Written: The Motivation and Context

The assistant had been working for days on deploying the Kimi K2.6 model with DDTree speculative decoding on a cluster of NVIDIA RTX PRO 6000 Blackwell GPUs and B300 SXM6 NVLink machines. DDTree had shown impressive throughput gains—up to 2.15× over autoregressive baselines—but only for greedy decoding. The user's application required temperature-based sampling for diverse generation, and DDTree simply crashed with a RuntimeError when any non-greedy sampling configuration was used.

This limitation was not accidental. The DDTree verify path had been built with a guard (visible in message [msg 11657]) that explicitly raised an error:

if (
    self.is_ddtree
    and not self.ddtree_shadow_linear
    and batch.sampling_info is not None
    and not batch.sampling_info.is_all_greedy
):
    raise RuntimeError(
        "DDTREE currently supports greedy verification only. Use temperature=0 or --speculative-ddtree-shadow-linear"
    )

This guard existed because the greedy path used a simple deterministic algorithm (follow_verified_tree) to walk the best-first tree and select the highest-probability path. Sampling required a fundamentally different approach: computing target probabilities, running rejection sampling against the draft distribution, and handling the stochastic acceptance of tree paths.

The assistant's reasoning, visible across messages [msg 11637] through [msg 11653], reveals a carefully constructed plan. The key insight was that DDTree and EAGLE (another speculative decoding method) actually share a common kernel interface. EAGLE's tree_speculative_sampling_target_only kernel already performed tree-structured rejection sampling—it just expected the tree to be encoded in a specific format: first-child and next-sibling pointers, with sibling nodes ordered by descending cumulative log-probability. DDTree's tree was built with parent pointers and child maps, a different representation entirely.

The bridge between these two worlds was compile_ddtree_retrieve, a new function that converts DDTree's parent-pointer representation into EAGLE's first-child/next-sibling encoding. As the assistant verified in message [msg 11645], the conversion was correct: a depth-first traversal of the encoded tree visited every non-root node exactly once, with siblings ordered by descending log-probability—exactly what the kernel expected.

How Decisions Were Made

The decision to reuse the existing EAGLE kernel rather than writing a new one was the architectural cornerstone of this change. This choice carried several implications:

First, it meant the sampling path could piggyback on a well-tested kernel. The tree_speculative_sampling_target_only kernel was already battle-hardened from EAGLE and linear DFlash deployments. Reusing it avoided the risk of introducing subtle bugs in a new rejection sampling implementation.

Second, it forced a tree representation conversion. DDTree's natural representation (parent pointers with child maps) had to be translated to EAGLE's first-child/next-sibling encoding. This conversion, implemented in compile_ddtree_retrieve, became the critical piece that made the entire approach work.

Third, it meant the commit path could be shared. The assistant realized that both greedy and sampling verification produce the same output contract: a list of accepted node indices and a list of proposed tokens, where the last token is the bonus token. By refactoring the verify method to split the derivation phase (greedy vs. sampling) from the shared commit phase, the assistant eliminated code duplication and ensured both paths would behave consistently.

The decision to order siblings by cumulative draft log-probability (rather than the raw log-probability of each node individually) was another subtle but important choice. The cumulative probability reflects the total likelihood of the path from root to that node, which is what the rejection sampling kernel uses to decide whether to accept or reject a candidate path. Ordering by this metric ensures that the most promising paths are visited first during the tree walk.

Assumptions Made by the Assistant

Several assumptions underpin this implementation, some explicit and some implicit:

The kernel's output format assumption: The assistant assumed that the kernel's accept_index output contains flat indices into the [bs * q_len] arrays, and that these could be converted back to local node indices within each request's tree using modulo arithmetic (accept_index_row % q_len). This assumption was verified by tracing through the kernel's output contract, but it was not empirically tested until the deployment phase.

The bonus token assumption: The assistant assumed that the kernel's predict[flat_idx] array directly gives the emitted tokens including the bonus token, and that the last accepted node's hidden state is the one needed for drafting the next block. This matched the greedy baseline behavior and was consistent with the EAGLE kernel's documented behavior.

The threshold parameters assumption: The _sample_tree_paths method imported get_global_server_args to access speculative_accept_threshold_single and threshold_acc, both defaulting to 1.0 for exact rejection sampling. The assistant assumed these defaults were appropriate for DDTree, which was reasonable but untested at this point.

The kernel availability assumption: The guard was relaxed from a hard block to a conditional block that only raises when the kernel is unavailable. The assistant assumed that the kernel would be available on the target deployment machines (CT200 with CUDA 12.8+), which was a reasonable assumption given that EAGLE was already working on those machines.

Mistakes and Incorrect Assumptions

While the implementation was sound, several assumptions would later prove incomplete:

The most significant issue was the CUDA graph incompatibility. As revealed in later chunks, the sampling path with budgets larger than 8 caused illegal memory accesses during CUDA graph capture on sm_103 (B300) GPUs. The eager-mode fallback worked correctly but lost the 3.8× graph speedup, meaning the temperature path was functional but not as fast as hoped. This wasn't a mistake in the implementation per se—the code was correct—but it reflected an incomplete understanding of the interaction between the tree-structured rejection sampling kernel and CUDA graph capture.

Another subtle issue was the readiness check race condition during service restart, which would surface in the next deployment attempt. The assistant assumed that stopping the old service and starting the new one would be a clean handoff, but in practice the old process answered health checks briefly before being killed, while the new process was still loading weights for approximately six minutes. This wasn't a code bug but an operational assumption that would need adjustment.

Input Knowledge Required

To fully understand this message, one needs knowledge spanning several domains:

Speculative decoding architecture: Understanding that draft models propose candidates and target models verify them, and that tree-based methods like DDTree and EAGLE propose multiple candidate paths simultaneously.

Tree representations: The difference between parent-pointer representations (natural for DDTree's best-first construction) and first-child/next-sibling representations (natural for EAGLE's tree-structured rejection sampling kernel).

CUDA kernel interfaces: The tree_speculative_sampling_target_only kernel's input/output contract, including how it expects tree topology to be encoded and how it returns accepted paths.

SGLang's DFlash worker architecture: How the worker builds verify inputs, runs the forward pass, and commits accepted tokens to the KV cache.

PyTorch tensor operations: The modulo arithmetic for converting flat indices to local indices, the reshaping of probability tensors, and the gather/scatter operations for hidden state selection.

Output Knowledge Created

This message produced several lasting artifacts:

The git commit (4c5b24f): A permanent record of the change, with a commit message that serves as a design document for future developers.

Three modified source files: ddtree_utils.py gained the compile_ddtree_retrieve function and node_logws tracking in the builder. dflash_info.py gained the _sample_tree_paths method, retrieve tensor fields in DDTreeVerifyInput, and the refactored verify method with greedy/sampling branching. dflash_worker.py gained the retrieve buffer construction in _build_ddtree_verify_input and the relaxed guard.

271 lines of new code, 16 lines deleted: A concrete measure of the change's scope.

A deployment plan: The assistant immediately followed this commit with a deployment to CT200 (message [msg 11661]), copying the files to the site-packages directory and restarting the service with budget=16, topk=4.

The Thinking Process Visible in the Reasoning

The assistant's reasoning across the preceding messages reveals a methodical engineering mind at work. The chain of thought in message [msg 11637] is particularly revealing:

"I'm seeing how the greedy verification produces an accepted path of node indices and a proposed list of tokens to emit, then commits those tokens while tracking which ones were appended. For the sampling branch, I need to generate the same structure—the node index path and corresponding token sequence—for each request."

This is the core insight: both paths produce the same contract, so they can share the commit phase. The assistant then traces through the kernel's output format:

"The kernel outputs give me flat indices into the flattened [bs*q_len] arrays: accept_index contains the flat positions of accepted nodes, predict holds the token at each position, and accept_token_num tracks the count. I can convert these flat indices back to local node indices within each request's tree using modulo arithmetic."

The modulo arithmetic insight (accept_index_row % q_len) is the key that unlocks the kernel's output for DDTree's tree structure. The assistant is thinking in terms of tensor layouts and index arithmetic, a skill essential for high-performance GPU programming.

The reasoning also shows careful attention to edge cases:

"The downstream commit loop expects proposed and accepted to have the same length, where the last element is the bonus token and accepted_drafts counts everything except that final one. The kernel's num_accepted includes the root node (position 0), so accepted_paths[i] starts with 0, which matches the greedy baseline."

This attention to the exact semantics of the commit loop—what counts as a draft versus a bonus token, how the root node is handled—demonstrates the kind of meticulous thinking required when working at the intersection of GPU kernels and inference engine internals.

Conclusion

Message [msg 11660] is a milestone in the DDTree deployment journey. It represents the moment when a carefully planned architectural change was committed to version control and set in motion toward production deployment. The commit message, dense with technical detail, encapsulates days of reasoning about kernel interfaces, tree representations, and inference engine architecture.

The change itself is a masterclass in software reuse: rather than writing a new rejection sampling kernel for DDTree, the assistant recognized that the existing EAGLE kernel could be adapted through a tree representation conversion. This kind of cross-system thinking—seeing the common interface beneath different representations—is a hallmark of experienced systems engineering.

The deployment that followed this commit would reveal new challenges (CUDA graph incompatibility, service restart race conditions), but the core implementation was sound. Temperature-based DDTree was now possible, and the path was clear for the next phase of optimization: building a custom C/C++/CUDA inference stack that could fully realize the potential of tree-structured speculative decoding on Blackwell and B300 hardware.