The Documentation That Capped a Debugging Odyssey: Capturing DDTree Sweep Results After the Mask Corruption Fix

Introduction

In the life of a machine learning engineer, few moments are as satisfying as the one where a long-standing bug is finally fixed and the system produces correct results for the first time. The euphoria, however, is fleeting—what follows is the equally important but less glamorous work of systematically characterizing the now-working system and preserving the findings for posterity. Message [msg 11687] in this opencode session captures exactly that transition: the moment when the assistant, having just resolved a critical mask corruption bug that had rendered DDTree (Draft-Tree speculative decoding) non-functional, pivots from debugging to documentation, committing the first-ever reliable performance results to the project's permanent record.

This message is deceptively simple on the surface—a bash command that copies a service file, writes a markdown document, and makes a git commit. But beneath that mundane exterior lies the culmination of an intense multi-hour debugging session that spanned CUDA ABI mismatches, Triton JIT compilation failures, pipeline parallelism fixes, and ultimately the discovery that the entire "trees hurt this drafter" conclusion was wrong because the trees were silently corrupted by a misaligned mask. The message [msg 11687] is the moment where all that work crystallizes into actionable knowledge.

The Message Itself

Let us examine the message in full. The assistant executes a bash command from the working directory /home/theuser/glm-kimi-sm120-rtx6000bw:

# Save the final ddtree service file and sweep results
scp -q root@10.1.2.200:/etc/systemd/system/sglang-k26-ddtree.service ct200_sglang_working/services/sglang-k26-ddtree.service 2>&1
cat > ct200_sglang_working/ddtree_sweep_results.md <<'EOF'
# DDTree budget/topk sweep (K2.6, TP8 + cuda graphs, C=1, after mask fix)

Single-stream greedy, avg over 3 coding/explanatory prompts (max_tokens=512).

| budget | topk_cap | C=1 tok/s |
|-------:|---------:|----------:|
| 8  | 4 | 150.2 |
| 16 | 4 | 139.0 |
| 16 | 8 | 133.4 |
| 24 | 6 | 127.0 |
| 32 | 8 | 113.6 |
| 48 | 8 | 101.4 |

Smaller budgets win at C=1: verify cost scales with budget+1 tokens, and the
drafter's depth is capped at block_size-1=7 so very wide trees add cost without
proportional acceptance. budget=8 topk=4 is the C=1 champion (1.53x over the
98 tok/s cuda-graph base).

## Concurrency sweep, budget=8 topk=4 (mixed 2048-token prompts)
| C | agg tok/s | accept_len (steady) |
|--:|----------:|--------------------:|
| 1   | 113.8 | ~3.0 |
| 8   | 326.5 | ~4.0 |
| 32  | 610.6 | ~4.4 |
| 64  | 739.4 | ~4.2 |
| 128 | 785.9 | ~4.0 |

DDTree wins at low concurrency (C=1: 113.8 vs 98 base). Autoregressive EP4 wins
at high concurrency (C=128: 1449 vs 786) because verify overhead competes with
batch saturation once compute-bound.
EOF
git add -A ct200_sglang_working/
git commit -m "Document DDTree results: mask fix makes trees work (commit 4-5 vs 2), budget sweep, concurrency" 2>&1 | tail -2
echo "=== git log ==="
git log --oneline -7

The output confirms success: three files changed (82 insertions, 7 deletions), a new file ddtree_sweep_results.md created, and the git log shows the chain of commits that led to this moment.

The Context: A Bug That Inverted Reality

To understand why this message matters, one must understand the debugging journey that preceded it. The assistant had been deploying Kimi K2.6 with DFlash speculative decoding on an 8× RTX PRO 6000 Blackwell system (PCIe-only) and later on an 8× B300 SXM6 NVLink machine. The DDTree algorithm—a tree-structured variant of speculative decoding that evaluates multiple candidate continuations in parallel—had been producing consistently worse results than the simpler linear DFlash chain. The throughput was lower, the acceptance rates were abysmal, and the outputs were often garbled. The natural conclusion was that "trees hurt this drafter."

But this conclusion was catastrophically wrong. The root cause, discovered and fixed in [msg 11678], was a subtle bug in the Triton attention backend: the num_draft_tokens variable was hardcoded to block_size (the DFlash block size, typically 8), but DDTree's target verification runs over budget+1 tree nodes. Whenever budget+1 didn't equal block_size, the custom-mask offsets for qo_indptr and mask_indptr would misalign, allowing real tree nodes to attend to padded (garbage) KV slots. The result was non-exact, corrupted output that looked like the trees were harming quality—when in fact they were being silently sabotaged by the mask alignment.

The fix was a one-line conceptual change: the target model's backend should use ddtree_budget+1 for its size, while the draft worker's backend keeps block_size. After this fix, the assistant observed a transformative improvement: commit length jumped from ~2 tokens per step (the chain baseline) to 5.0–6.0 tokens per step, and throughput at budget=16 topk=4 reached 150.7 tok/s for code generation—a 1.54× speedup over the 98 tok/s CUDA-graph baseline.

Why This Message Was Written

Message [msg 11687] exists for three reasons, each reflecting a different facet of the engineering mindset.

First, to preserve the winning configuration. The budget/topk sweep had identified budget=8 topk=4 as the champion at single-stream concurrency (150.2 tok/s), and the concurrency sweep had shown how this configuration scales under load. These numbers represent the first reliable performance characterization of DDTree on this model and hardware combination. Without documenting them, the assistant would have no baseline for future optimization work, no way to communicate findings to collaborators, and no record of what worked.

Second, to cement the git history as a narrative. The commit message—"Document DDTree results: mask fix makes trees work (commit 4-5 vs 2), budget sweep, concurrency"—is carefully crafted to tell a story. The parenthetical "(commit 4-5 vs 2)" is the punchline: before the fix, trees committed ~2 tokens (same as chain); after the fix, they commit 4–5. This single number encapsulates the entire debugging arc. The git log that follows shows the progression: the mask fix, the temperature support, the CUDA graph fix, and now the documentation. Each commit is a chapter.

Third, to snapshot the service configuration. Copying the systemd service file from the remote machine ensures that the exact configuration used to produce these results is preserved. This is crucial for reproducibility—if someone needs to recreate the setup months later, they have the precise command-line flags, environment variables, and model paths.

Decisions Embedded in the Message

Although this message is primarily about documentation, it encodes several important decisions:

The champion configuration is budget=8 topk=4. This is not an arbitrary choice but the result of a systematic sweep across six configurations (budget values 8, 16, 24, 32, 48 with various topk caps). The assistant's reasoning, visible in [msg 11684], explains why smaller budgets win at C=1: the verification cost scales with budget+1 tokens, and the drafter's depth is capped at block_size-1=7, so very wide trees add cost without proportional acceptance benefit. This is a genuine insight about the interaction between the tree structure and the drafter's architecture.

The documentation format is a standalone markdown file. Rather than burying the results in a notebook, a comment, or a README, the assistant creates a dedicated ddtree_sweep_results.md file. This signals that the results are a permanent reference artifact, not a transient note. The file includes both raw data tables and interpretive commentary (e.g., "Smaller budgets win at C=1: verify cost scales with budget+1 tokens").

The narrative framing emphasizes the mask fix. The commit message explicitly ties the results to the fix: "mask fix makes trees work." This is a deliberate choice to ensure that anyone reading the git log understands that the positive results are contingent on that fix. Without this framing, a future reader might assume trees always worked at these levels.

Assumptions and Potential Issues

The message makes several assumptions that deserve scrutiny:

The commit_len=1.00 readings are dismissed as artifacts. In the budget sweep output, several configurations showed avg_commit_len=1.00, which would indicate no speculative acceptance at all. The assistant correctly identifies these as "trailing log lines" from the warmup tail—the measurement script captured the acceptance metric before steady-state was reached. This is a reasonable interpretation, but it means the reported acceptance rates for those configurations are unreliable. The assistant compensates by running a separate concurrency sweep with proper steady-state measurement.

The throughput numbers are averaged over only three prompts. The budget sweep uses three coding/explanatory prompts, which may not be representative of the full distribution of workloads. The assistant acknowledges this implicitly by running a separate concurrency sweep with five different prompts (quicksort, TCP explanation, BST class, photosynthesis, Flask API), but even that is a small sample.

The comparison to autoregressive baselines uses different parallelism strategies. The concurrency sweep notes that "Autoregressive EP4 wins at high concurrency (C=128: 1449 vs 786)." But the DDTree configuration uses TP8 (tensor parallelism across 8 GPUs), while the EP4 comparison uses expert parallelism. These are fundamentally different parallelism strategies with different scaling behaviors. The comparison is informative but not apples-to-apples.

Input Knowledge Required

To fully understand this message, a reader needs familiarity with several concepts:

Speculative decoding is a technique where a smaller, faster "draft" model generates candidate tokens that a larger "target" model verifies in parallel. The key metric is acceptance length—how many draft tokens the target accepts per step.

DDTree (Draft-Tree) extends this by having the draft model generate a tree of candidate continuations rather than a single chain, increasing the chance that the target model will accept some path through the tree. The budget controls the total number of tree nodes, and topk_cap controls how many children each node can have.

CUDA graphs are a mechanism for capturing a sequence of GPU kernel launches into a reusable graph, eliminating kernel launch overhead. The assistant notes that CUDA graphs provide a ~3.8× speedup on this hardware.

TP8 vs EP4 refers to tensor parallelism (splitting individual layers across GPUs) vs expert parallelism (assigning different MoE experts to different GPUs). On PCIe-connected GPUs, EP avoids the AllReduce bottleneck on MoE layers.

Block_size is the DFlash block size (8 for this drafter), which constrains the maximum tree depth to block_size - 1 = 7.

Output Knowledge Created

This message creates several lasting artifacts:

ddtree_sweep_results.md is the primary output—a self-contained reference document with two data tables and interpretive commentary. It answers the question "what's the best DDTree configuration for this model on this hardware?" with concrete numbers.

The git commit (2d29d39) permanently links the results to the codebase state that produced them. Combined with the preceding commits (the mask fix, temperature support, CUDA graph fix), it forms a complete narrative of the DDTree implementation and optimization.

The service file snapshot preserves the exact deployment configuration, enabling exact reproduction.

The git log excerpt provides a high-level summary of the entire DDTree development arc in seven commits, from the initial snapshot through the mask fix to the documentation.

The Thinking Process

The assistant's reasoning, visible in the preceding messages, reveals a methodical approach to the documentation task. In [msg 11685], the assistant explicitly lays out the deliverables: "cuda graphs for both DDTree and DFlash, the critical mask corruption fix that made trees actually work correctly, temperature support via tree-structured rejection sampling, and confirming linear DFlash already handles temperature properly." The documentation in [msg 11687] is the final step—capturing the results of the budget sweep and concurrency sweep that validated these deliverables.

The assistant also shows awareness of the narrative structure. The commit message is not just a log entry; it's a summary of the entire debugging arc. The parenthetical "(commit 4-5 vs 2)" is the headline result that justifies all the work. The git log shows the progression in reverse chronological order, telling the story of how DDTree went from "broken" to "working" to "documented."

Conclusion

Message [msg 11687] is the final chapter of a debugging story that began with garbled outputs and a mistaken conclusion that "trees hurt this drafter." It is the moment when the engineer steps back from the code, looks at the now-working system, and says "let me write this down before I forget." The markdown file, the git commit, the service file snapshot—these are the artifacts that transform ephemeral experimental results into permanent knowledge.

In the broader context of the opencode session, this message marks the transition from the "debug and characterize" phase to whatever comes next. The DDTree algorithm has been validated, the optimal configuration identified, and the results preserved. The next phase—whether it's further optimization, deployment to production, or a pivot to a different approach—will build on this foundation. And when the next engineer (or the same engineer, six months later) asks "what was the best DDTree config for K2.6 on Blackwell?", the answer is waiting in ddtree_sweep_results.md, preserved by a single thoughtful commit.