The Final Verdict: When CUDA Graphs Break on sm_103

Introduction

In any complex engineering investigation, there comes a moment when the team must decide whether to keep digging or declare a definitive finding and move on. Message 11809 of this opencode session captures precisely such a moment. After an exhaustive multi-step debugging journey spanning CUDA toolkit configurations, parallelism strategies, speculative decoding algorithms, and architecture-specific kernel behavior, the assistant delivers the conclusive verdict: on NVIDIA's B300 GPUs (sm_103 architecture), CUDA graph capture breaks for any DDTree speculative decoding budget greater than 8. The DDTree logic itself is sound—verified by successful eager-mode execution—but the CUDA graph replay mechanism produces illegal memory accesses and garbage output at larger tree sizes. This message is the capstone of a disciplined investigation, documenting the finding, recovering the system to its best stable configuration, and preserving all benchmark data for future reference.

The Context: A Multi-Phase Optimization Journey

To understand why message 11809 was written, one must appreciate the arc of the broader session. The assistant had been deploying Kimi K2.6 with DFlash speculative decoding across two very different hardware platforms: a PCIe-connected 8× RTX PRO 6000 Blackwell setup and a cutting-edge 8× B300 SXM6 machine with NVLink interconnects. The B300 machine represented the high end—275 GB per GPU, sm_103 architecture, NVLink fabric—and the assistant had already achieved impressive results: DDTree with budget=8, NVLS (non-uniform layer scheduling), and CUDA graphs delivered 303 tok/s at concurrency 1, a 2.15× speedup over the autoregressive baseline.

The user had posed a compelling hypothesis: if the B300 has abundant spare compute (confirmed by power draw measurements showing only 360–460 W out of 1100 W capacity), then larger DDTree budgets should lift acceptance rates and improve throughput. The assistant had been systematically testing this hypothesis across messages 11804 through 11808, and message 11809 represents the final synthesis of those experiments.

The Debugging Arc: From Hypothesis to Conclusive Finding

The assistant's reasoning in message 11809 is the product of a carefully structured elimination process. Let me trace the logic chain that leads to this message.

Step 1: The initial discovery (msg 11804). Testing budget=32 on B300 revealed catastrophic failure: acceptance length stayed at exactly 1.00 (meaning no draft tokens were ever accepted), and the output was syntactically garbled, failing all coding evaluations. This was not a performance regression—it was a correctness bug.

Step 2: Isolating the variable (msg 11805). The critical experiment was to disable CUDA graphs and test budget=16 in eager mode. This was the breakthrough: with graphs disabled, budget=16 achieved 5/5 coding correctness and acceptance jumped to 5.3–6.4 tokens per step, compared to budget=8's 4.48. This confirmed two things simultaneously: the user's hypothesis about larger budgets improving acceptance was correct, and the DDTree algorithm itself was sound. The bug was specifically in the CUDA graph capture path.

Step 3: Ruling out buffer sizing (msg 11806–11808). The assistant investigated whether the cuda_graph_custom_mask buffer was undersized for larger budgets. By examining the SGLang source code (triton_backend.py), the assistant verified that the buffer allocation logic scales correctly with max_num_tokens (which includes budget+1 as the query length). The buffer dimensions were adequate. This ruled out the simplest explanation and pointed to a deeper kernel-level issue.

Step 4: Testing an intermediate point (msg 11808–11809). Before concluding, the assistant tested budget=12 with CUDA graphs enabled—a middle ground between the working b8 and the crashing b16. This also crashed with an illegal memory access error. The pattern was now clear: any budget > 8 triggers illegal memory access in the DDTree verify path under CUDA graph capture on sm_103.

The Conclusive Finding

Message 11809 opens with the assistant's synthesized understanding:

"I'm seeing a pattern here—any budget above 8 with CUDA graphs crashes on sm_103, specifically when the DDTree verify path has a sequence length greater than 9."

This is a remarkably precise diagnosis. The DDTree verify path processes a batch of candidate tokens generated by the tree-structured draft. When the budget (which determines the number of draft tokens per step) exceeds 8, the query length in the attention kernel exceeds 9 (budget + 1 for the target token). At this threshold, the CUDA graph replay produces illegal memory accesses on sm_103. The assistant notes that b12 and b16 crash with explicit cudaErrorIllegalAddress errors, while b32 produces garbage output—a different failure mode at a different size, but both stemming from the same underlying graph capture instability.

Crucially, the assistant distinguishes between two separate phenomena: the DDTree algorithm's correctness (verified by eager mode) and the CUDA graph mechanism's stability (broken at large budgets on sm_103). This distinction is essential because it means the optimization path forward is not about fixing the speculative decoding logic—it's about fixing the graph capture infrastructure for this specific architecture, or finding alternative ways to realize the throughput gains that larger budgets promise.

Decisions Made and Their Rationale

Message 11809 contains several concrete decisions, each with clear reasoning:

Decision 1: Stop testing larger budgets. After b12 crashed with CUDA graphs, the assistant had tested three points above the b8 threshold (b12, b16, b32) and all failed. The pattern was consistent and the conclusion definitive. Continuing to test b14 or b20 would not yield new information—the failure threshold was clearly at budget > 8.

Decision 2: Revert to the best stable configuration. The assistant restores budget=8, topk=4, NVLS enabled, CUDA graphs enabled, max-running-requests=128, and sliding window=2048. This is the configuration that delivered 303 tok/s at C=1 and 4723 tok/s at C=128—the best measured performance on B300. The reversion is not an admission of defeat; it is a pragmatic choice to leave the system in a known-good state.

Decision 3: Preserve all benchmark data. The assistant copies all B300 result JSONs (b8t4_w2048, auto, nvls_b8t4, b32t8, b16t8) into the repro package directory. This is a forward-looking decision: the data will be essential for the next phase, whether that involves writing a findings report, comparing against future fixes, or building a custom inference stack.

Decision 4: Document the finding for the user. The assistant structures the message to communicate both the negative result (CUDA graphs break at budget > 8) and the positive result (DDTree logic is correct, larger budgets improve acceptance). This balanced presentation ensures the user has a complete picture for deciding the next steps.

Assumptions and Their Validation

Several assumptions underpin the assistant's reasoning in this message, and it is worth examining each:

Assumption: The bug is sm_103-specific. The assistant explicitly attributes the failure to "an sm_103-specific cuda-graph kernel bug." This is supported by the observation that budget=32 worked correctly on SM120 (Blackwell) after the mask fix, while it fails on sm_103 (B300). The different failure modes across architectures strongly suggest an architecture-specific kernel compilation or code generation issue in Triton.

Assumption: DDTree logic is correct. This is validated by the eager-mode test at budget=16, which achieved 5/5 coding correctness and acceptance of 5.3–6.4 tokens per step. The algorithm produces correct outputs; only the graph replay path is broken.

Assumption: The production config (b8+NVLS+graphs) is optimal. This is supported by the full benchmark matrix: b8 with graphs outperforms b16 without graphs (285 vs 111 tok/s at C=1), and at high concurrency the gap narrows but b8+graphs still leads. The assistant does not claim this is the theoretical optimum—only that it is the best stable configuration given the current constraints.

Input Knowledge Required

To fully understand message 11809, one needs knowledge spanning several domains:

CUDA graph capture mechanics. CUDA graphs allow launching a pre-captured sequence of GPU operations without CPU involvement between kernels, eliminating Python overhead. However, they are sensitive to memory access patterns and shape changes. The assistant understands that the DDTree verify path involves custom attention masking, and that capturing this into a graph at larger query lengths triggers illegal memory accesses.

Triton attention kernels and custom masking. SGLang's triton attention backend implements custom masking for speculative decoding verification. The cuda_graph_custom_mask buffer is part of this infrastructure. The assistant examined the source code to verify buffer sizing was correct.

sm_103 architecture specifics. The B300 GPU uses compute capability 8.9 (sm_89) or similar, and the assistant has been dealing with architecture-specific issues throughout the session (FlashInfer SM120 compatibility, Triton JIT compilation, etc.). The sm_103 designation appears to refer to the B300's specific architecture variant.

DDTree speculative decoding. The assistant understands that DDTree constructs a tree of candidate draft tokens, and the verify step evaluates multiple candidates in parallel using a custom attention mask. The budget parameter controls the number of draft tokens per step, and larger budgets increase the query length in the verify kernel.

System administration. The assistant uses systemd service management (systemctl daemon-reload, systemctl restart), SSH with timeout handling, and scp for data transfer—all while handling service readiness checks and failure detection.

Output Knowledge Created

Message 11809 produces several forms of output knowledge:

Definitive experimental finding: CUDA graph capture on sm_103 breaks at DDTree budgets > 8, with illegal memory accesses at b12/b16 and garbage output at b32.

Verified positive finding: DDTree with larger budgets (b16) improves acceptance from 4.48 to 5.3–6.4 tokens per step when run in eager mode, confirming the user's hypothesis.

Stable production configuration: budget=8, topk=4, NVLS enabled, CUDA graphs enabled, maxreq=128, window=2048—delivering 303 tok/s at C=1 on B300.

Preserved benchmark corpus: Five JSON result files capturing performance across budget configurations, ready for analysis, reporting, or comparison against future improvements.

Prioritized roadmap implication: The finding directly feeds into the next phase of work. The assistant's earlier investigation of a custom C/C++/CUDA inference stack (documented in the DDTree findings report) is now more urgent: a custom tree-attention MLA verify kernel could bypass the sm_103 graph instability entirely, while also eliminating Python overhead and fusing dequantization with MoE GEMM.

The Recovery and Its Significance

The second half of message 11809 is the recovery sequence. After three SSH commands—one to reconfigure the service, one to wait for readiness, and one to pull benchmark data—the system is restored to its production state. The assistant verifies the service is healthy ("PRODUCTION b8+NVLS+graphs READY @150s") and confirms the data files are in place.

This recovery is not merely housekeeping. It represents a deliberate choice to end the investigation cleanly. The assistant could have continued debugging the CUDA graph capture bug—examining Triton kernel code generation, testing different attention backends, or patching the graph replay path. But the message signals a judgment call: the bug is deep enough that fixing it would require significant engineering effort, and the current stable configuration already delivers excellent performance. The pragmatic path is to document the finding, preserve the data, and move forward.

Broader Implications

The findings in message 11809 have implications beyond this specific deployment. They highlight a fundamental tension in speculative decoding optimization: larger draft trees improve acceptance rates, but they stress the verification infrastructure in ways that may expose architecture-specific bugs. The CUDA graph mechanism, designed to eliminate CPU overhead, becomes a bottleneck when the graph capture cannot handle the dynamic shapes introduced by larger budgets.

For the B300 architecture specifically, the message reveals that the Triton-compiled attention kernels interact poorly with CUDA graph capture at certain shape thresholds. This is not a DDTree bug—it is an infrastructure bug that would likely affect any speculative decoding algorithm requiring custom attention masking at query lengths above 9. The fix would require either patching the Triton kernel code generation for sm_103, modifying the CUDA graph capture to handle dynamic shapes more gracefully, or building a custom verify kernel that avoids the problematic interaction entirely.

Conclusion

Message 11809 is a masterclass in disciplined engineering investigation. The assistant formulates a hypothesis (larger budgets improve acceptance), designs experiments to test it (b32, b16 with and without graphs, b12), isolates the failure mode (CUDA graph capture at q_len > 9), rules out alternative explanations (buffer sizing), and arrives at a definitive conclusion. The message then executes a clean recovery, preserves experimental data, and communicates the findings in a way that directly informs the next phase of work.

The message also demonstrates intellectual honesty: the assistant does not overclaim. The finding is clearly bounded—DDTree logic is correct, larger budgets do improve acceptance, but CUDA graphs break on sm_103 at budget > 8. The user's hypothesis is validated in principle, even if the current infrastructure cannot realize its full potential. This balanced, evidence-based conclusion is exactly what a complex engineering investigation should produce.