The GPU That Wasn't There: Diagnosing a 3% Tensor Core Utilization Mystery in Speculative Decoding

Introduction

In the long arc of optimizing speculative decoding for large language models, few moments are as clarifying as the one where a performance bottleneck finally reveals its true face. Message [msg 12189] in this opencode session is exactly such a moment. It is a diagnostic pivot point — a message where the assistant, armed with GPU profiling screenshots from the user and weeks of cumulative investigation, definitively identifies the root cause of a catastrophic decode slowdown that had been plaguing the DDTree speculative decoding pipeline on RTX PRO 6000 Blackwell GPUs.

The message is deceptively short. It contains a reasoning block, a summary of findings, and a single bash command that reads a few lines of source code. But within that compact structure lies the culmination of a multi-step diagnostic journey: the assistant synthesizes two independent data sources — its own power/bandwidth probes and the user's cufall tensor-core utilization screenshots — into a unified theory of the bottleneck, then reaches into the running service's source code to confirm the structural constraint that makes the bottleneck inevitable. This article unpacks that reasoning in detail, examining the assumptions, the evidence, the mistakes avoided, and the knowledge created.

The Context: A Slowdown That Defied Easy Explanation

To understand the significance of this message, one must appreciate what preceded it. The session had deployed the Kimi K2.6 model with DFlash speculative decoding using DDTree (Draft Tree) on an 8-GPU RTX PRO 6000 Blackwell machine. The initial benchmarks revealed a perplexing pattern: at short context lengths, decode throughput was reasonable, but as context grew toward 128k and 185k tokens, per-step decode time ballooned from ~8 milliseconds to over 1.4 seconds — a 175× increase that could not be explained by the simple growth in KV cache size alone.

The assistant had spent several messages (see [msg 12182] through [msg 12187]) systematically ruling out hypotheses. Was the drafter failing at long context? No — with predictable text, commit length stayed at 7–8 tokens regardless of context length, proving the sliding-window attention and KV caching worked correctly. Was the GPU compute-bound? Nvidia-smi reported 99.8% SM utilization during decode, which would normally suggest the GPU was fully saturated. But deeper probing told a different story: power draw was only 133W (22% of TDP), memory controller utilization was 1.2%, and effective KV cache read bandwidth was a mere 14 GB/s — 130× below the 1.8 TB/s peak of the RTX PRO 6000's HBM3 memory.

This contradiction — high SM utilization but near-idle memory controllers and power draw — is a classic sign of a latency-bound workload, where the GPU's streaming multiprocessors are stalled waiting on data rather than doing useful computation. The assistant had already identified the likely culprit: DDTree's requirement of page_size=1 in the KV cache, which forces the Triton MLA attention kernel to gather KV data one token at a time via pointer-chasing rather than coalesced memory access. But this was still a hypothesis, not a confirmed diagnosis.

The User's Evidence: A Window into the Tensor Cores

The user's contribution in [msg 12188] was decisive. They ran cufall — a low-level GPU profiling tool — with the metric sm__pipe_tensor_cycles_active.avg divided by gr__cycles_elapsed.max, which measures the fraction of cycles where the tensor cores (the specialized matrix-multiply units) are actually doing useful work. They shared two screenshots: one from the prefill phase and one from the decode phase.

The prefill screenshot showed tensor core utilization ranging from 50% to 90% across all eight GPUs, with significant "red" (hot) coloring on the expert units — a healthy compute-bound workload where the GPU is genuinely busy. The decode screenshot told an entirely different story: tensor core utilization at approximately 3% across every SM on every GPU. The GPU was essentially idle.

This was the smoking gun that nvidia-smi could never provide. The utilization.gpu metric reported by nvidia-smi simply tracks whether any kernel is resident on the SMs — it does not distinguish between a kernel that is doing useful arithmetic and a kernel that is stalled on memory latency. The cufall metric, by contrast, measures actual tensor pipe activity — the real compute work. A 3% tensor core utilization during decode means the GPU is spending 97% of its cycles waiting.

The Assistant's Reasoning: Synthesizing Two Independent Data Sources

The assistant's reasoning in [msg 12189] is a masterclass in diagnostic synthesis. It takes the user's cufall screenshots and immediately recognizes them as independent confirmation of its own probe data. The 133W power draw, the 1.2% memory controller utilization, the ~14 GB/s effective bandwidth — all of these pointed to a GPU that was not doing useful work, but nvidia-smi's misleading 99.8% SM utilization had left room for doubt. The cufall metric removed that doubt entirely.

The assistant writes:

The cufall metric (sm__pipe_tensor_cycles_active / gr__cycles_elapsed = real tensor-core compute utilization, far more honest than nvidia-smi's "SM%") nails it: - Prefill (screenshot 1): tensor cores 50–90% active across all 8 GPUs, experts red/saturated → compute-bound, healthy. - Decode (screenshot 2): tensor cores ~3% across every SM on every GPU → the GPU is essentially idle during decode.

The key insight here is the contrast between prefill and decode. Prefill is healthy — the tensor cores are busy 50-90% of the time, the experts are saturated (red), and the GPU is operating at its full 302W TDP. Decode is pathological — 3% tensor core utilization, the GPU barely breaking a sweat at 133W. This is not a system that is fundamentally underpowered for the task; it is a system where the decode path has a structural inefficiency that prevents the GPU from doing useful work.

The assistant then reasons about the nature of the bottleneck:

Given that memory bandwidth is also minimal at 1.2%, the bottleneck during decode must be memory latency from scattered KV cache reads, kernel launch overhead from the 61-layer architecture with multiple kernels per layer at batch size 1, or CPU-side tree construction overhead — not compute or bandwidth saturation.

This is a careful logical deduction. If neither compute (3% tensor utilization) nor bandwidth (1.2% memctrl) is saturated, the bottleneck must be in the latency domain — the GPU is spending its cycles waiting for individual memory requests to complete, or for kernels to be launched, or for CPU-side orchestration. The assistant correctly identifies the most likely candidate: scattered KV cache reads caused by page_size=1.

The Structural Confirmation: Reading the Source Code

The most consequential action in this message is the bash command that reads the relevant lines from dflash_worker.py:

timeout 25 ssh -o StrictHostKeyChecking=no root@10.1.230.171 \
  'sed -n "105,170p;755,775p" /root/venv_sglang211/lib/python3.12/site-packages/sglang/srt/speculative/dflash_worker.py'

The output reveals the smoking gun:

if self.is_ddtree and self.page_size != 1:
    raise RuntimeError(
        "DDTREE currently requires --page-size 1 because accepted tree nodes "
        "are non-contiguous in the verify block."
    )

This is a hard constraint baked into the code: DDTree requires page_size=1. There is no configuration option, no fallback path, no alternative backend. The verify attention — the step where the target model evaluates the draft tree's proposals — must use Triton MLA with page_size=1, because the accepted tree nodes are non-contiguous in the KV cache and the tree mask requires fine-grained per-token addressing.

This confirmation is critical because it transforms the diagnosis from a hypothesis ("maybe page_size=1 is causing scattered access") into a structural analysis ("the code forces page_size=1, which forces scattered access, which causes the 3% tensor utilization, which causes the 175× decode slowdown"). The bottleneck is not a bug that can be fixed with a configuration change — it is an architectural limitation of the current SGLang DDTree implementation.

The Native Engine Thesis

With the structural cause confirmed, the assistant's reasoning naturally arrives at the strategic implication:

The native engine's approach is compelling because it can handle flash-MLA-style coalesced attention with the tree visibility bitmask and proper paging all at once, whereas SGLang's architecture forces everything through triton with page_size=1. Even splitting the prefix attention (which doesn't need masking) from the tree attention (which does) would help, but SGLang's current implementation does the whole thing together, making the prefix part unnecessarily expensive—exactly the inefficiency that the native verify kernel is designed to solve.

This is the thesis that drives the remainder of the session. The assistant recognizes that the only way to break the page_size=1 bottleneck is to build a custom attention kernel that can handle the tree mask with coalesced memory access — a native CUDA kernel that does for the verify step what FlashMLA does for standard attention. The subsequent chunks in this segment (Chunk 0 and Chunk 1 of Segment 66) describe exactly that effort: building a custom sm_120 verify attention kernel, making it CUDA graph capture-safe, and achieving a 3–6× decode speedup over the Triton baseline.

Assumptions and Their Validity

The assistant makes several assumptions in this message, all of which are well-justified:

The cufall metric is more reliable than nvidia-smi's SM% for measuring real compute utilization. This is correct. Nvidia-smi's utilization.gpu is known to be a coarse metric that reports any kernel activity as utilization. The sm__pipe_tensor_cycles_active metric from cufall/Nsight measures actual tensor core pipeline activity, which is the relevant metric for a workload dominated by matrix multiplications.
The page_size=1 constraint is structural and cannot be easily worked around within SGLang. The source code confirms this — it's a hard raise RuntimeError. The constraint stems from the fundamental requirement that accepted tree nodes are non-contiguous in the KV cache, which is inherent to DDTree's tree-structured speculative decoding.
The native engine approach is the right solution. This is validated by the subsequent work in the session, where the custom kernel achieves 3–6× speedup. However, it's worth noting that the assistant does briefly consider alternatives (splitting prefix attention from tree attention, or switching to FlashInfer/FA3 backends) before concluding that the native kernel is the best path.
Memory latency, not kernel launch overhead or CPU tree construction, is the dominant bottleneck. The assistant mentions these alternatives but correctly identifies scattered KV reads as the primary cause. The subsequent profiler data (in Chunk 1 of Segment 66) confirms that CPU orchestration (tree-build at 1.8ms, mask-build at 0.18ms) is negligible compared to the verify attention cost.

Mistakes and Incorrect Assumptions

The assistant avoids several potential mistakes in this message:

It does not over-interpret the 3% tensor utilization as a GPU hardware problem. A less experienced engineer might conclude that the GPU is faulty or that the Blackwell architecture has a fundamental issue with decode. The assistant correctly identifies the bottleneck as a software architecture issue — the page_size=1 constraint in SGLang's DDTree implementation.
It does not prematurely recommend a backend switch. The assistant considers FlashInfer and FA3 as alternatives but correctly recognizes that DDTree's custom tree mask requirement makes a simple backend swap infeasible. The source code confirmation validates this caution.
It does not conflate the two separate bottlenecks. Earlier in the session, the assistant had identified both the MLA attention bottleneck (which scales with context length) and the MoE low-batch inefficiency (a fixed cost per step). In this message, it correctly keeps them separate: the page_size=1 issue is the dominant bottleneck for long-context decode, while MoE imbalance is a separate concern that becomes visible only after the attention bottleneck is fixed. One could argue that the assistant could have been more precise about why page_size=1 causes scattered access. The explanation is implicit — "accepted tree nodes are non-contiguous" — but the mechanism by which non-contiguous pages force uncoalesced memory loads is not spelled out. However, given the audience (the user is clearly an expert in GPU computing), this level of detail is appropriate.

Input Knowledge Required

To fully understand this message, the reader needs:

Knowledge of GPU architecture and profiling: Understanding the difference between SM utilization (nvidia-smi) and tensor pipe utilization (cufall), and why the latter is more meaningful for compute-bound workloads.
Knowledge of speculative decoding with DDTree: Understanding that DDTree proposes multiple candidate tokens in a tree structure, and the verify step must evaluate all of them against the full KV cache. The tree structure means accepted tokens are non-contiguous in the KV cache, which forces fine-grained page addressing.
Knowledge of the SGLang architecture: Understanding that SGLang uses a paged KV cache with configurable page_size, and that the MLA attention kernel's memory access pattern depends critically on page_size.
Knowledge of the previous diagnostic work: The assistant's earlier probes (power draw, memory controller utilization, commit length at various context lengths) provide the context that makes the cufall screenshots interpretable.
Knowledge of the project's strategic direction: The "native engine" refers to a custom C/C++/CUDA inference engine being built alongside the SGLang deployment, intended to eventually replace SGLang's DDTree implementation with optimized kernels.

Output Knowledge Created

This message creates several important pieces of knowledge:

Confirmed diagnosis: The decode bottleneck is latency-bound, caused by page_size=1 forcing scattered KV cache access in the Triton MLA verify kernel. The GPU is essentially idle during decode (3% tensor utilization), waiting on memory latency rather than doing useful work.
Structural confirmation: The page_size=1 constraint is hard-coded in SGLang's DDTree implementation and cannot be configured away. This is confirmed by reading the actual source code.
Validated strategy: The native engine approach — building a custom CUDA verify kernel that can handle tree masks with coalesced memory access — is confirmed as the correct path forward. SGLang's current architecture cannot be incrementally fixed; it requires a new kernel.
Two-bottleneck model: The decode slowdown has two components: the MLA attention bottleneck (which scales with context length and is caused by page_size=1) and the MoE low-batch inefficiency (a fixed cost per step). The attention bottleneck is the primary target for the custom kernel; MoE imbalance is a separate concern.
Diagnostic methodology: The message demonstrates a powerful diagnostic pattern: use multiple independent measurement tools (nvidia-smi, power monitoring, cufall) to cross-validate findings, and when they conflict, trust the more specific metric (tensor pipe cycles) over the coarse one (SM utilization).

The Thinking Process: A Window into Diagnostic Reasoning

The assistant's reasoning in this message reveals a disciplined diagnostic process. It begins by interpreting the user's evidence, immediately recognizing its significance. It then synthesizes this with its own probe data, building a unified picture. It considers alternative explanations (kernel launch overhead, CPU tree construction) but correctly identifies the most likely cause. It then reaches into the running service's source code to confirm the structural constraint, transforming a hypothesis into a confirmed diagnosis.

The reasoning also shows the assistant's awareness of the broader strategic context. It doesn't just diagnose the bottleneck — it connects the diagnosis to the project's ongoing work on the native engine, showing how the finding validates the existing roadmap. This is the mark of an engineer who understands not just the immediate problem but its place in the larger system.

The final line — "Let me confirm the structural cause (page_size=1 + custom-mask backend lock) precisely" — is a model of scientific humility. The assistant has strong evidence for its diagnosis, but it doesn't stop there. It seeks direct confirmation from the source code, turning a probabilistic inference into a deterministic fact.

Conclusion

Message [msg 12189] is a turning point in the session. Before this message, the decode bottleneck was a mystery — the assistant knew something was wrong (high SM utilization but low power draw, low memory bandwidth) but couldn't definitively pin down the cause. After this message, the bottleneck is understood at every level: the hardware level (3% tensor utilization, 133W power draw), the software level (page_size=1 in Triton MLA), and the architectural level (DDTree's non-contiguous tree nodes force fine-grained page addressing).

This understanding unlocks the next phase of the project: building a custom sm_120 verify attention kernel that can handle tree masks with coalesced memory access. The subsequent work in this segment — making the kernel CUDA graph capture-safe, optimizing it for occupancy, achieving 3–6× speedup over Triton — is all predicated on the diagnosis confirmed in this message. Without this diagnostic clarity, the optimization effort would have been aimless, a blind search for improvements in an unknown space.

The message also serves as a case study in effective GPU performance debugging. It demonstrates the importance of using multiple independent measurement tools, of understanding what each metric actually measures (and doesn't measure), of seeking structural confirmation from source code, and of connecting local findings to strategic direction. For anyone working on GPU-accelerated LLM inference, the diagnostic pattern on display here — from symptom to hypothesis to confirmation to action — is worth studying.