The Screenshot That Sealed the Diagnosis: How Visual GPU Profiling Confirmed a 130× Attention Bottleneck

Introduction

In the midst of an intense optimization session for speculative decoding on NVIDIA RTX PRO 6000 Blackwell GPUs, a user message arrived that contained nothing more than two screenshots and a brief description. Yet this seemingly simple message at index 12188 was a pivotal moment in the conversation—it provided the decisive visual evidence that confirmed a weeks-long investigation into why long-context decode performance was catastrophically slow. The message reads:

two screenshots of a high freq gpu sampler during previous run, prefill is quite active with bottleneck on saturated experts (red), but decode is mostly idle), ./cufall --metric sm__pipe_tensor_cycles_active.avg --denominator gr__cycles_elapsed.max --color

The message also includes the system's record of calling a "Read tool" to load the two screenshot files from disk. This terse, almost telegraphic communication—a user sharing two images and a recommended profiling command—was the culmination of a diagnostic chain that had been building across dozens of prior messages. To understand why this message matters, one must understand the context that preceded it and the explosive implications of what those screenshots revealed.

The Context: A Mystery of Vanishing Throughput

The conversation leading up to message 12188 was a deep-dive investigation into the performance characteristics of speculative decoding using DFlash with DDTree (Draft Tree) on the GLM-5-NVFP4 / Kimi K2.6 model family. The team had deployed a custom SGLang-based inference service on a machine with 8× RTX PRO 6000 Blackwell GPUs and was attempting to use speculative decoding—a technique where a smaller, faster "drafter" model proposes multiple candidate tokens that a larger "target" model verifies in parallel—to accelerate generation.

The problem was stark: at short context lengths (~1k tokens), the system achieved respectable throughput with commit lengths of 7–8 tokens per step. But as context grew to 64k, 128k, and 185k tokens, decode throughput collapsed. At 185k context, the system was generating a mere 0.7 tokens per second. The per-step time ballooned from 8 milliseconds at short context to 1.4 seconds at 185k—a 175× increase that could not be explained by the growth in KV cache size alone.

The assistant had been systematically investigating possible causes. Was the drafter's sliding-window attention failing at long context? Was the YaRN (Yet another RoPE scaling) positional encoding breaking down? Was the draft KV cache corrupting across steps? Through a series of carefully designed probe experiments using trivially predictable text ("The cat sat on the mat" repeated thousands of times), the assistant ruled out all of these hypotheses. The drafter maintained commit lengths of 7–8 tokens at every context length from 1k to 128k when the text was predictable. The commit_len=1 observed in earlier benchmarks was purely an artifact of the synthetic prompt producing novel prose that the drafter could not predict—not a bug in the long-context pipeline.

But this left a deeper mystery: if the drafter was working correctly, why was decode so slow? The assistant had already gathered nvidia-smi metrics showing 99.8% SM utilization during decode, yet power draw was only 133W (22% of TDP) and memory controller utilization was a mere 1.2%. This suggested the GPU was "busy" in name only—kernels were resident on the streaming multiprocessors but doing almost no useful work. The assistant hypothesized that the root cause was page_size=1 in the KV cache, which forced the Triton-based MLA (Multi-head Latent Attention) kernel to perform scattered, non-coalesced memory access. The effective bandwidth was calculated at approximately 14 GB/s—roughly 130× below the 1.8 TB/s peak memory bandwidth of the RTX PRO 6000.

What the Screenshots Revealed

This is where message 12188 enters the story. The user had independently run a high-frequency GPU sampler using a tool called cufall—a CUDA-level profiling utility that measures actual tensor core pipeline activity rather than the coarse-grained kernel residency that nvidia-smi reports. The two screenshots captured the tensor core utilization (sm__pipe_tensor_cycles_active.avg / gr__cycles_elapsed.max) across all 8 GPUs during both prefill and decode phases.

The prefill screenshot showed tensor cores operating at 50–90% utilization across all streaming multiprocessors, with significant red coloring indicating hot spots on saturated expert modules. This was a healthy, compute-bound profile—the GPU was doing real work during prefill, saturating the tensor cores with matrix multiplications from the Mixture-of-Experts (MoE) layers.

The decode screenshot told a radically different story: tensor core utilization was approximately 3% across every SM on every GPU. Not 30%. Not 10%. Three percent. The GPU was essentially idle during decode, despite nvidia-smi reporting 99.8% utilization. The discrepancy between these two metrics—nvidia-smi's "SM%" and cufall's tensor-active cycles—exposed the fundamental nature of the bottleneck.

This was the smoking gun. The assistant had been reasoning from indirect evidence (power draw, memory controller utilization, bandwidth calculations) toward the conclusion that decode was latency-bound rather than compute-bound or bandwidth-bound. The screenshots provided direct, visual confirmation at the hardware level. The tensor cores—the very units that perform the matrix multiply-accumulate operations at the heart of attention and MLP layers—were nearly silent during decode. The GPU was spending its cycles on memory latency stalls, pointer chasing through the fragmented KV cache, and the overhead of launching thousands of tiny serial kernels across 61 layers at batch size 1.

The Thinking Process Visible in the Assistant's Response

The assistant's reasoning in the subsequent message (msg 12189) shows exactly how it interpreted this evidence. The key insight was recognizing that cufall's tensor-active metric is "far more honest than nvidia-smi's 'SM%'" because it measures actual compute pipeline occupancy rather than mere kernel residency. The assistant immediately connected this to its earlier bandwidth calculations: the 14 GB/s effective bandwidth, the 133W power draw, the 1.2% memory controller utilization. All of these independent measurements converged on the same diagnosis.

The assistant also used this opportunity to verify the structural constraint that made the bottleneck unavoidable: DDTree requires page_size=1 because "accepted tree nodes are non-contiguous in the verify block." This is a fundamental architectural constraint of the draft-tree verification algorithm—the tree structure means the attention mechanism must gather KV cache entries from scattered locations, which is incompatible with the coalesced memory access patterns that larger page sizes enable. The verify attention is also locked to the Triton backend because of the custom tree attention mask, preventing a simple swap to FlashInfer or FlashMLA.

This created a clear picture: the latency-bound decode bottleneck was not a bug that could be patched, but a structural limitation of running DDTree verification through SGLang's Triton-based MLA backend with page_size=1. The fix would require either a custom attention kernel optimized for the tree-mask pattern on sm_120 (the Blackwell consumer architecture) or a fundamental redesign of how the verify step accesses KV cache.

Assumptions and Their Validation

The user's message made several implicit assumptions that turned out to be correct. First, that the high-frequency GPU sampler data would be more informative than nvidia-smi's aggregate metrics—this was validated when cufall revealed the 3% tensor utilization that nvidia-smi's 99.8% SM% had hidden. Second, that the prefill/decode dichotomy was worth highlighting—the contrast between 50–90% tensor utilization during prefill and 3% during decode was the key diagnostic signal. Third, that the user's observation of "decode is mostly idle" was accurate despite the conflicting nvidia-smi numbers.

The message also included a suggested command invocation (./cufall --metric sm__pipe_tensor_cycles_active.avg --denominator gr__cycles_elapsed.max --color), which served both as documentation of how the screenshots were generated and as a recommendation for future profiling. This command-level detail reveals the user's deep familiarity with CUDA profiling tools and their understanding that tensor-core activity, not SM residency, is the correct metric for diagnosing compute bottlenecks.

Input Knowledge Required

To fully understand this message, a reader would need familiarity with several domains: GPU architecture (the distinction between SM occupancy and tensor-core pipeline activity, the significance of memory controller utilization and power draw as diagnostic signals), speculative decoding (the drafter-target paradigm, tree verification, commit length metrics), the specific model architecture (MLA with KV latent cache, MoE layers, YaRN scaling), and the SGLang inference framework's internals (page_size configuration, attention backend selection, the DDTree verification algorithm). Without this background, the message reads as a cryptic note about two images; with it, the message becomes a decisive data point in a complex optimization effort.

Output Knowledge Created

This message created several important outputs. First, it independently validated the assistant's hypothesis about the decode bottleneck being latency-bound rather than compute-bound or bandwidth-bound—the 3% tensor utilization was direct hardware evidence that confirmed the indirect measurements. Second, it sharpened the diagnosis from "something is slow" to "the tensor cores are essentially idle during decode," which focused subsequent optimization efforts on reducing memory latency and kernel overhead rather than increasing compute throughput. Third, it motivated the decision to build a custom sm_120 verify attention kernel, which became the central project of the following chunks—if the Triton backend was structurally incapable of efficient long-context decode, the only path forward was to write a kernel that could handle tree-masked attention with coalesced memory access.

Conclusion

Message 12188 is a masterclass in the power of the right diagnostic tool. Where nvidia-smi reported 99.8% utilization and suggested a compute-bound system, cufall's tensor-active metric revealed 3% utilization and exposed a latency-bound pathology. The two screenshots transformed the investigation from hypothesis-driven inference (calculating bandwidth, measuring power draw, questioning nvidia-smi artifacts) to direct observation. They confirmed that the GPU was almost completely idle during decode, that the 130× bandwidth gap was real, and that the structural constraints of DDTree's page_size=1 requirement were the root cause. This message didn't just answer the question of why decode was slow—it redirected the entire optimization strategy toward building custom kernels that could work around SGLang's architectural limitations, setting the stage for the dramatic 3–6× speedup achieved in subsequent work.