The <1% Signal: How a User's cufall Profile Exposed the Tensor-Core Bottleneck in Blackwell Decode

The Message

In the middle of an intensive optimization campaign on a DeepSeek-V4-Flash deployment across 8× NVIDIA RTX PRO 6000 (Blackwell sm_120) GPUs, the user delivered a short but decisive message:

My compute profile showed dramatically low compute util, <1% through ./cufall --devices all --metric sm__pipe_tensor_cycles_active.avg --denominator gr__cycles_elapsed.max --color --hw-buffer-mb 250 --frame-us 50000

This single sentence, backed by a concrete measurement from the cufall GPU profiling tool, crystallized the entire optimization effort. The message reports that tensor-pipe utilization—the fraction of GPU cycles where the tensor cores are actively processing—is below one percent during decode. On a Blackwell GPU whose primary architectural advantage is its FP4/FP8 tensor-core throughput, this finding is devastating: the model's decode phase is essentially ignoring the hardware's most powerful compute units.

Why the Message Was Written: Reasoning, Motivation, and Context

The user's message did not appear in a vacuum. It arrived at a critical inflection point in a multi-session optimization campaign (see [msg 12474] and [msg 12482] for the immediate preceding context). The assistant had just completed a methodical, three-configuration benchmark sweep: a baseline without optimizations, Config A with FP8 autotune configs and NCCL LL protocol, and Config B adding MTP (multi-token prediction) speculative decoding. The results were sobering. Aggregate throughput at concurrency C≥8 was hard-capped at approximately 23 tokens per second, and none of the configuration levers—FP8 kernel tuning, NCCL protocol changes, or speculative decoding—had moved the needle beyond incremental single-request gains.

The assistant's own profiling had identified _tiled_sparse_decode_kernel, a Triton fallback for sparse MLA attention, as consuming 63% of decode time. But this was an internal, model-level observation. The user's cufall measurement provided the complementary, hardware-level perspective: the tensor cores—the very feature that makes Blackwell expensive—were sitting nearly idle. The user was not merely reporting a number; they were delivering a diagnosis that reframed the entire problem. The bottleneck was not a matter of tuning parameters or selecting the right speculative decoding algorithm. It was structural: the software stack (SGLang's Triton-based sparse attention kernel, the MXFP4 MoE slot-GEMV implementation) was executing on CUDA cores (SIMT) rather than the FP4/FP8 tensor cores (SIMD) that Blackwell was designed for.

The user's motivation was to accelerate the assistant's diagnosis. They had run an independent measurement using a tool (cufall) that the assistant had not yet deployed, and the result was unambiguous. The message implicitly says: "Stop chasing configuration knobs. The problem is at the kernel level. The tensor cores are idle."

How Decisions Were Made

This message triggered an immediate and decisive shift in the assistant's strategy. In the very next round ([msg 12485]), the assistant's reasoning shows the pivot: "Critical signal: <1% tensor-pipe-active means decode is running almost entirely on CUDA cores (FMA/bmm), not the FP4/FP8 tensor cores — consistent with the slot-GEMV MoE (tl.sum) and the bmm-based sparse decode. That makes 'write tensor-core kernels' the core lever."

The decision flow was:

  1. Accept the user's measurement as authoritative. The assistant did not question the methodology or ask for a repeat. The cufall tool's metric sm__pipe_tensor_cycles_active.avg normalized by gr__cycles_elapsed.max is a standard NVIDIA GPU utilization metric. The <1% figure was internally consistent with the assistant's own profiling showing 63% time in a non-tensor-core kernel.
  2. Validate the finding by attempting to replicate it. The assistant tried to locate cufall on the remote machine and considered running a C=16 profile with the same tool, but the user clarified ([msg 12486]) that cufall is too interactive for automated use. The assistant pivoted to using torch.profiler under C=16 load instead, which confirmed the kernel ranking.
  3. Commit to the Tier-B kernel strategy. The assistant's earlier findings document had already proposed "split-K sparse-MLA decode + grouped FP4 MoE" as the next attack vector. The user's <1% tensor utilization signal elevated this from a hypothesis to a confirmed diagnosis.

Assumptions Made by the User and Agent

The user's message makes several implicit assumptions. First, that the assistant understands the significance of the sm__pipe_tensor_cycles_active metric—which it does, correctly interpreting it as tensor-core utilization. Second, that the <1% figure is representative of steady-state decode behavior rather than an artifact of the profiling window (e.g., capturing idle time between requests). The --frame-us 50000 flag (50-millisecond sampling frames) and --hw-buffer-mb 250 (250 MB hardware buffer) suggest a reasonable profiling setup, but the user does not specify whether this was measured under concurrency load or single-request decode. Third, the user assumes that the assistant has the capability to write custom CUDA kernels targeting tensor cores—an assumption validated by the earlier K2.6 work where the assistant built a custom sm_120 verify attention kernel achieving 3–6× speedup over Triton.

The assistant's own assumptions are visible in its reasoning: it immediately maps "tensor cores idle" to "the sparse decode and MoE kernels are running on CUDA cores." This is correct, but it carries the implicit assumption that fixing the tensor-core path will yield proportional throughput gains. In practice, even if the attention kernel were rewritten for tensor cores, other bottlenecks (memory bandwidth, MoE imbalance, PCIe communication) could cap the improvement. The assistant is aware of this—its earlier profiling showed MoE slot-GEMV at 39% and sparse decode at 38% at C=16—so the assumption is that attacking both kernels in sequence is the correct prioritization.

Mistakes or Incorrect Assumptions

There are no factual errors in the user's message, but it is worth examining what it does not say. The <1% figure is an aggregate across all GPU pipes. On Blackwell (sm_120), there are multiple compute pipes: tensor, CUDA (FP32/FP64), and special-function units (SFU). A low tensor-pipe utilization does not necessarily mean the GPU is underutilized overall—the CUDA cores could be fully occupied. In fact, the assistant's own profiling confirms that the CUDA cores are busy: the Triton fallback kernel and the slot-GEMV kernels are occupying the CUDA pipes. The user's framing ("dramatically low compute util") could be misinterpreted as "the GPU is idle," whereas the correct interpretation is "the GPU is busy, but on the wrong compute units."

The assistant's response does not fall into this trap—it correctly interprets the signal as "decode is on CUDA cores, not tensor cores"—but a less experienced reader might conflate low tensor utilization with low overall utilization. The distinction matters because the optimization strategy is different: if the GPU were truly idle (underutilized), the fix might be increasing batch size or reducing pipeline bubbles. If the GPU is busy on the wrong units, the fix is kernel replacement.

Another subtle point: the user ran cufall with --devices all, meaning it profiled all 8 GPUs simultaneously. On a TP4 (tensor parallelism 4) deployment split across two NUMA nodes (GPUs 0–3 for prefill, GPUs 4–7 for decode), the decode GPUs (4–7) would show the <1% tensor utilization, while the prefill GPUs might show different characteristics. The aggregate across all devices could dilute the signal. The assistant does not question this aggregation, but it is worth noting that a per-device breakdown might have shown even more extreme underutilization on the decode GPUs specifically.

Input Knowledge Required to Understand This Message

To fully grasp the significance of this message, a reader needs:

  1. Knowledge of the Blackwell (sm_120) architecture: The RTX PRO 6000 Blackwell GPU features fourth-generation tensor cores with FP4 and FP8 support, offering vastly higher throughput for quantized matrix operations than CUDA-core execution. The sm__pipe_tensor_cycles_active metric specifically measures activity on these tensor-core pipelines.
  2. Understanding of the DeepSeek-V4-Flash model architecture: The model uses Multi-head Latent Attention (MLA) with sparse (top-k) selection and Mixture-of-Experts (MoE) layers. Both of these operations have fast tensor-core paths on sm_100 (Hopper) GPUs via fused kernels (DeepGEMM, trtllm-gen, FP4 indexer), but these paths are architecture-gated and fall back to Triton/CUDA implementations on sm_120.
  3. Familiarity with the cufall profiling tool: cufall is a lightweight GPU profiling utility that reads hardware performance counters via NVIDIA's CUPTI or nvml interfaces. The specific metric sm__pipe_tensor_cycles_active.avg counts cycles where the tensor-core pipeline is actively processing, normalized by total elapsed cycles (gr__cycles_elapsed.max). The --hw-buffer-mb 250 and --frame-us 50000 parameters control the sampling buffer size and frame duration.
  4. Context of the optimization campaign: The message arrives after the assistant has exhausted configuration-level levers (FP8 autotuning, NCCL protocol tuning, MTP speculative decoding) and identified the Triton sparse-decode kernel as the primary bottleneck. The user's measurement provides the hardware-level confirmation that the assistant's software-level analysis was correct.

Output Knowledge Created by This Message

This message created several critical outputs:

  1. A confirmed diagnosis: The <1% tensor utilization transforms the bottleneck from a hypothesis ("the Triton fallback kernel might be slow") into a confirmed hardware-level observation ("the tensor cores are idle during decode"). This is a qualitatively different kind of knowledge—it ties the software behavior to the hardware capability.
  2. A prioritization signal: The message implicitly ranks the optimization opportunities. Before this measurement, the assistant could have reasonably pursued FP8 GEMM tuning (which PR #25696 claimed gave 4–5× improvement on other hardware), NCCL protocol optimization, or speculative decoding tuning. After this measurement, the only strategy that addresses the root cause is kernel replacement targeting tensor cores.
  3. A boundary condition for the project: The user's throughput target was approximately 1000 tok/s. The <1% tensor utilization explains why the current ~23 tok/s is so far from that target: the model is running on the wrong compute units entirely. Closing a ~40× gap requires not tuning but fundamentally rewriting the compute path.
  4. Validation of the assistant's analytical approach: The assistant had already proposed the Tier-B kernel strategy in its findings document before receiving this message. The user's independent measurement confirmed that the assistant's diagnosis was correct, building trust and alignment in the collaborative process.

The Thinking Process Visible in Reasoning

The assistant's reasoning in response to this message ([msg 12485]) reveals a tight, three-step analytical process. First, it immediately recognizes the metric's significance: "Critical signal: <1% tensor-pipe-active means decode is running almost entirely on CUDA cores (FMA/bmm), not the FP4/FP8 tensor cores." This is not a rote repetition of the user's claim—it is an interpretation that connects the hardware counter to the specific kernel implementations (slot-GEMV MoE using tl.sum, sparse decode using bmm).

Second, the assistant maps this finding to the existing strategy: "That makes 'write tensor-core kernels' the core lever." This shows that the assistant already had a mental model of the solution space (Tier A: config tuning; Tier B: kernel replacement) and the user's signal cleanly selects Tier B.

Third, the assistant immediately attempts to operationalize the finding: it tries to locate cufall on the remote machine, polls for Config A server readiness, and plans a C=16 profile to rank the kernels. When the user clarifies that cufall is too interactive for automated use ([msg 12486]), the assistant pivots to torch.profiler without hesitation. This flexibility—accepting a constraint and finding an alternative measurement path—is characteristic of effective collaborative debugging.

The assistant also demonstrates intellectual humility. It does not claim credit for the finding or treat it as redundant with its own profiling. Instead, it treats the user's measurement as complementary and decisive: the assistant's profiling showed which kernel was slow; the user's profiling showed why it was slow (running on CUDA cores rather than tensor cores). Together, the two measurements form a complete diagnosis.

Conclusion

The user's four-line message reporting <1% tensor-pipe utilization is a masterclass in concise, high-signal communication. It arrives at precisely the right moment—after configuration tuning has been exhausted, before kernel development begins—and delivers exactly the information needed to make a strategic decision. It confirms the assistant's analysis, validates the proposed kernel-replacement strategy, and provides a hardware-level explanation for the performance ceiling that no amount of software tuning could address. In a conversation spanning dozens of messages, tool calls, and benchmark runs, this single measurement may be the most consequential: it answers the question "why is the model slow?" with the definitive answer "because the tensor cores are idle," and in doing so, it charts the only path forward.