Diagnostics at the Frontier: Benchmarking Autoregressive Baselines to Unlock GPU Utilization in Speculative Decoding

Introduction

In the high-stakes world of large language model inference deployment, few moments are as revealing as the first power reading. After weeks of infrastructure wrestling—CUDA toolkit mismatches, Triton JIT compilation failures, FlashInfer SM120 rejections, and a 590 GB model download—the assistant had finally deployed Kimi K2.6 with DDTree (Draft-Tree) speculative decoding on an 8× B300 SXM6 machine with NVLink interconnect. The throughput numbers were impressive: 303 tok/s at concurrency 1, scaling to over 4700 tok/s at high concurrency. But the user noticed something puzzling in message [msg 11785]: the GPUs were drawing only 300–400 watts out of their 1100-watt capacity. At 100% utilization reported by nvidia-smi, where was all the headroom going?

Message [msg 11787] is the assistant's response to this observation—a masterclass in systematic diagnostic reasoning that bridges hardware-level power metrics, software-level parallelism strategies, and the subtle interplay between speculative decoding overhead and communication bottlenecks. This single message captures a critical inflection point in the deployment: the moment when the team pivots from "does it work?" to "how do we make it faster?" It is a message about asking the right questions, gathering the right data, and letting the numbers guide the next move.

The Context: A Pivot Point in the Deployment

To understand why this message matters, we must first understand the journey that led here. The broader session (Segment 64 of the conversation) had been a marathon of infrastructure triage and performance optimization. The assistant had:

  1. Fixed CUDA toolkit and FlashInfer SM120 compatibility issues for Blackwell GPUs
  2. Systematically benchmarked four parallelism strategies (TP8, PP8, EP8, EP4) on a PCIe-based 8× RTX PRO 6000 system
  3. Deployed DFlash speculative decoding for Kimi K2.6 with sliding window attention
  4. Built a comprehensive benchmark harness for DDTree evaluation
  5. Fixed a coding evaluation code extraction bug
  6. Debugged a config sweep readiness race condition
  7. Deployed K2.6+DDTree on the B300 SXM6 NVLink machine The B300 deployment had been particularly dramatic: streaming a working virtual environment from a CT200 machine, downloading the 590 GB model via aria2 at 575 MiB/s, fixing a Triton JIT compilation issue (missing Python.h), skipping the vision tower warmup (which required unavailable flash_attn.cute), and finally getting the service to launch. The initial DDTree benchmarks showed strong results—303 tok/s at C=1 (2.15× over the 133 tok/s autoregressive baseline) and 4723 tok/s at C=128, with 5/5 coding correctness. But the power draw told a different story. The GPUs were sipping power at 300–400W, barely a third of their thermal design power. This is the kind of observation that separates a surface-level deployment from a deeply optimized one. The user's question in [msg 11785]—"Only 300-400W power use out of 1100W, maybe worth trying EP now that we have nvlink with multicast support? Cuda graphs working?"—cut straight to the heart of the matter. Were the GPUs truly idle despite showing 100% utilization? Was the bottleneck in communication, in speculative verification overhead, or in the parallelism strategy itself?

The Reasoning Process: Connecting Hardware Signals to Software Decisions

The assistant's reasoning in [msg 11787] reveals a sophisticated mental model of the inference stack, connecting observations across multiple layers of abstraction. The first order of business is to answer the user's specific questions.

On CUDA graphs: The assistant checks the service logs and finds a distribution of 51 decode batches with CUDA graphs enabled versus 80 without. The immediate interpretation—"the 80 'False' are prefills, which is normal"—demonstrates deep familiarity with SGLang's internals. CUDA graphs in SGLang are used to accelerate the decode phase (where the model generates tokens one at a time) by capturing a sequence of GPU operations into a reusable graph that can be launched with a single kernel invocation. Prefill phases (processing the input prompt) are typically not graph-captured because their shapes vary with input length. The 51/80 split is thus not a sign of partial failure but of correct behavior: decode batches are being accelerated, prefill batches are not. This distinction matters because it tells the assistant that CUDA graph optimization is already active and functioning correctly—the low power draw is not due to missing graph capture.

On the bottleneck: The assistant then connects the low throughput at C=64 (3795 tok/s with DDTree, compared to the autoregressive baseline that would soon be measured at 3870 tok/s) to "spec verify overhead and communication being the bottleneck rather than compute." This is a crucial insight. Speculative decoding works by having a small draft model propose candidate tokens that the large target model then verifies in parallel. The verification step requires running the target model's forward pass on multiple candidate sequences simultaneously—a compute-intensive operation that should, in theory, saturate the GPUs. But the power draw suggests the GPUs are not fully engaged. The assistant hypothesizes that the bottleneck is not in the compute kernels themselves but in the communication required to synchronize across the 8 GPUs during tensor-parallel verification.

The optimization landscape: The assistant then maps out three intervention points:

The Experimental Design: Measuring What Matters

The assistant's experimental design in this message is a model of efficient benchmarking under real-world constraints. The autoregressive baseline service was already starting from the previous message ([msg 11784]), so the assistant sets up a polling loop to wait for it to become ready, then runs a targeted benchmark.

The benchmark parameters are carefully chosen:

The Results: Data That Tells a Story

The autoregressive baseline results arrive after a 120-second wait for the service to become ready:

     ctx    C  agg tok/s  per-req   wall  prompt
      60    1      132.7    132.8    3.1      52
      60   32     2260.3     77.7    6.6      35
      60   64     3870.2     66.0    7.8      40

These numbers are immediately informative. The single-stream throughput of 132.7 tok/s on B300 NVLink is roughly 2× the ~65 tok/s seen on the PCIe PRO 6000 system for the EP8 configuration—confirming that NVLink's higher bandwidth dramatically improves tensor-parallel scaling. But the aggregate throughput at C=64 (3870 tok/s) represents only about 29× scaling from the single-stream rate, suggesting diminishing returns at high concurrency due to contention or communication overhead.

The power data is even more telling:

0, 100, 362.75
1, 100, 361.27
2, 100, 361.65
3, 100, 368.67
4, 100, 359.12
5, 100, 369....

All 8 GPUs show 100% utilization but draw only ~360–370W each—roughly one-third of their 1100W thermal design power. This is the diagnostic goldmine. The GPUs are reporting 100% utilization because they are never idle—there is always work to do—but they are not drawing peak power because the work they are doing is memory-bandwidth-bound or communication-bound, not compute-bound. A compute-bound kernel would saturate the GPU's arithmetic units and draw closer to the TDP. A memory-bound or communication-bound kernel, by contrast, spends most of its time waiting for data movement, with the compute units idly stalled.

This observation validates the user's intuition and the assistant's hypothesis: the bottleneck is not in the compute kernels but in the data movement. The GPUs have ample compute headroom—they could be drawing 3× more power if the workload were compute-saturated—but they are spending their cycles waiting for AllReduce completions, memory fetches, or speculative verification synchronization.

Input Knowledge Required

To fully appreciate the reasoning in this message, one needs a fairly deep understanding of several technical domains:

CUDA Graphs: A CUDA feature that allows a sequence of GPU kernel launches to be captured and replayed as a single unit, eliminating kernel launch overhead and enabling optimizations like grid-wide synchronization. In SGLang, CUDA graphs are used to accelerate the decode phase of transformer inference, where the same sequence of operations (attention, MLP, etc.) repeats for each generated token.

Tensor Parallelism (TP) vs. Expert Parallelism (EP): Two strategies for distributing a large model across multiple GPUs. TP shards each tensor (weight matrix) across GPUs, requiring AllReduce operations to synchronize results after each layer. EP, designed for Mixture-of-Experts models, places different experts on different GPUs, so each GPU computes its experts independently without cross-GPU communication during the expert feedforward—only a token routing step (all-to-all) is needed. On PCIe systems, EP dramatically outperforms TP because it avoids the bandwidth-limited AllReduce. On NVLink systems, the tradeoff is less clear-cut because NVLink provides much higher bandwidth for AllReduce.

NVLink and NVLS (NVLink Switch): NVIDIA's high-bandwidth GPU interconnect. NVLink provides direct GPU-to-GPU communication at significantly higher bandwidth than PCIe. NVLS (NVLink Switch) extends this with multicast capabilities, allowing one GPU to send data to multiple recipients simultaneously—a natural fit for tensor-parallel AllReduce where one GPU's partial result must be broadcast to all others.

Speculative Decoding and DDTree: A technique where a small, fast draft model proposes multiple candidate token sequences (organized as a tree), and the large target model verifies them in parallel. DDTree (Draft-Tree) is a specific implementation that constructs a tree of draft tokens and uses a tree attention kernel to verify all branches simultaneously. The overhead comes from the tree construction, the tree attention computation, and the synchronization between draft and target models.

GPU Power Draw as Diagnostic: GPU power draw is a surprisingly informative metric for understanding workload characteristics. Compute-bound kernels (matrix multiplications, convolutions) tend to draw near TDP because they keep the arithmetic units active. Memory-bound kernels (attention, element-wise operations) draw less power because the compute units stall waiting for data. Communication-bound workloads (AllReduce, all-to-all) draw even less because the GPUs spend significant time waiting for data from other GPUs over the interconnect.

Assumptions and Potential Pitfalls

The assistant's reasoning in this message rests on several assumptions that are worth examining:

That the CUDA graph distribution (51 decode, 80 prefill) is normal. This is a reasonable assumption based on knowledge of SGLang's internals, but it's worth noting that the assistant doesn't verify this by checking whether the 51 graph-captured batches actually correspond to decode phases. A more thorough investigation might correlate the graph usage with the generation phase of each request. However, in practice, this distribution pattern is well-known in the SGLang community, and the assumption is safe.

That NVLS flags will improve performance on this specific workload. NVLS multicast is designed to accelerate broadcast and reduce operations in tensor parallelism. However, the benefit depends on the specific communication pattern and whether the SGLang implementation actually uses NVLS-capable collectives. The assistant acknowledges this uncertainty by framing the test as an experiment rather than a guaranteed improvement.

That the power measurement during a single burst is representative. The assistant samples power during one long-running generation request, which provides a snapshot but not a statistically robust measurement. Power draw can vary with the specific token sequences being generated, the batch size dynamics, and the phase of generation (prefill vs. decode). A more rigorous approach would average multiple samples across different workloads. However, for a quick diagnostic, the single sample is sufficient to confirm the qualitative observation that GPUs are far from their TDP.

That the autoregressive baseline is directly comparable to the DDTree run. The autoregressive service was started with --skip-server-warmup and without speculative decoding flags. While this provides a valid no-spec baseline, it's possible that the warmup phase affects performance (e.g., through Triton autotuning or CUDA graph capture). The assistant mitigates this by running both services on the same hardware with the same model and similar configuration.

Output Knowledge Created

This message creates several valuable pieces of knowledge that inform the next phase of optimization:

  1. Confirmed CUDA graph functionality: The logs show CUDA graphs are active on decode batches, ruling out graph capture issues as the cause of low utilization.
  2. Autoregressive baseline established: The B300 TP8 autoregressive throughput is 132.7 tok/s at C=1, 2260.3 at C=32, and 3870.2 at C=64. These numbers serve as the reference point for all future speculative decoding speedup calculations.
  3. Power draw characterization: Under sustained load, each B300 GPU draws ~360–370W at 100% utilization, confirming the system is memory/communication-bound rather than compute-bound. This is the key diagnostic finding that drives the optimization strategy.
  4. Validation of the optimization direction: The data supports the hypothesis that EP8 + NVLS is worth exploring. If the bottleneck is communication (as the low power draw suggests), then reducing communication through expert parallelism or accelerating it through NVLS multicast should improve throughput.
  5. Methodology for future benchmarks: The assistant establishes a pattern of measuring both throughput and power draw, providing a template for systematic performance evaluation.

Conclusion: The Pivot Point

Message [msg 11787] represents a critical transition in the deployment workflow. It is the moment when the team moves from "is it working?" to "how do we make it optimal?" The assistant's response is a model of systematic diagnostic reasoning: it answers the user's specific questions with evidence from logs, formulates a hypothesis about the bottleneck based on throughput and power data, designs an experiment to gather the necessary baseline, and executes it efficiently. The results—the autoregressive baseline numbers and the power draw snapshot—provide the foundation for the next phase of optimization, where the assistant will explore EP8 with NVLS, test larger DDTree budgets, and ultimately write a comprehensive findings report.

What makes this message particularly instructive is how it connects observations across multiple layers of the system: the CUDA graph log entries, the throughput numbers from the benchmark harness, the power readings from nvidia-smi, and the architectural knowledge of tensor parallelism, expert parallelism, and speculative decoding. Each piece of data informs the interpretation of the others, creating a coherent picture of where the system's bottlenecks lie and what to do about them. It is a reminder that in ML infrastructure optimization, the most valuable skill is not knowing the answers but knowing which questions to ask—and how to let the hardware tell you what it needs.