The 33 Tokens-Per-Second Wall: Diagnosing Memory-Latency-Bound Inference on Blackwell GPUs

Introduction

In the high-stakes world of large language model inference, few moments are as revealing as the one where every measurement tool points to the same bottleneck. This is the story of such a moment—a single message from an AI assistant engaged in a deep optimization campaign for DeepSeek-V4-Flash, a state-of-the-art mixture-of-experts model running on eight NVIDIA RTX PRO 6000 Blackwell GPUs. The message, indexed as <msg id=12532> in the conversation, represents a pivotal diagnostic synthesis: the assistant takes three seemingly disconnected observations from the user—about benchmark measurement methodology, PCIe bandwidth utilization, and GPU power draw—and weaves them into a single, coherent, and actionable diagnosis of why throughput is stuck at approximately 33 tokens per second.

This article examines that message in detail: the reasoning that produced it, the assumptions that underpin it, the knowledge it draws upon, and the knowledge it creates. It is a case study in how expert practitioners diagnose performance bottlenecks in modern ML inference systems, and a window into the kind of multi-dimensional reasoning required when the usual tools—profiling, benchmarking, hardware metrics—all point in the same direction but require a human (or AI) interpreter to connect the dots.

The Context: An Exhaustive Optimization Campaign

To understand <msg id=12532>, one must understand what came before it. The conversation leading up to this message spans dozens of rounds across multiple segments, representing weeks of engineering effort compressed into an intense coding session. The assistant and user have been optimizing DeepSeek-V4-Flash—a massive MoE (mixture-of-experts) model with sparse MLA (Multi-head Latent Attention)—on a machine equipped with eight RTX PRO 6000 Blackwell GPUs (sm_120 architecture). The journey has included:

The Quoted Message

Let us examine the assistant's response in full. The message begins with a lengthy reasoning section—the assistant's internal deliberation—followed by a structured response to the user and a set of tool calls to execute the next diagnostic step.

## Agent Reasoning The user is questioning whether the benchmark is measuring thinking tokens correctly, noting that throughput starts low during the ramp-up phase at C=64 (around 5.00) before climbing to 32.85 later, and asking why early output is slower than subsequent output.

>

The issue isn't with thinking token measurement—it's a prefill scheduling artifact. When 64 prompts are submitted, they all need prefill first (256 input tokens each), which serializes and stalls the decode phase. The median TTFT of 14 seconds at C=64 confirms the prefill queue is massive. Once all 64 requests move into decode, throughput stabilizes at 32.85. The early-slow pattern is purely from the prefill ramp, not a measurement problem. The benchmark correctly counts output tokens, but the reported throughput at C=64 (29.71) gets dragged down by that long prefill latency phase.

>

The PCIe communication at C=64 being only ~200MB/s is actually reassuring—it means the GPUs aren't communication-bound. That's negligible compared to PCIe5 x16's 64GB/s capacity, which indicates the GPUs are spending most of their time in compute kernels with long gaps between all-reduces. This aligns with the profile showing NCCL at just 1-2%, confirming decode is compute and memory-bound, not communication-bound.

>

The cufall metric measuring SM active cycles (sm__cycles_active.avg / gr__cycles_elapsed.max) shows the SMs are "really pegged"—nearly 100% active—which is healthy. Power consumption at 340W is reasonable for that utilization level.

>

For my response, I need to clarify that the thinking-token discrepancy comes from the prefill ramp (TTFT of 14s at C=64), not a measurement error. The steady-state decode throughput of 32.85 tokens/sec is the real ceiling, separate from the prefill overhead. The low PCIe communication confirms decode is bottlenecked by compute and memory latency in the kernels themselves, not by inter-GPU communication.

>

The cufall data—SMs pegged but only 340W power draw—directly confirms that the sparse MLA attention and MoE kernels are memory-latency or SIMT-bound, running on CUDA cores with high stalls and low instruction throughput rather than saturating tensor pipes. Looking at the scaling pattern, the marginal cost per request stabilizes around 30ms across batch sizes, suggesting the model hits an asymptote near 33 tokens/sec at C=64, which matches the observed steady-state. The next step is to profile individual kernels at high batch to identify which one dominates and measure its memory throughput versus compute utilization.

The reasoning continues, working through the implications and planning the next diagnostic step. The assistant then produces a structured response for the user, followed by bash commands to orchestrate a steady-state profile capture.

Prefill Ramp: The Hidden Confound in Throughput Measurement

The first user observation—"why is early output slower vs later"—is a classic pitfall in LLM serving benchmarks. When a benchmark tool like sglang.bench_serving submits a batch of prompts, it does not wait for all prompts to finish prefill before starting to measure throughput. Instead, it reports aggregate throughput over the entire benchmark run, including the ramp-up phase where some requests are still being prefilled while others have started decoding.

The assistant's diagnosis is precise: at C=64 with 96 prompts, the median Time-To-First-Token (TTFT) is 13.9 seconds. This means each request spends nearly 14 seconds in the prefill queue before generating its first output token. During this period, the decode phase is starved—only a few requests have reached the decode stage, so throughput appears low (around 5 tok/s). As more requests complete prefill and enter decode, throughput climbs to the steady-state value of 32.85 tok/s. The aggregate throughput of 29.71 tok/s reported by the benchmark is an average that includes both the ramp-up and steady-state phases, making it an underestimate of the true decode ceiling.

This is not a measurement error in the sense of miscounting tokens. The benchmark correctly counts completion tokens. The "thinking tokens" concern is a red herring—the random dataset used for benchmarking has no chat template or reasoning structure, so there are no hidden thinking tokens to miscount. The issue is purely temporal: the benchmark's aggregation window includes the prefill phase, which dilutes the decode throughput measurement.

The assistant's key contribution here is to separate the signal from the noise. By looking at the #running-req logs—which report instantaneous generation throughput at regular intervals—the assistant can extract the steady-state decode throughput independent of the prefill ramp. This yields a clean scaling table:

| Concurrency | Steady-state gen tok/s | Step time | Marginal cost per request | |-------------|------------------------|-----------|--------------------------| | C=1 | 12.2 | 82 ms | — | | C=16 | ~29 | ~550 ms | ~31 ms/req | | C=64 | 32.85 | ~1950 ms | ~29 ms/req |

From this, the assistant derives a linear model: step_time ≈ 52 + 30·N ms, where N is the number of concurrent requests. The 52 ms baseline is the fixed cost per decode step (the irreducible kernel launch overhead, scheduling, and single-request computation), and the 30 ms per additional request is the marginal cost of processing one more sequence through the attention and MoE kernels. This model predicts an asymptotic throughput of ~33 tok/s (1000 ms / 30 ms per token), which matches the observed C=64 steady-state value exactly.

This is a powerful diagnostic result. The linear scaling with near-constant marginal cost means the bottleneck is not a resource that can be shared (like memory bandwidth, which would show sublinear scaling as more requests share the same bandwidth) but rather a serialized per-request computation. Each additional request adds a fixed amount of GPU kernel time, and these kernels run sequentially rather than in parallel. The question is: which kernel or operation is responsible for this 30 ms per request?

PCIe at 200 MB/s: Ruling Out Communication

The second user observation—PCIe bandwidth at only ~200 MB/s—seems at first like a problem. TP4 (tensor parallelism across 4 GPUs) requires all-reduce operations to synchronize activations and gradients. If the GPUs are spending significant time waiting for these communications, that would explain the throughput ceiling.

But the assistant recognizes that 200 MB/s is actually reassuringly low. PCIe 5.0 x16 has a theoretical bandwidth of approximately 64 GB/s (each direction). At 200 MB/s, the system is using only 0.3% of available PCIe bandwidth. This means the GPUs are not waiting on communication—they are spending almost all their time in compute kernels, with long gaps between all-reduce operations.

This aligns with earlier profiling that showed NCCL (the NVIDIA Collective Communications Library) consuming only 1-2% of GPU time. The TP all-reduce payload is small—hidden dimension 4096 × batch size × 2 bytes (bf16) ≈ a few megabytes per operation—and the all-reduce itself is fast. The GPUs are not communication-bound; they are compute-bound or memory-bound.

This insight is critical because it rules out an entire class of potential optimizations. If the bottleneck were communication, the fix might involve using NVLink (which these GPUs lack, being connected only via PCIe), or MSCCL++ (a Microsoft research library for optimized collective communication), or flashinfer's fused attention that avoids some all-reduces. But since communication is not the problem, none of these would help. The assistant correctly identifies that the bottleneck lies elsewhere.

The cufall Puzzle: Pegged SMs at Low Power

The third observation is the most revealing. The user reports that cufall (a GPU performance monitoring tool, likely part of NVIDIA's CUDA profiling toolkit) shows the SM active cycles metric is "really pegged"—meaning the streaming multiprocessors are active nearly 100% of the time. Yet power draw is only 340W, far below the RTX 6000 Blackwell's ~600W TDP.

This combination—high SM occupancy but low power—is a classic signature of memory-latency-bound execution. When SMs are active but stalled on memory accesses (waiting for data to arrive from DRAM or L2 cache), they consume much less power than when they are actively computing on tensor cores or FMA (floating-point multiply-add) units. The SMs are "pegged" in the sense that they are resident and not idle, but they are spending most of their cycles stalled on memory dependencies rather than executing useful arithmetic.

The assistant's reasoning connects this to the earlier profiling data: the sparse MLA attention and MoE kernels are running on CUDA cores (SIMT path) rather than tensor cores. Tensor core operations (like the MMA kernels the assistant built earlier) are highly power-efficient and throughput-optimized—they can saturate the GPU's compute units and draw near-TDP power. SIMT fallback kernels, by contrast, execute on CUDA cores with lower instruction throughput and are often limited by memory access patterns. The 340W power draw, combined with pegged SMs, indicates that the GPU is spending most of its time in these SIMT fallback kernels, stalled on memory latency.

The assistant quantifies this: "If we were at the 1.9 TB/s roofline, power would be much higher and we'd be near the ~1000s-t/s memory bound. We're ~30× off roofline → latency/occupancy-bound CUDA-core fallback kernels, which is fixable, not a hardware wall."

This is a crucial distinction. A memory-bandwidth-bound kernel would saturate the DRAM interface and draw higher power. A latency-bound kernel, by contrast, is limited by the time it takes to service individual memory requests, not by the aggregate bandwidth. This is typically caused by poor memory access patterns (e.g., random access, uncoalesced loads, pointer chasing) that defeat the GPU's memory hierarchy. The fix is not to add more bandwidth (which is already abundant) but to restructure the kernel to improve locality, coalescing, and occupancy.

The Synthesis: One Story, Three Clues

The assistant's most impressive achievement in this message is the synthesis of the three observations into a single narrative. Each observation independently points to the same conclusion, but together they form an airtight case:

  1. The scaling model (52 + 30·N ms) shows a fixed per-request cost that is too high for a well-optimized kernel. At 30 ms per request, a single decode step for 64 requests takes ~2 seconds. For a model of this size on Blackwell hardware, this is far too slow—it suggests the kernels are not using the hardware efficiently.
  2. The PCIe bandwidth (200 MB/s, 0.3% of link capacity) confirms the GPUs are not waiting on each other. The bottleneck is within each GPU's own computation, not in inter-GPU communication.
  3. The cufall metrics (pegged SMs, 340W power) identify the nature of the intra-GPU bottleneck: memory-latency-bound SIMT execution, not compute-bound tensor-core execution. The assistant's conclusion is that the remaining bottleneck is in the "glue" operations—the elementwise copies, multiplications, and reductions that connect the attention and MoE kernels. These operations are running on CUDA cores with poor memory access patterns, causing high stall rates and low power efficiency. The 30 ms per request is the time spent in these glue kernels, and fixing them could potentially bring throughput closer to the hardware's true capability.

The Profiling Plan: From Diagnosis to Action

Having synthesized the diagnosis, the assistant does not stop at explanation. The message includes a concrete plan for the next diagnostic step: capture a fresh profile at steady-state C=32 to identify which specific kernel or operation dominates the 30 ms per request.

The assistant writes an orchestration script (prof_steady.sh) that:

  1. Launches a benchmark load with 220 prompts at C=32
  2. Waits 38 seconds for the system to reach steady-state decode (all requests past prefill)
  3. Triggers the SGLang profiler via HTTP POST to /start_profile with num_steps=12
  4. Waits 50 seconds for the profile to complete
  5. Kills the load and signals completion The choice of C=32 is strategic: it is high enough to show the bottleneck clearly (the marginal cost should be visible) but low enough to avoid the OOM issues that plagued C=64 in earlier runs. The 38-second wait ensures the profile captures pure decode, uncontaminated by prefill. The 12-step capture provides enough samples for statistical significance without being so long that the trace becomes unwieldy. This plan demonstrates a methodical approach to performance debugging: form a hypothesis from aggregate measurements, design an experiment to isolate the suspected bottleneck, execute the experiment with controlled conditions, and analyze the results to confirm or refute the hypothesis. The assistant is not guessing—it is building a chain of evidence.

Assumptions and Potential Pitfalls

The assistant's analysis rests on several assumptions that deserve scrutiny:

Assumption 1: The benchmark correctly counts output tokens. The assistant asserts that the random dataset has no chat template or reasoning structure, so there are no hidden thinking tokens. This is correct for the specific benchmark configuration used (--dataset-name random --random-input-len 256 --random-output-len 128), but it assumes the server is not adding any special tokens or processing that could affect the count. Given that the assistant has previously fixed the chat template and reasoning parser (see chunk 2 of segment 68), this assumption is well-founded.

Assumption 2: The steady-state decode throughput is the true ceiling. The assistant separates the prefill ramp from the decode steady state, arguing that the latter is the "real" throughput. This is valid for understanding decode performance, but it ignores the fact that in production, prefill and decode interleave. A request's end-to-end latency includes both phases, and optimizing decode alone may not improve user-perceived performance if prefill remains a bottleneck. The assistant acknowledges this implicitly by noting the 14-second TTFT at C=64, but the focus of this message is on decode.

Assumption 3: Low power + pegged SMs = memory-latency bound. This is a well-established diagnostic in GPU performance analysis, but it is not definitive without corroborating evidence. Other causes of low power at high occupancy include: instruction-level parallelism limits (the kernel has few independent instructions per thread), control divergence (threads within a warp take different paths), or simply that the kernel is very short and the GPU cannot ramp up power quickly enough. The assistant's conclusion is plausible and consistent with the broader profile, but the planned steady-state profile will provide the definitive evidence.

Assumption 4: The 30 ms marginal cost is from glue operations. The assistant's reasoning attributes the per-request cost to the elementwise/copy/reduce "glue" that was identified in earlier profiling as consuming ~69% of GPU time. However, that earlier profile was taken before the indexer fix (which reduced glue from 69% to ~4%) and before the custom MMA kernel was deployed. The current bottleneck may have shifted. The assistant's plan to profile at C=32 is designed to answer this question empirically.

Knowledge Created by This Message

This message creates several pieces of valuable knowledge:

  1. A validated scaling model for DeepSeek-V4-Flash decode on Blackwell GPUs: step_time ≈ 52 + 30·N ms. This model can be used to predict throughput at any concurrency level and to estimate the impact of optimizations.
  2. A diagnostic framework for identifying bottleneck type: The combination of throughput scaling, PCIe bandwidth utilization, and SM activity/power metrics provides a multi-dimensional signature that distinguishes communication-bound, compute-bound, and memory-latency-bound execution.
  3. A methodology for separating prefill from decode in throughput benchmarks: By using instantaneous throughput logs (#running-req) rather than aggregate benchmark results, practitioners can extract the true decode ceiling independent of prefill overhead.
  4. A concrete next step: The profiling orchestration script and the plan to capture steady-state kernel breakdown at C=32 provide an actionable path forward.

Broader Implications

The analysis in <msg id=12532> has implications beyond this specific deployment. It illustrates a general principle of performance debugging on modern GPUs: the most informative metrics are often the ratios between measurements, not the absolute values. The assistant does not look at SM occupancy alone (which is "pegged" and might seem good) or power alone (which is "low" and might seem bad). Instead, it looks at the combination of high occupancy and low power, which reveals the memory-latency-bound nature of the execution.

This principle applies broadly. In any performance-critical system, the most valuable insights come from the intersections of different measurement dimensions. A high IPC (instructions per cycle) with low cache hit rate tells a different story than high IPC with high cache hit rate. High network bandwidth utilization with high latency variance tells a different story than high bandwidth with low variance. The assistant's ability to read these multi-dimensional signatures is what makes the diagnosis possible.

The message also demonstrates the importance of domain knowledge in interpreting hardware metrics. Knowing that the RTX 6000 Blackwell has a ~600W TDP, that PCIe 5.0 x16 provides ~64 GB/s, that tensor cores draw more power than CUDA cores—these facts are not in the measurements themselves but in the engineer's mental model of the hardware. The assistant draws on this knowledge to interpret the raw numbers.

Conclusion

Message <msg id=12532> is a masterclass in performance diagnosis. In a single response, the assistant: