The 1.4 Tok/s Cliff: Diagnosing Long-Context Decode Collapse on Blackwell GPUs

Introduction

In the high-stakes world of large language model deployment, few metrics matter more than decode throughput. When a user requests 200,000 tokens of context, they expect the model to generate completions at a usable speed—not to wait minutes for each token. In a pivotal message from an opencode coding session, an AI assistant deployed a Kimi K2.6 model with DFlash speculative decoding on an 8-GPU RTX PRO 6000 Blackwell system and benchmarked it at progressively longer context lengths. The results revealed a devastating performance cliff: decode throughput collapsed from 118.6 tokens per second at 1,400 tokens of context to just 1.4 tokens per second at 127,800 tokens—a staggering 85× slowdown that would make long-context inference practically unusable.

This message ([msg 12171]) is a critical inflection point in the session. It does not yet diagnose the root cause, but it produces the hard data that drives the entire subsequent investigation: the custom sm_120 verify attention kernel, CUDA graph optimization, KV defragmentation, and the eventual 3–6× speedup over the Triton baseline. Understanding this message means understanding how a systematic benchmark exposed a hidden bottleneck that no amount of reasoning could have predicted.

Context: The 200k Deployment

The message sits within a larger arc spanning Segment 66 of the conversation. The assistant had been working for days to deploy Kimi K2.6 with DFlash speculative decoding on a cluster of RTX PRO 6000 Blackwell GPUs (sm_120 architecture). The user's explicit requirement was 200,000 tokens of context length—a demanding target that required careful memory management.

In the messages immediately preceding [msg 12171], the assistant had navigated a series of crises. The initial deployment with mem-fraction-static 0.94 left only ~4.2 GB of GPU memory headroom, causing an out-of-memory crash during a 32k-token prefill ([msg 12165]). The assistant responded by reducing the memory fraction to 0.93, enabling chunked-prefill-size 4096 to shrink activation spikes, and adding expandable_segments:True to combat fragmentation ([msg 12167]). The server came back with a max_total_num_tokens of 195,011—97.5% of the 200k target—and a healthier ~5.5 GB reserve ([msg 12170]).

With the server stable, the assistant deployed a streaming benchmark script that used a single prefill per measurement point (avoiding the dual-prefill approach that had caused the earlier OOM). The first chunk of benchmarks (1.4k, 11k, and 32k actual context) showed a clear but not yet alarming trend: decode throughput dropped from 118.6 tok/s at 1.4k to 6.3 tok/s at 32k. The assistant's reasoning in [msg 12171] reveals that it expected this linear scaling to continue, projecting "around 1-2 tok/s" for longer contexts. That projection proved accurate—but the implications were far more serious than the assistant initially recognized.

The Message Itself: Running Chunk B

The subject message is deceptively straightforward. It contains three elements: an Agent Reasoning block where the assistant reflects on the chunk A results and plans chunk B, a table summarizing the chunk A data, and a bash command that executes chunk B (46k and 92k requested context, mapping to approximately 64k and 128k actual tokens).

Let me quote the core of the message exactly:

Streaming is clean. Clear steep decode falloff with context (optimistic synthetic text, but the scaling is the signal):

>

| actual ctx | TTFT | prefill tok/s | decode tok/s | |---|---|---|---| | 1.4k | 0.46s | 3104 | 118.6 | | 11k | 4.29s | 2659 | 19.9 | | 32k | 17.55s | 1820 | 6.3 |

>

Now the long points (actual ≈ 64k, 128k):

>

`` ctx_req prompt_tok TTFT_s prefill_tok/s decode_tok/s gen_tok decode_wall 46000 63903 50.94 1255 2.8 24 8.31 92000 127802 183.72 696 1.4 24 16.48 ``

>

Server is alive, stable.

The table tells a stark story. At 1.4k context, the model generates at 118.6 tok/s—usable, even fast. At 11k, it drops to 19.9 tok/s—a 6× slowdown. At 32k, it's 6.3 tok/s—another 3×. At 64k, 2.8 tok/s—another 2.2×. At 128k, 1.4 tok/s—another 2×. The pattern is roughly a halving of decode throughput each time context doubles, which is consistent with a linear-time attention mechanism dominating the step time.

But the magnitude is the problem. At 128k context, generating a single 100-token response would take over 70 seconds. A 500-token response would take nearly 6 minutes. This is not a production-ready system for long-context workloads.

The Reasoning Process: What the Assistant Was Thinking

The Agent Reasoning block reveals the assistant's mental model at this point:

Looking at the streaming performance metrics across different token counts, I can see the decode speed drops significantly as the output gets longer—from 118.6 tok/s with a short response down to just 6.3 tok/s with a much longer one, which suggests memory or computational constraints are kicking in at scale.

>

The pattern is revealing: decode time per output token scales roughly linearly with context size, which makes sense since attention over the full KV cache dominates the step time. Extrapolating this trend, I'd expect performance to degrade further at even larger contexts like 128k or 185k tokens, dropping to around 1-2 tok/s.

>

Now I need to actually measure those longer sequences. I'll run chunk B with the larger contexts (around 64k and 128k tokens) to see how the prefill and decode times play out in practice.

Several things stand out in this reasoning. First, the assistant correctly identifies that attention over the full KV cache is the dominant cost at decode time—this is textbook transformer behavior. Second, it extrapolates the trend and predicts 1-2 tok/s at 128k, which matches the measured 1.4 tok/s almost exactly. Third, the assistant frames this as a natural consequence of linear attention scaling, not yet recognizing that something is pathologically wrong.

The key insight that the assistant does not yet have is that 1.4 tok/s is far below what the hardware should be capable of. An RTX PRO 6000 Blackwell GPU has 1.8 TB/s of HBM3 memory bandwidth. Even with a full 128k KV cache, a well-optimized attention kernel should achieve far higher throughput. The 1.4 tok/s figure implies an effective bandwidth utilization of roughly 14 GB/s—less than 1% of the theoretical peak. Something fundamental is broken in the attention implementation, not just "linear scaling."

Decisions Made in This Message

The message makes several concrete decisions:

  1. Run chunk B with 46k and 92k requested context: The assistant chose these values based on the ~1.39× inflation factor observed in earlier runs (requested 32k → actual 45,507 tokens). The mapping worked: 46k → 63,903 actual, 92k → 127,802 actual, both well under the 195k pool limit.
  2. Use 24 generation tokens: The --gen 24 flag limits each benchmark to 24 decode steps. This is a pragmatic choice—longer generations would increase the risk of OOM or timeout, and 24 tokens is enough to get a stable throughput measurement. The decode wall times confirm this was reasonable: 8.31 seconds at 64k, 16.48 seconds at 128k.
  3. Set a 500-second timeout: Based on the chunk A results (32k took 17.55s prefill + 3.64s decode), the assistant estimated that 128k would take roughly 4× longer to prefill and 4× longer per decode step, totaling ~200-300 seconds. The 500-second timeout provided a comfortable margin.
  4. Verify server health after the benchmark: The assistant explicitly checks systemctl is-active and the /health endpoint after chunk B. This is critical discipline—the earlier OOM crash had demonstrated that long-context prefills can kill the server, and the assistant needed to confirm stability before proceeding.

Assumptions and Their Validity

The message rests on several assumptions, some explicit and some implicit:

The ~1.39× inflation factor holds for longer contexts. This was an empirical observation from earlier runs, and it proved accurate. The 46k request produced 63,903 tokens (1.389×) and the 92k request produced 127,802 tokens (1.389×). This consistency suggests the prompt generator's tokenization is stable across lengths.

Decode time scales linearly with context. This assumption is stated explicitly in the reasoning: "decode time per output token scales roughly linearly with context size." The data supports this—the decode tok/s roughly halves each time context doubles—but the assumption obscures the deeper question of why the scaling is so steep. A well-optimized system should show sub-linear scaling due to KV cache locality and efficient attention kernels.

The server will survive the benchmark. After the OOM crash at 0.94 mem-fraction, the assistant was rightly cautious. The 0.93 configuration with chunked prefill and expandable segments proved stable, and the health check confirmed the server was "active" and responding to /health with "OK."

The streaming benchmark methodology is robust. The assistant had rewritten the benchmark to use streaming POST requests with SSE parsing, measuring TTFT from the first token arrival and decode rate from the remaining tokens. This approach avoids the radix-cache contamination that plagued the earlier dual-prefill method and gives cleaner measurements.

The Mistake That Wasn't Yet a Mistake

The most important aspect of this message is what the assistant doesn't realize. The 1.4 tok/s figure is treated as an expected consequence of linear attention scaling. In reality, it's a symptom of a catastrophic bottleneck: the DDTree verify attention kernel was locked to Triton MLA with page_size=1, causing scattered KV cache accesses that achieved only ~14 GB/s of effective bandwidth—130× below the GPU's 1.8 TB/s peak.

This is not a mistake in the message itself—the assistant didn't have the data to diagnose the root cause yet. The message is doing exactly what a good benchmark should do: producing hard numbers that reveal a problem. The mistake would be to accept 1.4 tok/s as inevitable, and the assistant does not do that. In the subsequent messages, the user challenges the low utilization, and the assistant investigates thoroughly, eventually identifying the Triton MLA verify kernel as the bottleneck and building a custom sm_120 kernel that achieves 3–6× speedup.

But in this moment, the assistant's reasoning reflects a dangerous assumption: that poor performance at long context is normal. It's a natural cognitive trap for engineers—when you've been fighting deployment issues for days, a system that "works" (doesn't crash, produces correct tokens) can feel like a success, even if it's 100× slower than it should be.

Input Knowledge Required

To fully understand this message, a reader needs:

  1. The hardware context: 8× RTX PRO 6000 Blackwell GPUs (sm_120 architecture) with ~95 GB HBM3 each, connected via PCIe (not NVLink). The assistant is using TP8 (tensor parallelism across 8 GPUs), meaning each rank handles 1/8 of the attention heads.
  2. The model context: Kimi K2.6, a Mixture-of-Experts (MoE) model with Multi-Head Latent Attention (MLA). The DFlash speculative decoding drafter uses a smaller model to propose tokens, which the target model verifies in parallel.
  3. The deployment context: SGLang serving framework with custom DDTree speculative decoding integration. The server was configured with mem-fraction-static 0.93, chunked-prefill-size 4096, and expandable_segments:True.
  4. The benchmark methodology: Streaming requests with SSE parsing, single prefill per measurement point, 24 generation tokens, measuring TTFT (time to first token) and decode throughput from token arrival timestamps.
  5. The earlier OOM incident: The server crashed at 0.94 mem-fraction during a 32k prefill, leading to the configuration changes that made this benchmark possible.

Output Knowledge Created

This message produces several pieces of critical knowledge:

Concrete performance numbers for long-context inference on Blackwell GPUs with DFlash DDTree:

| Context (tokens) | Prefill Throughput | Decode Throughput | |---|---|---| | 1,427 | 3,104 tok/s | 118.6 tok/s | | 11,402 | 2,659 tok/s | 19.9 tok/s | | 31,952 | 1,820 tok/s | 6.3 tok/s | | 63,903 | 1,255 tok/s | 2.8 tok/s | | 127,802 | 696 tok/s | 1.4 tok/s |

The scaling pattern: Decode throughput approximately halves each time context doubles. This is consistent with an O(n) attention mechanism where each step must process the entire KV cache.

Prefill scaling: Prefill throughput degrades more gracefully, dropping from 3,104 tok/s at 1.4k to 696 tok/s at 128k—a 4.5× slowdown over a 90× context increase. This suggests prefill is compute-bound (matrix multiplications scale well with tensor cores) while decode is memory-bound (attention over KV cache is bandwidth-limited).

Server stability confirmation: The server survives all five benchmark points without OOM, demonstrating that the 0.93 mem-fraction configuration is stable for production use at up to ~128k context.

The seed of the investigation: The 1.4 tok/s figure is so low that it demands explanation. In the subsequent conversation, this data point drives the investigation into GPU utilization, the discovery that tensor core utilization is at ~3% during decode despite 99.8% SM occupancy, and ultimately the realization that the Triton MLA verify kernel with page_size=1 is causing scattered KV cache accesses at 14 GB/s effective bandwidth.

Significance: The Turning Point

Message [msg 12171] is the moment when the assistant moves from "deployment mode" to "optimization mode." The 200k context deployment is technically complete—the server runs, handles requests, and doesn't crash. But the performance numbers are so poor that the deployment is not practically useful. The user's next message will question the low GPU utilization, and the assistant will begin the deep investigation that leads to:

  1. Identifying the verify attention kernel as the bottleneck (Triton MLA with page_size=1)
  2. Discovering that all optimized MLA kernels (FlashMLA, cutlass-MLA, flashinfer-MLA) lack sm_120 support
  3. Building a custom sm_120 verify attention kernel from scratch
  4. Making the kernel CUDA graph capture-safe
  5. Implementing KV defragmentation
  6. Achieving a 3–6× decode speedup over the Triton baseline None of this work would have happened without the hard data produced in this message. The 1.4 tok/s number is a flashing red light that cannot be ignored. It transforms the conversation from "make it work" to "make it fast."

Conclusion

Message [msg 12171] is a masterclass in systematic benchmarking. The assistant takes a newly deployed, barely-stable server and runs it through a rigorous context-length sweep, producing clean, reproducible performance numbers. The results are devastating—1.4 tok/s at 128k context—but the assistant treats them as data, not as failure. It verifies server health, documents the scaling pattern, and prepares for the next step.

The message also reveals a common cognitive trap in engineering: the tendency to normalize poor performance when you've been fighting fires. The assistant's reasoning frames the 1.4 tok/s result as an expected consequence of linear attention scaling, when in reality it's a symptom of a fundamental implementation flaw. This is not a criticism—it's a natural response to a complex system where every component is new and fragile. The true value of the message is that it produces data so stark that it forces a deeper investigation, ultimately leading to a 3–6× improvement and a custom kernel that becomes the foundation of the production system.

In the end, the 1.4 tok/s cliff is not a failure—it's a discovery. And like all good discoveries, it opens the door to something better.