The 32 Tokens Per Second Puzzle: Diagnosing Throughput Collapse in Speculative Decode

In the high-stakes world of large language model inference, few things are as disorienting as a performance regression that defies explanation. The assistant in this opencode session found itself in exactly that position: the user had reported a severe throughput collapse — roughly 32 tokens per second at ~6,000 tokens of context — against a known baseline of 138 tokens per second. The assistant had already begun a systematic diagnostic campaign, ruling out obvious suspects one by one. But when it deployed a clever two-point measurement technique and got results that contradicted both the user's report and its own expectations, the investigation took an unexpected turn. The subject message, <msg id=12128>, captures the moment when the assistant confronts this discrepancy, designs a reproduction experiment, and discovers that the real problem is more subtle — and more instructive — than a simple context-scaling curve.

The Context: A Performance Regression Under the Microscope

To understand the significance of <msg id=12128>, we need to trace the investigation that led to it. The conversation centers on deploying the Kimi K2.6 model (a 671B-parameter Mixture-of-Experts architecture with Multi-Head Latent Attention, or MLA) on a machine with 8× RTX PRO 6000 Blackwell GPUs, using SGLang with a custom speculative decoding technique called DDTree (Draft-Tree). The assistant had recently enabled reasoning and tool-call parsers on the live service (see <msg id=12121> and <msg id=12122>), and the user quickly reported a performance problem: at roughly 5,000 tokens of total context, throughput had dropped to about 32 tokens per second — far below the 138 t/s baseline measured at shorter contexts.

The assistant's first reaction in <msg id=12125> was to hypothesize that context length was the culprit. The MLA attention mechanism, which is the backbone of the K2.6 architecture, scales linearly with context length: every decode step must attend over the entire KV cache, which at 6,000 tokens across 61 layers creates a significant computational burden. The assistant also considered whether the newly enabled parsers might be introducing overhead, but quickly ruled that out since parsers operate on output text, not on the decode loop itself.

In <msg id=12126>, the assistant checked the DDTree acceptance metrics from the server logs and found them healthy: approximately 4 drafts accepted per step, yielding about 5 tokens per speculative decode step. This ruled out "acceptance collapse" — the hypothesis that the draft model was failing to predict useful tokens at longer contexts. With acceptance healthy but throughput still low, the assistant calculated that the per-step time must have grown from roughly 32 milliseconds (at short context) to about 155 milliseconds — a 4× slowdown. This pointed squarely at long-context attention as the bottleneck.

To confirm this hypothesis, the assistant designed a two-point measurement method in <msg id=12126>: for a given prompt length, it would send two requests with different max_tokens values (8 and 264), measure the total time for each, and compute pure decode throughput as 256 / (t_264 - t_8). This isolates decode time from prefill time, giving a clean measurement of how fast the model generates tokens once the KV cache is populated.

The Two-Point Results: A Puzzle Emerges

In <msg id=12127>, the assistant deployed this diagnostic script — bench_context_decode.py — against the live service. The results were surprising:

| Context (tokens) | Prompt Tokens | Decode tok/s | |-----------------|---------------|-------------| | 64 | 82 | 255.7 | | 1024 | 1407 | 216.4 | | 3072 | 4257 | 113.3 | | 5120 | 7107 | 66.8 |

The two-point method showed a clear context-scaling curve: decode throughput dropped from 256 tok/s at very short context to 67 tok/s at 5,120 tokens of context. This was a 3.8× slowdown — consistent with the assistant's earlier estimate of a 4× step-time increase. The curve itself was expected for a C=1 (single concurrent request) speculative decoding system where attention dominates the per-step cost.

But here was the puzzle: 67 tok/s at 5k context is roughly double the 32 t/s the user reported. If the two-point method was accurate, the user should have been seeing about 67 tokens per second, not 32. Something was wrong — either the measurement technique was flawed, or the user's scenario involved additional factors that the two-point method didn't capture.

The Subject Message: Confronting the Discrepancy

This is where <msg id=12128> begins. The assistant opens with a detailed reasoning trace that shows it wrestling with the discrepancy:

Surprising — two-point decode is actually healthy and scales as expected for C=1 (attention-bound): 256 → 216 → 113 → 67 tok/s from ctx 64 → 5k. That's ~2× the user's reported 32. So the gap isn't decode-rate-vs-context alone.

The assistant then considers several possible explanations:

  1. Measurement methodology differences: The earlier bench_service.py showed 138 tok/s for 256 tokens in 1.85 seconds, but the two-point method shows 256 tok/s for 192 tokens in 0.75 seconds at similar context. The assistant correctly identifies that bench_service.py averaged decode speed across the entire request including prefill overhead, while the two-point method isolates pure decode. However, this doesn't explain the 67 vs 32 gap since both methods should converge for long generations.
  2. The user's measurement includes prefill: The user's scenario — a 3,239-token prompt generating 2,750 tokens — might include prefill time in the total, dragging down the effective rate. But the assistant quickly calculates that even with a 1.5-second prefill and 90 tok/s generation, the effective rate would be ~87 tok/s — still not 32.
  3. Something degrades during long continuous generation: This is the key insight. The two-point method measures decode throughput over a short burst (~200 tokens), while the user's scenario involves generating 2,750 tokens continuously. If something degrades progressively during a long generation — acceptance rate dropping, KV cache fragmentation, or some other cumulative effect — the two-point method would miss it entirely. The assistant's reasoning is sharp: it recognizes that the two-point method, while elegant for isolating decode from prefill, may not capture the full picture of sustained generation performance. The gap between 67 tok/s (short burst at 5k context) and 32 tok/s (long generation at similar context) suggests that something about the generation process itself is getting progressively slower.

The Reproduction Experiment

The assistant's response is immediate and decisive: it designs a reproduction script that matches the user's exact scenario — a ~3,200 token prompt with max_tokens=1500 at temperature 0 — and measures both the client-observed throughput and the isolated decode rate. The script uses the streaming API to measure time-to-first-token (TTFT) and then computes decode-only throughput by subtracting TTFT from total time.

The results are striking:

prompt_tokens=2701  completion=1500
TTFT (prefill, 1 tok stream): 0.06s
full gen: 45.90s  -> client tok/s (ct/total) = 32.7
decode-only tok/s = 32.7  (excludes prefill)

The TTFT is only 0.06 seconds — prefill is essentially instantaneous, confirming that the bottleneck is entirely in the decode phase. The full generation of 1,500 tokens takes 45.9 seconds, yielding exactly the 32.7 tok/s the user reported. And crucially, the decode-only tok/s is also 32.7 — subtracting the tiny prefill time makes no meaningful difference.

This is a critical finding: the two-point method was wrong about the sustained decode rate. At 5k context, the two-point method predicted 67 tok/s, but the actual sustained generation achieved only 32.7 tok/s — less than half the predicted rate. Something about generating 1,500 tokens continuously causes performance to degrade far below what short-burst measurements would suggest.

What This Reveals About the System

The subject message doesn't contain the final answer — that comes in subsequent messages where the assistant checks the DDTree metrics during the long generation and discovers that the acceptance rate drops significantly on reasoning/analysis text (from ~5 tokens/step to ~2.9 tokens/step), and that KV cache fragmentation under page_size=1 degrades step time over sustained generation. But the subject message is where the critical pivot happens: the assistant realizes that short-burst benchmarks can be misleading for speculative decoding systems.

The two-point method is a standard technique for measuring decode throughput — it's widely used in the LLM inference community because it's simple, fast, and isolates the decode phase from prefill. But it assumes that the decode phase is stationary: that the first 200 tokens of generation have the same characteristics as the next 1,300 tokens. For autoregressive decoding without speculation, this is approximately true — the step time is constant, and throughput is simply batch_size / step_time. But for speculative decoding with a draft model, the system has two additional degrees of freedom: the acceptance rate (how many draft tokens are accepted per step) and the step time (which can vary with KV cache state). Both can change over the course of a long generation.

The assistant's reasoning in <msg id=12128> shows it grappling with this complexity. It considers whether "acceptance collapses partway through long generations, or something about output length itself degrades performance." The subsequent investigation (in <msg id=12129> and beyond) confirms both factors: the acceptance rate drops on reasoning text, and KV cache fragmentation increases step time.

Assumptions, Correct and Incorrect

The assistant made several assumptions in this message, some validated and some not:

Correct assumptions:

Input and Output Knowledge

To understand <msg id=12128>, the reader needs several pieces of input knowledge:

  1. The system architecture: Kimi K2.6 is a 671B MoE model with MLA attention, deployed on 8× RTX PRO 6000 GPUs using SGLang with DDTree speculative decoding.
  2. The two-point measurement method: A technique for isolating decode throughput by measuring the time difference between two requests with different max_tokens values.
  3. The context-scaling behavior of attention: MLA attention cost scales linearly with context length, so decode throughput naturally decreases as the KV cache grows.
  4. The user's reported scenario: A prompt of ~3,239 tokens generating ~2,750 tokens at 32.2 t/s. The message creates several pieces of output knowledge:
  5. The two-point method overestimates sustained throughput for speculative decoding: Short-burst measurements miss degradation that occurs during long generations.
  6. The reproduction methodology: A Python script that uses the streaming API to measure TTFT and compute decode-only throughput, which becomes a reusable diagnostic tool.
  7. The confirmed baseline: At 5k context with a 1,500-token generation, the system achieves 32.7 tok/s — matching the user's report and establishing a reference point for future optimization.
  8. The insight that something degrades during long generations: This becomes the launching point for the deeper investigation into acceptance rate and KV cache fragmentation.

The Thinking Process

The assistant's reasoning in <msg id=12128> is particularly interesting because it shows a double-take: the assistant initially interprets the two-point results as "healthy and scales as expected," then realizes they don't match the user's report, and pivots to a reproduction experiment. The reasoning trace reveals several cognitive steps:

  1. Surprise and recognition: "Surprising — two-point decode is actually healthy" — the assistant is surprised because it expected to find a smoking gun (e.g., acceptance collapse) and instead found a clean context-scaling curve.
  2. Discrepancy analysis: "That's ~2× the user's reported 32. So the gap isn't decode-rate-vs-context alone." — The assistant immediately recognizes that the two-point results don't explain the user's observation.
  3. Hypothesis generation: The assistant considers whether the user's measurement includes prefill, whether there's a measurement methodology issue, or whether something degrades during long generations.
  4. Experimental design: The assistant designs a reproduction that matches the user's exact scenario — same prompt length, same generation length, same temperature — and measures both client-observed throughput and decode-only throughput.
  5. Execution and confirmation: The reproduction confirms the user's report (32.7 tok/s) and reveals that the decode-only throughput matches the client-observed throughput (prefill is negligible). The most impressive aspect of the reasoning is the assistant's intellectual honesty. It doesn't try to explain away the discrepancy or blame the user's measurement. Instead, it treats the discrepancy as a signal that its own model of the system is incomplete, and designs an experiment to find the missing piece.

Broader Implications

The subject message has implications beyond this specific debugging session. It highlights a fundamental challenge in benchmarking speculative decoding systems: short-burst measurements can be misleading. The LLM inference community often uses benchmarks that generate 200-500 tokens per request, assuming that performance is stationary. But speculative decoding systems have dynamics — acceptance rates that depend on text difficulty, KV cache fragmentation that accumulates over long generations, and draft model behavior that changes with context length — that only manifest in sustained generation.

This is particularly relevant for production deployments where users generate thousands of tokens per request (e.g., code generation, document analysis, long-form reasoning). A system that benchmarks well at 200 tokens may collapse to half the expected throughput at 2,000 tokens, as this investigation discovered.

The message also demonstrates the value of reproducing the user's exact scenario rather than relying on synthetic benchmarks. The two-point method is a perfectly valid measurement technique, but it measures a different thing than what the user experiences. By matching the user's prompt length, generation length, and temperature, the assistant was able to confirm the report and then investigate the root cause.

Conclusion

The subject message, <msg id=12128>, captures a pivotal moment in a complex performance investigation. The assistant had deployed a sophisticated two-point measurement technique that revealed clean context-scaling behavior, but the results contradicted the user's reported throughput. Rather than dismissing the discrepancy, the assistant designed a reproduction experiment that confirmed the user's observation and revealed that short-burst benchmarks can overestimate sustained decode throughput by 2× or more for speculative decoding systems.

This message is a masterclass in diagnostic methodology: question your measurements, reproduce the exact scenario, and treat discrepancies as signals that your model of the system is incomplete. The assistant's willingness to confront the puzzle — "67 tok/s at ctx 5120... but the user sees 32" — and its systematic approach to resolving it, turned a confusing contradiction into a deeper understanding of the system's behavior. The investigation didn't end here, but this message is where the critical insight emerged: something degrades during long generations, and finding that something would require looking beyond the context-scaling curve.