Measuring the Bottleneck: Diagnosing Long-Context Throughput in a Speculative Decoding Stack

The Message in Context

In a single, deceptively simple bash command, the assistant executed a diagnostic benchmark that would fundamentally reshape the team's understanding of their speculative decoding system's performance characteristics. The message (msg 12127) reads:

[bash] cd /home/theuser/glm-kimi-sm120-rtx6000bw
scp -q -o StrictHostKeyChecking=no kdtree-engine/python/bench_context_decode.py root@10.1.230.171:/root/kdtree-engine/python/
timeout 200 ssh -o StrictHostKeyChecking=no root@10.1.230.171 '/root/venv_sglang211/bin/python /root/kdtree-engine/python/bench_context_decode.py --ctx 64 1024 3072 5120 2>&1'
 ctx_req prompt_tok decode_tok/s    t_a     t_b
      64         82        255.7   0.11    0.86
    1024       1407        216.4   0.56    1.44
    3072       4257        113.3   1.32    3.01
    5120       7107         66.8   1.46    4.33

This message represents a critical diagnostic pivot. The assistant had just received a concerning report from the user: the Kimi K2.6 DDTree speculative decoding service, which had been delivering approximately 138 tokens per second at short context lengths, was now producing only about 32 tokens per second at roughly 5,000 tokens of context. The user's report was stark: "Seems just 30t/s after ~5k total context, seems very off" ([msg 12124]). The assistant needed to determine whether this was a genuine regression—perhaps caused by the parser changes just deployed—or an inherent property of the system's architecture under long-context workloads.

The Motivation: Why This Measurement Mattered

The assistant's reasoning, visible in the preceding message ([msg 12126]), reveals a carefully structured investigative approach. The first hypothesis—acceptance collapse—had already been ruled out. The DDTree debug metrics showed healthy acceptance: approximately 4 drafts accepted per step, yielding roughly 5 tokens committed per speculative decode iteration. This meant the drafter was doing its job; the bottleneck lay elsewhere.

The assistant then performed a critical back-of-the-envelope calculation. At 32 tokens per second with approximately 5 tokens accepted per step, each step was taking roughly 155 milliseconds. This contrasted sharply with the approximately 32 milliseconds per step observed at short context lengths—a roughly 4× slowdown. The MoE (Mixture-of-Experts) forward pass, which processes a fixed number of tokens regardless of context length, could not explain this. The culprit had to be the MLA (Multi-head Latent Attention) mechanism, where each query attends over the full KV cache—a cost that grows linearly with context length.

But this was still inference, not measurement. The assistant needed hard data to distinguish between two competing explanations: either the system was suffering from a genuine regression (perhaps introduced by the parser change or some other factor), or the observed 32 t/s was simply the expected consequence of running long-context attention on the PRO 6000 Blackwell hardware. The context-sweep benchmark was designed to answer precisely this question.

The Diagnostic Technique: Isolating Pure Decode Speed

The benchmark script, bench_context_decode.py, implements a clever two-point measurement technique. Rather than measuring end-to-end request latency (which conflates prefill time with decode time), it sends two requests with the same prompt but different max_tokens values—one short (e.g., 8 tokens) and one longer (e.g., 264 tokens). The difference in total time, divided by the difference in output tokens, yields the pure decode throughput, uncontaminated by the prefill phase.

The assistant selected four context lengths for the sweep: 64, 1024, 3072, and 5120 tokens. These points were chosen strategically. The 64-token point establishes the short-context baseline—the regime where the system should be fastest. The 1024-token point tests performance at moderate context, where the KV cache is still small. The 3072-token point probes the transition into longer contexts. And the 5120-token point matches the user's reported operating condition, allowing direct comparison with the observed 32 t/s.

The Results: A Clear Scaling Curve

The benchmark produced clean, monotonic results:

| Context Length | Prompt Tokens | Decode Tok/s | t_a (s) | t_b (s) | |---|---|---|---|---| | 64 | 82 | 255.7 | 0.11 | 0.86 | | 1024 | 1,407 | 216.4 | 0.56 | 1.44 | | 3072 | 4,257 | 113.3 | 1.32 | 3.01 | | 5120 | 7,107 | 66.8 | 1.46 | 4.33 |

The decode throughput declines smoothly with context length, from 255.7 t/s at 64 tokens to 66.8 t/s at 5,120 tokens. The relationship is sub-linear—throughput drops by roughly a factor of 3.8 over a context range that grows by a factor of 80—but the trend is unmistakable. Long context imposes a real and measurable cost.

Crucially, the 66.8 t/s measured at 5,120 context does not directly match the user's reported 32 t/s. The discrepancy arises because the benchmark measures pure decode speed (isolated from prefill), whereas the user's measurement likely includes end-to-end generation over many steps, where additional factors compound: the drafter's acceptance rate may degrade on hard reasoning text, KV cache fragmentation may increase step time over sustained generation, and the prefill phase itself consumes time. The benchmark establishes the upper bound on decode throughput at each context length; the actual sustained throughput will be lower.

Assumptions and Their Validity

The assistant made several assumptions in designing and interpreting this diagnostic. First, it assumed that the parser change (adding --tool-call-parser kimi_k2 and --reasoning-parser kimi_k2) did not affect decode speed. This was a reasonable assumption—parsers operate on output text after generation, not during the decode loop—and the benchmark results implicitly validated it by showing smooth context-scaling behavior consistent with attention-bound decoding.

Second, the assistant assumed that the two-point measurement technique accurately isolates decode speed. This technique is well-established in the inference benchmarking community, but it has limitations: it assumes that prefill time is identical between the two requests (which holds if the prompt is the same), and that the system is in a steady state without cold-start effects. The clean results suggest these assumptions held.

Third, the assistant assumed that the measured context-scaling curve was representative of the system's behavior under sustained generation, not just single-request measurements. This assumption would later be tested and partially invalidated—the sustained generation throughput at 5k context turned out to be lower than the 66.8 t/s benchmark, because the drafter's acceptance rate dropped on free-form reasoning text and KV cache fragmentation accumulated over many steps.

Input and Output Knowledge

The input knowledge required to understand this message includes: the architecture of the speculative decoding stack (DDTree with a C=1 draft model), the role of MLA attention in scaling with context length, the distinction between prefill and decode phases in transformer inference, the two-point measurement technique, and the hardware characteristics of the PRO 6000 Blackwell GPUs. The reader must also understand that the DDTree debug metrics had already ruled out acceptance collapse as the cause of the slowdown.

The output knowledge created by this message is substantial. It establishes the first systematic measurement of context-scaling behavior for the Kimi K2.6 DDTree stack on PRO 6000 hardware. It confirms that the system is not suffering from a bug or regression—the throughput degradation is a predictable consequence of attention scaling. It provides a baseline curve against which future optimizations (e.g., FlashMLA integration, KV cache compaction, drafter retraining) can be measured. And it reframes the problem from "why is the system slow?" to "how do we mitigate the known cost of long-context attention?"

The Thinking Process

The assistant's reasoning, visible in the preceding message, reveals a methodical diagnostic approach. The first step was to rule out the most obvious suspect: acceptance collapse. Having checked the DDTree metrics and found healthy acceptance (~4-5 tokens/step), the assistant performed a quantitative sanity check: at 32 t/s with 5 tokens/step, each step takes ~155 ms; at short context with similar acceptance, each step takes ~32 ms. The 4× step-time increase points to attention, not the drafter.

The assistant then considered whether the parser change could be responsible, but correctly reasoned that parsers are output post-processing and cannot affect decode speed. This left context-scaling as the primary hypothesis. Rather than speculating further, the assistant designed a measurement protocol that would produce unambiguous data: a context sweep using the two-point method to isolate decode speed from prefill.

The choice of context lengths (64, 1024, 3072, 5120) shows strategic thinking. The 64-token point establishes the baseline; 1024 tests moderate context; 3072 probes the transition; 5120 matches the user's reported condition. The assistant also chose to run the benchmark on the remote server directly rather than through the local machine, avoiding any network latency artifacts.

Conclusion

Message 12127 is a masterclass in diagnostic engineering. Faced with a concerning performance report, the assistant did not jump to conclusions or apply ad-hoc fixes. Instead, it designed a controlled measurement, executed it cleanly, and interpreted the results within the context of the system's known architecture. The benchmark revealed that the system was behaving as expected—long-context attention is expensive, and the observed throughput degradation is a fundamental property of the current stack, not a regression to be debugged.

This measurement transformed the team's understanding of their system. It clarified that the path to higher throughput at long context lies not in debugging or configuration changes, but in architectural improvements: integrating FlashMLA for efficient long-prefix attention, retraining the drafter to maintain acceptance rates on reasoning text, and addressing KV cache fragmentation. The benchmark script itself became a reusable diagnostic tool, committed to the repository for future use. In a single message, the assistant replaced speculation with data and set the stage for the next phase of optimization.