The 32 Tokens-Per-Second Question: When a Performance Regression Reveals System Truth

"Seems just 30t/s after ~5k total context, seems very off - 3239/2750 tok 32.2 t/s"

This single message from the user, sent as message [msg 12124] in a long-running opencode session, arrives at a pivotal moment. The assistant had just completed a configuration change — enabling --tool-call-parser kimi_k2 and --reasoning-parser kimi_k2 on a live SGLang DDTree service hosting the Kimi K2.6 model across 8 RTX PRO 6000 Blackwell GPUs. The restart had taken roughly ten minutes to reload the 548GB model. Both parsers were verified working: reasoning content was now properly separated into reasoning_content fields, and tool calls were being parsed into structured JSON. Everything looked good. Then the user ran a real generation and saw the number: 32.2 tokens per second.

The message is deceptively brief — just three data points and a judgment. The user provides a prompt token count (3239), a completion token count (2750), and an observed throughput (32.2 t/s). The phrase "seems very off" carries the weight of the message. It is not a question; it is an observation tinged with concern. The user is implicitly comparing against a baseline — the ~138 t/s that had been measured earlier on this same hardware at shorter context lengths. Something has changed, and the timing is suspicious: the parser flags were just added, and now throughput has collapsed by roughly 4×. The natural inference is that the parser change caused the regression.

The Context That Shapes the Message

To understand why this message carries such weight, one must understand what preceded it. The session had been building toward production deployment of a speculative decoding system called DDTree (Draft-Tree) on the Kimi K2.6 model. The assistant had spent dozens of messages building a native C/C++/CUDA inference engine, diagnosing earlier throughput issues, and tuning the SGLang service. The baseline expectation — 138 t/s — came from measurements taken before the parser change, at short context lengths where the speculative decoding stack performed efficiently.

The parser change itself was not trivial. It required editing the systemd unit file, performing a daemon-reload, and restarting the service — which triggered another cold load of the massive model across 8 GPUs. The assistant had carefully verified that the correct parser variant (kimi_k2, not the older kimi which uses different thinking markers) was selected by examining the model's tokenizer configuration and the SGLang parser source code. Everything was deliberate and correct. But from the user's perspective, the sequence was: parser change → restart → slow generation. The correlation was impossible to ignore.

What the Numbers Actually Say

The user's data — 3239 prompt tokens, 2750 completion tokens, 32.2 t/s — tells a specific story when examined closely. The total context at the end of generation would be approximately 3239 + 2750 = ~5989 tokens, which matches the "~5k total context" the user mentions. The throughput of 32.2 t/s is an end-to-end measurement: total completion tokens divided by total wall-clock time. This includes everything — prefill, speculative decoding steps, and any overhead.

What makes this number striking is the comparison. At short context (~64 tokens), the same stack had been measured at 255 t/s using a two-point decode isolation method. At 1024 tokens context, it was 216 t/s. At 3072 tokens, 113 t/s. The user's observed 32 t/s at ~6k context represents a further drop that seems disproportionate — a 4× reduction from the 138 t/s baseline that doesn't follow the smooth curve of the earlier measurements. This is what triggers the "seems very off" judgment.

The Assumption That Drives the Investigation

The implicit assumption in the user's message — and it is a reasonable one — is that the parser change caused or contributed to the slowdown. In production ML systems, configuration changes are the most common source of regressions, and the temporal correlation here is strong. The user is not accusing; they are reporting a discrepancy and signaling that something needs investigation.

This assumption turns out to be wrong, but proving it wrong requires rigorous work. The assistant cannot simply say "the parsers don't affect speed" — it must measure, isolate, and demonstrate. The diagnostic journey that follows this message ([msg 12125] through [msg 12136]) is a masterclass in performance debugging: building custom measurement tools, running controlled experiments, separating confounding variables, and ultimately reconciling all the data into a coherent model of system behavior.

The Input Knowledge Required

To fully grasp this message, one needs to understand several layers of context. First, the speculative decoding architecture: DDTree is a draft-tree method where a smaller "drafter" model proposes multiple candidate tokens in a tree structure, and the target model verifies them in parallel. The throughput depends on two factors: how many tokens are accepted per verify step (acceptance rate) and how long each verify step takes (step time). Second, the hardware: 8× RTX PRO 6000 Blackwell GPUs with NVidia's latest architecture, connected via PCIe. Third, the model architecture: Kimi K2.6 uses Multi-Head Latent Attention (MLA) and Mixture-of-Experts (MoE), which have different scaling properties than dense transformers. Fourth, the software stack: SGLang with the Triton attention backend, which handles the MLA attention but may not be as optimized for long-context scenarios as alternative backends like FlashInfer.

The Output Knowledge Created

This message catalyzes one of the most thorough diagnostic investigations in the session. The assistant builds a context-sweep benchmark (bench_context_decode.py), runs controlled two-point decode measurements across context lengths from 64 to 5120 tokens, isolates the effect of output length versus context length, and captures per-step acceptance metrics from the service logs. The investigation produces a clear, quantitative model of the system's behavior.

The key finding is that the 32 t/s is not a regression at all — it is the mathematically expected performance given two compounding effects. First, the per-step verify forward pass is attention-bound and its latency grows with context length: from 34 milliseconds at 13 tokens of context to 144 milliseconds at 5.5k tokens. This is inherent to the C=1 (single-sequence) decode path where each verify step must attend over the entire growing KV cache through 61 layers of MLA. Second, the drafter model — which the team had previously flagged as undertrained — achieves a dramatically lower acceptance rate on the user's analysis and reasoning text (approximately 2.9 tokens committed per step) compared to predictable or repetitive text (7–8 tokens per step). The product of these two effects — 2.9 tokens per step divided by approximately 100 milliseconds step time at the user's context length — yields exactly 29–32 tokens per second.

The Deeper Truth Revealed

What makes this message significant is not the number itself but what it reveals about the nature of performance debugging in complex ML systems. The user's instinct — blame the most recent change — is natural but misleading. The parsers are output post-processing; they run on the generated text after decoding completes and cannot affect token generation speed. The real causes are structural: the attention mechanism's inherent context scaling and the drafter's limited capability on hard text. These are not bugs; they are properties of the current system design.

The message also reveals a tension between two modes of evaluation. The user is comparing against a remembered baseline (138 t/s), but that baseline was measured under different conditions — shorter context, different prompt characteristics, possibly different measurement methodology. The assistant's controlled experiments show that the system is behaving exactly as its components dictate: the step time grows predictably with context, and the acceptance rate depends on text predictability. There is no hidden bug, no parser-induced overhead, no configuration error. The system is simply showing its true colors under a realistic workload.

The Broader Implications

This message and the investigation it triggers illustrate a fundamental challenge in speculative decoding: the gap between benchmark performance and real-world performance. Benchmarks typically use short, predictable prompts and measure burst throughput. Real workloads involve long context windows, reasoning and analysis text, and sustained generation. The drafter that achieves 7–8 token acceptance on repetitive padding text may drop to 2–3 on the kind of open-ended reasoning that makes large language models valuable. The attention backend that delivers 255 t/s at 64 tokens context may struggle at 6000 tokens.

The user's message, in its brevity, captures this entire problem. "Seems very off" is the voice of someone who has seen the system perform better and is trying to understand why it isn't now. The answer — that the system is performing exactly as designed, and the design has inherent limitations that only appear under realistic conditions — is both unsatisfying and deeply informative. It points toward the real levers for improvement: a better-trained drafter, a more efficient attention backend for long contexts, and perhaps a larger draft window to give the drafter more context to work with. These are not quick fixes. They are engineering investments.

In the end, the message [msg 12124] serves as a reality check. It forces the team to look past the convenient benchmark numbers and confront the actual performance characteristics of their system under production-like conditions. The 32.2 t/s is not a bug report. It is a system diagnosis, delivered in three numbers and a phrase of concern.