The Mathematics of Speculative Decoding Viability: When 150 tok/s Meets 30ms Reality

In the high-stakes world of large language model inference, every millisecond counts. When you're running a 1-trillion-parameter Mixture-of-Experts model like Kimi-K2.5 across eight PCIe-connected GPUs, the difference between a system that accelerates throughput and one that actively hurts it can come down to a single number: the verify cycle time. This is the story of a single message — message 4921 in a long opencode coding session — where an AI assistant performs a rapid-fire mathematical analysis to answer a deceptively simple question: "What accuracy do we need for 150/200tps?"

The Context: A Speculative Decoding System in Crisis

To understand why this message matters, we need to understand what led to it. The session had been building toward deploying EAGLE-3 speculative decoding — a technique where a small "draft" model proposes multiple tokens in parallel, and the large target model verifies them in a single forward pass. The promise is dramatic throughput improvements: if the draft model is accurate enough, the system can generate multiple tokens per target model invocation, effectively multiplying the generation rate.

But the reality was harsh. After days of debugging, tuning NCCL environment variables, patching SGLang's engine code, experimenting with attention modes, and profiling every millisecond of the speculative pipeline, the assistant had discovered a brutal truth: the EAGLE-3 speculation setup was delivering 59-61 tok/s — a full 27% worse than the baseline of 82-83 tok/s without speculation. The verify step, which runs the target model on the draft tokens to check their validity, was taking 30 milliseconds per cycle regardless of whether prefill or decode attention mode was used. This was not a regression from a code change or a configuration error; it was the fundamental cost of running a 3-token extend through a 1T MoE model on eight PCIe GPUs without the benefit of CUDA graphs.

The user, absorbing this analysis in the previous message ([msg 4919]), cut straight to the chase. Instead of asking "how do we fix the 30ms verify time?" or "should we try a different approach?", they asked a precise, forward-looking question: "What accuracy do we need for 150/200tps?" This question reframes the entire problem. It accepts the 30ms verify cost as a given and asks what the draft model would need to achieve to make speculation worthwhile at ambitious throughput targets. Message 4921 is the answer.

The Mathematical Framework

The assistant's response is a model of concise, data-driven reasoning. It opens with a table that maps target throughputs to required acceptance lengths and estimated next-token accuracy:

| Target tok/s | Required accept_len | Required next-token accuracy (est.) | |:---:|:---:|:---:| | 82 (break-even) | 2.46 | ~75% (we have this) | | 100 | 3.0 | ~80% | | 120 | 3.6 | ~85% | | 150 | 4.5 | ~90% | | 200 | 6.0 | ~95% |

The derivation is straightforward: accept_len = target_tps × cycle_time. With a fixed 30ms verify cycle, generating 150 tokens per second requires accepting 4.5 tokens per cycle (150 × 0.030 = 4.5), and 200 tok/s requires 6.0 tokens per cycle.

But acceptance length is not accuracy. The second table performs the crucial conversion using the chain speculation formula for topk=1:

accept_len = 1 / (1 - p)

where p is the per-step conditional accuracy — the probability that the draft model's next token matches what the target model would have generated, given the previous draft tokens were correct. This formula captures the geometric distribution of the number of consecutive correct predictions before a miss.

| accept_len | Conditional accuracy p | |:---:|:---:| | 2.0 (ours now) | 50% | | 2.46 (break-even) | 59% | | 3.0 | 67% | | 3.6 | 72% | | 4.5 | 78% | | 6.0 | 83% |

The relationship is nonlinear and unforgiving. Going from accept_len 2.0 to 2.46 (break-even) requires improving accuracy from 50% to 59% — a 9 percentage point gain. But going from 2.0 to 6.0 (200 tok/s) requires jumping from 50% to 83% — a 33 percentage point leap that represents a fundamentally different regime of draft model quality.

Anchoring to Reality

The assistant then grounds these theoretical targets in the actual performance of real draft models. Their current drafter, trained on 37,000 samples, achieves approximately 64% conditional accuracy (from training validation at TTT step 1-4). This yields an accept_len of roughly 2.0 — well below the 2.46 break-even point, which explains why speculation is currently hurting performance.

The comparison point is AQ-MedAI's Kimi-K2 drafter, trained on 1.4 million samples, which achieves accept_len 3.2-3.5 (~69-71% conditional accuracy). This is the only published data point for a drafter in this model family, and it provides a crucial reality check: even with 38× more training data, the best known drafter achieves only ~70% conditional accuracy — still short of the 78% needed for 150 tok/s.

This comparison reveals the immense gap between where they are and where they need to be. Going from 64% to 78% conditional accuracy is not a matter of simply scaling up data by another factor of 10. The relationship between training data volume and draft model accuracy is itself nonlinear and likely exhibits diminishing returns. The AQ-MedAI result with 1.4M samples suggests that even massive data scaling may not bridge the gap to 78%.

The Verdict: Aggressive but Plausible vs. Very Unlikely

The assistant's assessment is admirably nuanced. For 150 tok/s: "aggressive but plausible" — requiring ~78% conditional accuracy, "well above what even AQ-MedAI achieved." The assistant estimates this would likely require 1M+ training samples and potentially a deeper draft model architecture (2 layers instead of 1). This is not a casual recommendation; it's a sobering assessment of the resources required.

For 200 tok/s: "very unlikely with 2-step speculation at 30ms verify." The required 83% conditional accuracy is characterized as "near state-of-the-art for same-model distillation." The assistant outlines what it would take: massive training data, deeper speculation with a much better model, and possibly reducing the verify cycle time itself. This is effectively a judgment that 200 tok/s is not achievable with the current approach.

The Critical Insight: The Verify Cycle Time Bottleneck

The most important analytical contribution of this message comes in its final paragraph. After laying out the daunting accuracy requirements, the assistant pivots to identify the true bottleneck:

"The verify cycle time is really the bottleneck. If we could get verify down to 15ms (e.g., through CUDA graph support for verify), the targets become much more reachable — 150 tok/s would need accept_len 2.25 (56% accuracy, easily achievable)."

This is the key insight that reframes the entire problem. The 30ms verify time is not a fixed physical constant; it's an artifact of the current implementation, specifically the inability to use CUDA graphs for the verify forward pass because it needs to capture hidden states for the draft model. If this engineering challenge could be solved — if the verify step could be made to run with CUDA graphs, cutting its time in half — then 150 tok/s would require only 56% conditional accuracy, which is below their current drafter's performance. The entire problem flips from "how do we train a dramatically better drafter" to "how do we optimize the verify path."

This reframing is the kind of insight that only emerges from doing the math explicitly. By working through the numbers, the assistant reveals that the problem has two independent levers: draft model accuracy (improved through training) and verify cycle time (improved through engineering). The message implicitly argues that the engineering lever may be the more promising one, since halving the verify time would make 150 tok/s achievable with their current drafter quality.

Assumptions and Limitations

The analysis rests on several assumptions that deserve scrutiny. First, it assumes the 30ms verify cycle time is stable and reproducible — an assumption that had been validated through extensive profiling in the preceding messages. Second, it assumes the chain speculation formula accept_len = 1/(1-p) holds for topk=1, which is mathematically correct for independent per-step predictions but may not capture dependencies between draft steps in practice. Third, it assumes the relationship between training data volume and accuracy follows a predictable trajectory, when in reality it may plateau or exhibit other nonlinearities.

The assistant also implicitly assumes that the draft model architecture (single layer, hidden_size=7168) is sufficient to reach the required accuracy levels with enough data. This is an optimistic assumption — it's possible that the architecture itself is the bottleneck and that even with infinite data, a single-layer draft model cannot achieve 78% conditional accuracy on a 1T-parameter MoE target.

The Broader Significance

This message is a masterclass in applied performance analysis. It takes a complex systems problem — speculative decoding on a distributed GPU system — and distills it into a clean mathematical framework that enables clear decision-making. The two tables encode the entire trade-off space: throughput targets map to acceptance lengths, which map to accuracy requirements, which map to training data and architecture decisions.

The message also demonstrates the importance of asking the right question. The user's query — "What accuracy do we need for 150/200tps?" — cuts through the noise of debugging and profiling to focus on the fundamental viability question. The assistant's response honors that precision with equally precise mathematics, then adds the crucial insight about the verify time bottleneck that the user didn't ask about but needed to know.

In the end, this single message serves as a decision document. It tells the user that 150 tok/s is a stretch goal requiring significant investment in both data and engineering, that 200 tok/s is effectively out of reach with the current approach, and that the most promising path forward may not be training a better drafter but fixing the verify path to enable CUDA graphs. These are the kinds of conclusions that shape the direction of an entire project — and they emerged from 30 lines of markdown tables and a few paragraphs of clear reasoning.