The Speedup Question: Estimating Speculative Decoding Performance from Partial Training Data

In the middle of a marathon EAGLE-3 training session on an 8-GPU RTX PRO 6000 Blackwell machine, a critical moment arrives. The user, who has been monitoring the training progress of a speculative decoding drafter for the massive Kimi-K2.5 language model, asks a simple but loaded question: "What speedup at current perf?" ([msg 4305]). The assistant's response — message 4306 — is a masterclass in bridging the gap between raw training metrics and real-world inference performance, and it reveals the complex reasoning chain required to translate model accuracy numbers into practical throughput estimates.

The Message in Full

The assistant's response begins with a clear restatement of the current drafter accuracy metrics at the end of epoch 1 of 5:

Current drafter accuracy (end of epoch 1): - Step 0: 74.2% (full_acc) - Step 1: 47% (full_acc, meaning both step 0 and 1 correct) - Step 2: 29% - Step 3: 15% - Step 4: 9.7%

It then walks through the mathematics of expected accepted tokens per speculation round, builds a speedup formula factoring in the overhead of running the drafter alongside the verifier, calibrates against a previous benchmark of an earlier (worse) drafter, and concludes with projections for both the current epoch and the expected final performance after all five epochs complete.

Why This Message Was Written: The Motivation and Context

This message sits at a pivotal juncture in a much larger project. The team has been building a speculative decoding system for Kimi-K2.5 — a 1-trillion-parameter Mixture-of-Experts model — using the EAGLE-3 architecture. Speculative decoding is a technique where a small, fast "drafter" model proposes multiple candidate tokens, and the large "verifier" model validates them in a single forward pass. The goal is to accelerate inference by amortizing the cost of the large model's forward pass across multiple accepted tokens.

The project had already gone through multiple iterations. An earlier drafter trained on only 10,000 samples had achieved an acceptance length of approximately 2.1 tokens — but this actually degraded performance compared to the baseline, dropping from 90 tokens per second (tok/s) to 82.3 tok/s ([msg 4292]). The overhead of running the drafter and managing the speculation pipeline exceeded the benefit of such a modest acceptance rate.

Now, a new drafter was being trained on 100,000 samples with a more sophisticated training regime (TTT=5, meaning 5 "think-while-transit" steps per training example). The training had been running for roughly 10.8 hours and was only 60% through epoch 1 of 5 ([msg 4298]). Yet the metrics already looked dramatically better than the 10K drafter. The user, seeing the promising loss curves and accuracy charts ([msg 4304]), wanted to know: is this actually going to be worth it? The question "What speedup at current perf?" is fundamentally a go/no-go checkpoint — a request to translate the intermediate training metrics into a concrete prediction about whether the engineering effort would pay off.

The Decision-Making Process: How the Assistant Constructed the Estimate

The assistant's response reveals a carefully structured reasoning process. It doesn't just quote the accuracy numbers and declare victory. Instead, it builds a multi-layered argument.

Step 1: Extracting the raw metrics. The assistant starts by listing the "full_acc" (full accuracy) numbers for each of the 5 TTT steps. These numbers represent the probability that all tokens from step 0 through step N are correct — a cumulative probability, not a conditional one. This distinction is crucial. The step 0 accuracy of 74.2% means the drafter predicts the very next token correctly about three-quarters of the time. The step 4 accuracy of 9.7% means that the drafter gets a full 5-token sequence exactly right less than 10% of the time. On the surface, these numbers look modest — but they compound.

Step 2: Computing the expected acceptance length. The assistant applies the standard formula for expected accepted tokens in speculative decoding:

E[accept] = sum(full_acc_i for i in 0..N-1) + 1

The "+1" accounts for the guaranteed token produced by the verifier even when all draft tokens are rejected. At N=5 speculative tokens, this yields 0.742 + 0.47 + 0.29 + 0.15 + 0.097 + 1 = 2.75. This is the core metric: on average, each speculation round produces 2.75 tokens.

Step 3: Extrapolating to deeper speculation. The assistant notes that deeper speculation (10-16 tokens) could yield better results, but acknowledges that training metrics only exist for 5 TTT steps. It extrapolates from the conditional accuracy trend (~55-58% at steps 3-4) to estimate ~50-55% at deeper steps, projecting accept lengths of ~3.1 (at depth 10) and ~3.3 (at depth 16). This is a reasoned guess, not a hard calculation.

Step 4: Building the speedup model. The assistant constructs a speedup formula:

speedup = E[accept] / (1 + draft_overhead)

It estimates draft_overhead ≈ 0.15 — the drafter (a tiny 1-layer model running on a single GPU) takes about 15% of the time of one verifier decode step (an 8-GPU allreduce-heavy forward pass through a 1T MoE). This yields an optimistic 2.75 / 1.15 ≈ 2.39x, or ~215 tok/s.

Step 5: Calibrating against reality. Crucially, the assistant doesn't stop at the optimistic estimate. It recognizes that the 10K drafter, with accept_len 2.1, achieved worse throughput than the baseline (82.3 vs 90 tok/s). This implies significant unaccounted overhead — CUDA graph replay issues, KV cache management costs, verification of N+1 tokens being slower than 1 token, and other engineering realities. The assistant uses this as a calibration point, scaling linearly from the breakeven point (~2.2 accept_len) to estimate ~110-120 tok/s at accept_len 2.75.

Step 6: Projecting forward. Finally, the assistant looks ahead to the end of training (epochs 2-5), projecting accept_len 3.0-3.5 and throughput of ~130-160 tok/s — a 1.4-1.8x speedup. This frames the current performance as a "not earth-shattering yet" intermediate result, with the real gains coming from deeper speculation and further training.

Assumptions Embedded in the Analysis

The assistant's reasoning rests on several key assumptions, some explicit and some implicit.

The linear calibration assumption. The assistant assumes that the relationship between accept_len and throughput is roughly linear between the breakeven point (~2.2) and the target (~2.75). This is a reasonable first-order approximation, but the actual relationship is likely nonlinear — small improvements near the breakeven point could produce disproportionately large gains as fixed overheads are amortized.

The overhead extrapolation assumption. The assistant assumes that the unaccounted overheads from the 10K drafter benchmark (CUDA graph replay, KV cache management, etc.) scale proportionally with accept_len. In reality, some overheads are fixed per speculation round (regardless of how many tokens are accepted), while others scale with the number of draft tokens. The assistant's linear calibration implicitly averages these effects.

The conditional accuracy extrapolation. For deeper speculation (10-16 tokens), the assistant extrapolates conditional accuracy of ~50-55% from the observed 55-58% at steps 3-4. This assumes that the conditional accuracy doesn't collapse at deeper levels — a reasonable but unverified assumption.

The drafter overhead estimate. The 15% overhead figure (draft_overhead ≈ 0.15) is based on the drafter being a "tiny model" running on a single GPU. But the EAGLE-3 drafter, while small relative to the 1T verifier, still requires attention operations over the hidden states, and its runtime depends on sequence length and speculation depth. The assistant doesn't provide a source for this 15% figure, and it may be optimistic.

The PCIe communication model. The entire analysis is predicated on the bottleneck being PCIe allreduce communication across 8 GPUs. This is stated explicitly in the message: "each verification step requires a full forward pass of the 1T MoE across 8 GPUs with PCIe allreduce." This assumption is well-supported by earlier benchmarks showing that the baseline SGLang server achieved 90 tok/s with NCCL tuning and continuous decode steps ([msg 4292]), but it's worth noting that the actual bottleneck could shift with different batch sizes or speculation configurations.

Potential Mistakes and Incorrect Assumptions

While the assistant's analysis is thorough, several points warrant scrutiny.

The breakeven accept_len calculation. The assistant states that the breakeven accept_len was ~2.2 (where speculation equals baseline). This is inferred from the 10K drafter's performance (82.3 tok/s at accept_len 2.1 vs 90 tok/s baseline). However, the breakeven point could be higher or lower depending on the specific overheads of the speculation implementation. If the overhead is higher than assumed, the breakeven point shifts right, and the current accept_len of 2.75 might yield less speedup than estimated.

The optimistic 2.39x calculation. The assistant presents the formula 2.75 / (1 + 0.15) ≈ 2.39x but immediately dismisses it as "optimistic." This is good scientific practice — presenting the idealized model before applying real-world corrections. But a reader might miss the qualification and take the 215 tok/s figure at face value.

The missing batch-size dimension. The user's question "What speedup at current perf?" doesn't specify batch size. The assistant assumes single-stream (BS=1) throughout, which is the latency-sensitive regime where speculative decoding matters most. However, the user had just asked about "other batch sizes" in the previous exchange ([msg 4293]), and the assistant had acknowledged that systematic benchmarking hadn't been done. The speedup estimate would be very different at BS=4 or BS=8, where the verifier is already well-saturated and speculation provides diminishing returns.

The missing verification cost model. The assistant's speedup formula treats the verification forward pass as costing exactly the same as a single decode step, regardless of how many tokens are being verified. In reality, verifying N+1 tokens in a single forward pass requires processing a longer sequence, which increases the attention computation cost. For a 1T MoE model, this cost increase is sublinear (the expert computation dominates), but it's not zero.

Input Knowledge Required to Understand This Message

To fully grasp the assistant's response, a reader needs substantial background knowledge spanning multiple domains.

Speculative decoding fundamentals. The reader must understand the basic mechanism: a small drafter model proposes multiple candidate tokens, the large verifier model validates them in a single forward pass, and accepted tokens are kept while rejected ones are discarded. The key metric — acceptance length — represents the average number of tokens accepted per speculation round.

The EAGLE-3 architecture. EAGLE-3 is a specific speculative decoding framework that uses a lightweight transformer layer conditioned on the verifier's hidden states. The "TTT" (think-while-transit) training regime involves multiple steps of autoregressive prediction during training, which is why the metrics report accuracy at steps 0 through 4.

Full accuracy vs. conditional accuracy. The assistant uses "full_acc" (the probability that all tokens up to step N are correct) to compute the expected acceptance length. A reader unfamiliar with this distinction might misinterpret the 9.7% step-4 accuracy as "the drafter is terrible at depth 4" rather than understanding it as the compound probability of getting all 5 tokens right.

The PCIe bottleneck context. The entire speedup analysis hinges on the fact that this is a PCIe-connected multi-GPU system (not NVLink). The assistant has previously established that the verifier's decode step is dominated by allreduce communication across 8 GPUs over PCIe, which is why speculation can help — it reduces the number of allreduce rounds per token.

The previous benchmark results. The assistant references the 10K drafter's performance (82.3 tok/s at accept_len 2.1) as a calibration point. Without this context, the reader wouldn't understand why the assistant doesn't simply use the optimistic 2.39x formula.

Output Knowledge Created by This Message

This message generates several important pieces of knowledge that shape the subsequent direction of the project.

A concrete speedup projection. The assistant translates abstract accuracy metrics into actionable throughput estimates: ~110-120 tok/s at current epoch, ~130-160 tok/s after full training. This gives the user a clear expectation of whether the training investment is paying off.

A validated methodology for estimating speculative decoding performance. The assistant demonstrates a reproducible approach: extract full_acc metrics from training logs, compute expected acceptance length, build a speedup formula with overhead terms, calibrate against real-world benchmarks, and project forward. This methodology can be reused for future drafter iterations.

Identification of the breakeven point. By calibrating against the 10K drafter, the assistant implicitly identifies that the breakeven accept_len for this system is approximately 2.2 tokens. This is a critical engineering parameter — it tells the team the minimum quality threshold their drafter must exceed to be net beneficial.

A roadmap for further improvement. The message identifies three levers for future gains: more training epochs (loss still dropping), deeper speculation (10-16 tokens amortizes fixed overhead), and the strong conditional accuracy (55-70%) which enables deeper speculation to add more accepted tokens.

A framing of "not earth-shattering yet." The assistant's honest assessment — that the current speedup is modest but the trend is positive — sets realistic expectations. This prevents premature celebration or, conversely, premature abandonment of the approach.

The Thinking Process: A Window into Engineering Reasoning

The assistant's reasoning in this message is notable for its structure and intellectual honesty. It proceeds through several distinct phases.

Phase 1: Data extraction. The assistant begins by pulling the raw accuracy numbers from the training metrics. This is mechanical but essential — it grounds the analysis in actual measurements rather than speculation.

Phase 2: Naive calculation. The assistant computes the expected acceptance length using the standard formula. This is the "textbook" answer — what you'd get if you applied the theory without considering engineering realities.

Phase 3: Optimistic extrapolation. The assistant builds the idealized speedup model (2.39x) with the explicit caveat that it's optimistic. This serves as an upper bound — the best-case scenario if all overheads were perfectly accounted for.

Phase 4: Reality calibration. The assistant introduces the real-world benchmark of the 10K drafter, which performed worse than the baseline despite having a positive accept_len (2.1). This forces a downward revision of the speedup estimate. The assistant doesn't have a detailed breakdown of the overhead sources, but it uses the empirical data to calibrate.

Phase 5: Conservative projection. The final estimate (~110-120 tok/s) is significantly lower than the optimistic bound but grounded in actual system behavior. The assistant then projects forward to the end of training, acknowledging uncertainty but providing a plausible range.

This multi-phase reasoning is characteristic of good engineering analysis. The assistant doesn't just report numbers — it builds a model, tests it against reality, adjusts, and communicates the uncertainty at each step. The message also shows intellectual humility: the assistant explicitly states "that's optimistic" and "rough estimate" rather than presenting the numbers as definitive predictions.

Conclusion

Message 4306 is a pivotal moment in the EAGLE-3 training pipeline. It answers the user's urgent question — "is this working?" — with a nuanced, multi-layered analysis that bridges the gap between training metrics and real-world inference performance. The assistant's reasoning demonstrates how to translate abstract accuracy numbers into concrete throughput projections, calibrate theoretical models against empirical benchmarks, and communicate uncertainty honestly. The resulting estimates (~110-120 tok/s at epoch 1, ~130-160 tok/s after full training) set realistic expectations and provide clear direction for the remaining engineering work. In doing so, the message exemplifies the kind of rigorous, context-aware analysis that separates a useful AI assistant from a mere number-cruncher.