"What Speedup at Current Perf?" — The Pivotal Question That Validates an EAGLE-3 Training Run

Message: [user] What speedup at current perf?

The Moment of Truth in Speculative Decoding

In any machine learning project that involves training a new model component, there comes a critical inflection point: the moment when raw training metrics must be translated into real-world inference performance. The user's question — "What speedup at current perf?" — represents exactly this inflection point in the development of an EAGLE-3 speculative decoding drafter for the Kimi-K2.5 1-trillion-parameter Mixture-of-Experts model. The question is deceptively simple, but it carries enormous weight: after hours of training across four NVIDIA RTX PRO 6000 Blackwell GPUs, consuming tens of thousands of training samples, the team needs to know whether their investment is paying off.

The message arrives at a specific moment in the conversation. The assistant had just generated and displayed a training progress chart ([msg 4304]) showing the drafter's performance at step 4,216 — roughly 60% through epoch 1 of a planned 5-epoch training run. The chart revealed promising numbers: a step-0 (next-token) accuracy of 74.2%, conditional accuracy across all five TTT (Training-The-Token) steps ranging from 55% to 71%, and an estimated acceptance length of approximately 2.79 tokens. But these are training metrics — they measure the drafter's predictive accuracy on held-out validation data, not its actual impact on inference throughput. The user's question demands the bridge between these two domains.

Why This Question Matters: The History of Speculation on This System

To understand the weight of this question, one must appreciate the history that precedes it. Earlier in the session, the team had deployed a smaller EAGLE-3 drafter trained on only 10,000 samples ([msg 4292]). That drafter achieved an acceptance length of approximately 2.1 tokens — meaning that on average, only about 1 out of every 5 draft tokens was accepted by the verifier. When benchmarked on the actual SGLang inference server, this drafter produced 82.3 tok/s, which was actually slower than the no-speculation baseline of 90.0 tok/s. The overhead of running the drafter and the verification forward pass exceeded the savings from the few accepted tokens.

This is the nightmare scenario for speculative decoding: a drafter that is not accurate enough becomes a net liability. The 10K drafter's failure established a clear breakeven point: the acceptance length needed to exceed approximately 2.2 tokens just to match the baseline, and anything beyond that would yield genuine speedup. The new 100K-sample drafter, trained with TTT=5 (five levels of teacher-forced token prediction) and a more sophisticated training pipeline, was supposed to be the solution. But at epoch 1, with the training only 60% complete, the user is asking the critical question: are we there yet?

The Assumptions Embedded in a Three-Word Question

The user's question, while brief, makes several implicit assumptions that deserve examination. First, it assumes that training metrics (full accuracy and conditional accuracy at each TTT step) can be reliably mapped to inference throughput. This mapping is not straightforward — it depends on the speculative decoding algorithm's implementation details, the hardware characteristics of the inference server, and the interaction between the drafter and the verifier model.

Second, the question assumes that the current performance at epoch 1 is representative enough to warrant a speedup calculation. Training is still in progress; the loss curve is still descending, and the learning rate has just reached its peak value of 3e-5. The cosine decay schedule has not even begun. The user is implicitly asking for a projection based on partial data, which carries significant uncertainty.

Third, the question assumes that "speedup" is a well-defined quantity in this context. In speculative decoding, speedup depends on the number of speculative tokens per round, the overhead of the drafter forward pass, the verification cost, and the system's communication topology. On a PCIe-connected multi-GPU system (as opposed to NVLink-connected GPUs), the allreduce communication cost per decode step is particularly high, which actually favors deeper speculation — each accepted token saves an expensive allreduce round. But this also means the speedup calculation must account for system-specific overheads that are not captured in training metrics.

The Assistant's Reasoning: Translating Accuracy to Throughput

The assistant's response to this question ([msg 4306]) reveals a sophisticated multi-step reasoning process. The calculation proceeds in several layers:

Layer 1: Extracting the raw accuracy numbers. The assistant pulls the full accuracy (cumulative probability that all tokens up to step i are correct) from the training metrics: 74.2% at step 0, 47% at step 1, 29% at step 2, 15% at step 3, and 9.7% at step 4. These numbers are notably more conservative than the conditional accuracy figures shown in the chart, because full accuracy is the product of all conditional probabilities up to that step — it's a stricter measure.

Layer 2: Computing expected acceptance length. The expected number of tokens accepted per speculation round is the sum of full accuracies plus 1 (the verifier always produces at least one token, the one that was verified). For N=5 speculative tokens: E[accept] = 0.742 + 0.47 + 0.29 + 0.15 + 0.097 + 1 = 2.75. The assistant then extrapolates to deeper speculation depths (10 and 16 tokens) by assuming the conditional accuracy trend continues at ~50-55%, yielding estimates of ~3.1 and ~3.3 respectively.

Layer 3: Modeling the speedup formula. The theoretical speedup is E[accept] / (1 + draft_overhead), where draft_overhead is the ratio of drafter runtime to verifier decode step runtime. From the previous benchmark, the verifier decode step takes approximately 11ms (90 tok/s baseline), and the EAGLE-3 drafter (a single transformer layer on one GPU) takes approximately 1-2ms for N tokens. This gives draft_overhead ≈ 0.15, leading to an optimistic speedup of 2.75 / 1.15 ≈ 2.39x, or approximately 215 tok/s.

Layer 4: Calibrating against real benchmarks. The assistant wisely recognizes that the theoretical model is too optimistic. The previous 10K drafter benchmark provides a crucial calibration point: at accept_len 2.1, the measured throughput was 82.3 tok/s — a 0.91x slowdown relative to the 90 tok/s baseline. This implies that the real overhead (including CUDA graph replay, KV cache management, and verification of N+1 tokens being slower than 1 token) is significantly higher than the simple model predicts. Using this calibration, the assistant estimates that accept_len 2.75 would yield approximately 110-120 tok/s (1.2-1.3x speedup), and projects that after 4 more epochs, accept_len 3.0-3.5 could yield 130-160 tok/s (1.4-1.8x).

The Knowledge Boundaries: What You Need to Understand This Message

To fully grasp the significance of this exchange, a reader needs substantial context. They must understand the architecture of speculative decoding — how a small "drafter" model proposes tokens that a large "verifier" model checks in parallel. They need to know the specific challenges of PCIe-connected multi-GPU systems, where communication overhead dominates decode latency. They must be familiar with the EAGLE-3 architecture (a single transformer layer that predicts hidden states rather than token IDs directly) and the TTT training methodology (where the drafter is trained to predict tokens at multiple depths using teacher forcing).

The reader also needs the history of this specific project: the failed 10K drafter that was slower than baseline, the 100K training data pipeline that was built from scratch using OpenRouter API generation, the Triton shared-memory OOM bugs that were fixed during training setup (<msg id=4278-4280>), and the batch-size optimization that increased GPU utilization from ~250W to ~400W (<msg id=4287-4289>). Without this context, the question "What speedup at current perf?" appears to be a simple request for a number. With it, the question becomes a high-stakes evaluation of whether weeks of infrastructure work will pay off.

What This Message Creates: A Decision Framework

The output of this message and its response is not just a number — it's a decision framework. The assistant's analysis tells the user that at epoch 1, the drafter is already above the breakeven point (accept_len 2.75 > ~2.2 breakeven), meaning it would likely produce a modest speedup of 1.2-1.3x if deployed immediately. But more importantly, the analysis projects that continued training (4 more epochs) could push the speedup to 1.4-1.8x. This creates a clear cost-benefit calculation: the remaining ~9.5 hours of training time is worth it if the speedup target is above ~1.3x.

The message also implicitly validates the entire training approach. The 100K-sample dataset, the TTT=5 training methodology, the batch packing optimization, and the careful handling of Triton compilation issues — all of these decisions are vindicated by the fact that the drafter is already outperforming its predecessor at epoch 1. The user's question, by asking for the speedup projection, signals that they are ready to make deployment decisions based on the training results. The answer they receive confirms that the project is on track.

Potential Pitfalls in the Analysis

While the assistant's response is thorough, it rests on several assumptions that warrant scrutiny. The calibration using the 10K drafter benchmark assumes that the overhead structure is the same between the two drafters — but the new drafter is trained with TTT=5, which may produce different draft token distributions that interact differently with the verification process. The extrapolation from N=5 to N=10-16 speculative tokens assumes that conditional accuracy remains stable at deeper depths, but the training only provides metrics up to step 4. The speedup calculation also assumes that the verifier's forward pass cost scales linearly with the number of tokens being verified, which may not hold for the MoE architecture with its expert routing and allreduce communication.

Additionally, the analysis does not account for the possibility of adaptive speculation — dynamically adjusting the number of draft tokens based on recent acceptance rates — which could improve real-world throughput beyond the static estimates. Nor does it consider the impact of serving multiple concurrent requests, where the drafter and verifier might be scheduled differently.

Conclusion

The user's three-word question — "What speedup at current perf?" — encapsulates the entire purpose of the EAGLE-3 training project. It bridges the gap between training metrics and inference performance, between academic accuracy numbers and real-world tokens-per-second throughput. The assistant's response provides not just an estimate but a calibrated, multi-layered analysis that accounts for theoretical models, empirical benchmarks, and system-specific overheads. In doing so, it transforms a simple question into a comprehensive validation of the training pipeline and a clear roadmap for the remaining work. This message, brief as it is, represents the moment when the team asks: is this working? — and receives an answer that justifies the effort invested so far.