The Moment of Truth: Benchmarking the EAGLE-3 Drafter at 24.8 tok/s

In the long arc of a complex machine learning engineering project, certain messages mark inflection points — moments where weeks of effort are distilled into a single number. Message [msg 3535] is one such moment. After an extensive debugging session spanning multiple segments, the assistant finally launches a systematic benchmark of the newly trained EAGLE-3 draft model integrated with SGLang, and the results land with a thud: approximately 24.8 tokens per second. This is not merely a disappointing number; it is a diagnostic signal that unravels assumptions built over days of work, pointing toward a fundamental architectural mismatch between how the draft model was trained and how it is being served at inference time.

The Road to This Benchmark

To understand the weight of this message, one must appreciate the journey that preceded it. The project had been building toward EAGLE-3 speculative decoding for the Kimi-K2.5 large language model — a technique where a smaller "draft" model generates candidate tokens that a larger "target" model verifies in parallel, theoretically accelerating inference. The assistant had successfully trained an EAGLE-3 drafter from scratch using hidden states extracted via SGLang ([chunk 25.0]), completing a full training pipeline on 10,000 samples. The training metrics showed a validation loss plateauing around 6.13 with step-0 accuracy at approximately 74.5% — numbers that, while not stellar, suggested the model had learned something meaningful.

The user, in [msg 3508], raised a critical question about whether the model was data-limited given only ~21 million tokens of unique training data for a 1.2-billion-parameter model. The assistant proposed a pragmatic two-step plan: benchmark the current checkpoint first, then decide between generating more data or running a "grokking" continuation with many more epochs. This message is the execution of the first step — the benchmark that would determine the entire trajectory of the project.

What the Message Contains

The message itself is deceptively brief. It consists of a single bash command executed on the remote server, followed by its output:

[assistant] It's generating output. Now let me run the proper benchmark for single-stream tok/s and check if we get acceptance rate logging:
[bash] ssh root@10.1.230.174 "/root/ml-env/bin/python3 /tmp/sglang_bench.py --url http://localhost:8000 --mode single --num-requests 5 --max-tokens 512"

The benchmark script, previously written and stored at /tmp/sglang_bench.py, runs five single-stream requests with a maximum of 512 tokens each. The output shows three completed requests with throughput measurements:

• Prompt on general relativity: 256 completion tokens, 10.33 seconds, 40.4ms TPOT, 24.8 tok/s
• Prompt on merge sort: 256 completion tokens, 10.47 seconds, 40.9ms TPOT, 24.5 tok/s
• Prompt on climate change: 256 completion tokens, 10.42 seconds, 40.7ms TPOT, 24.6 tok/s The assistant's opening line — "It's generating output" — references the preceding message ([msg 3534]) where a single curl request confirmed the server was producing coherent text about Fibonacci memoization. That verification was a sanity check; this benchmark is the systematic measurement.

The Crushing Significance of 24.8 tok/s

To the uninitiated, 24.8 tokens per second might sound respectable — it is, after all, faster than human reading speed. But in the context of this project, the number is devastating. Earlier in the session ([chunk 25.0]), the assistant had tuned SGLang single-stream performance to 90 tok/s without speculative decoding. The EAGLE-3 drafter, far from accelerating inference, is actively slowing it down by a factor of 3.6×.

This immediately signals that the speculative decoding mechanism is not functioning as intended. In a properly working EAGLE-3 system, the draft model proposes multiple candidate tokens (in this configuration, 5 draft tokens with top-k sampling of 4), and the target model verifies them in a single forward pass. If the draft model's predictions are accurate, the acceptance rate should be high — ideally above 70-80% — and throughput should exceed the baseline. If the draft model's predictions are no better than random, every proposed token gets rejected, the system falls back to generating one token at a time, and the overhead of running both models makes throughput worse than the baseline.

The 24.8 tok/s figure tells the assistant, before any deeper log analysis, that the drafter is producing essentially garbage predictions at inference time. This is the same pattern observed with the previous vLLM-trained drafter ([msg 3536] confirms this: accept len: 1.00, accept rate: 0.20 — meaning zero draft tokens accepted out of 5 proposed).

The Thinking Process Visible in the Message

The assistant's reasoning is revealed in the brief preamble: "It's generating output. Now let me run the proper benchmark for single-stream tok/s and check if we get acceptance rate logging." This sentence encodes a multi-step thought process:

1. Verification completed: The single curl request in [msg 3534] confirmed the server is alive and generating coherent text. The EAGLE-3 integration did not crash or produce gibberish at the API level.
2. Need for systematic measurement: A single request is insufficient for performance evaluation. The assistant deliberately chooses a benchmark with 5 requests to average out variance.
3. Dual measurement goal: The assistant explicitly wants both throughput (tok/s) and acceptance rate. The acceptance rate is the more fundamental metric — it reveals why the throughput is what it is. The assistant anticipates that the raw throughput number alone will not tell the full story.
4. Comparison mindset: The assistant is implicitly comparing against the 90 tok/s baseline. There is no explicit mention of the baseline in this message, but the entire framing of "proper benchmark" implies a standard to measure against. The assistant also demonstrates disciplined engineering practice: rather than interpreting the single successful curl as proof of success, it immediately moves to rigorous benchmarking. This is the behavior of an engineer who has been burned by false positives before — a theme that recurs throughout the session as hidden state mismatches, weight key name issues, and API incompatibilities are systematically uncovered and resolved.

Assumptions Embedded in This Message

Every benchmark rests on assumptions, and this one is no exception. The assistant assumes that:

• The SGLang server has correctly loaded the EAGLE-3 draft model: The server startup sequence (messages [msg 3530] through [msg 3532]) involved a prolonged wait and eventual success, but there was no explicit verification that the draft model weights were loaded correctly. The assistant had not yet inspected the server logs for weight loading messages.
• The draft model architecture matches what SGLang expects: The training was done using the speculators library, which saves weights under key names like layers.0.*. SGLang's LlamaForCausalLMEagle3 class expects keys like midlayer.*. This mismatch, discovered in subsequent messages ([msg 3537] onward), means the trained weights were silently dropped during loading — the draft model was effectively running with random initialization.
• The hidden state dimensions match between training and inference: The draft model was trained on 21504-dimensional fused hidden states (concatenating three auxiliary layer outputs), but SGLang passes 7168-dimensional single-layer hidden states at inference time. The fc fusion layer that projects 21504→7168 is never applied because the shape check hidden_states.shape[-1] != embeds.shape[-1] evaluates to 7168 != 7168 → False, bypassing the fusion entirely.
• The benchmark is measuring the system in a steady state: The server had just finished warming up. The benchmark runs immediately after, which is appropriate, but there is no check for whether CUDA graphs have been compiled or whether the speculative decoding pathway has been exercised.

Input Knowledge Required

To fully understand this message, the reader needs knowledge of:

1. The baseline performance: The 90 tok/s single-stream throughput achieved in [chunk 25.0] through NCCL tuning. Without this baseline, 24.8 tok/s might not appear problematic.
2. The EAGLE-3 speculative decoding mechanism: How draft models propose tokens, how acceptance rates work, and the relationship between acceptance rate and throughput. A 0.20 acceptance rate with 5 draft tokens means zero draft tokens are accepted — only the verification pass's base token is kept.
3. The server startup saga: The preceding 20+ messages (from [msg 3514] to [msg 3532]) documenting a server hang that required adding --disable-cuda-graph to resolve. The assistant had to kill the process, free GPU memory, and restart multiple times.
4. The training history: The EAGLE-3 drafter was trained from scratch on 10,000 samples of hidden states extracted via SGLang. The training showed diminishing returns with validation loss plateauing at ~6.13.
5. The user's strategic question: [msg 3508] asking about data scaling vs. grokking, which frames this benchmark as a decision point between two paths.

Output Knowledge Created

This message produces several pieces of actionable knowledge:

• Throughput measurement: 24.6-24.8 tok/s for single-stream inference with EAGLE-3 speculation enabled.
• Verification of server functionality: The server handles multiple concurrent requests correctly, returning coherent responses.
• A clear negative signal: The speculation is actively harming performance, ruling out the "try grokking" path and confirming that something is fundamentally broken.
• A diagnostic direction: The poor throughput necessitates inspecting acceptance rates and weight loading, which the assistant proceeds to do in the following messages ([msg 3536] through [msg 3539]).

Mistakes and Incorrect Assumptions

The most significant incorrect assumption embedded in this message is that the benchmark would produce meaningful results about the trained drafter's quality. In reality, the benchmark measures the quality of the weight loading and architecture integration, not the quality of the trained model. The trained weights were silently discarded due to key name mismatches between the speculators library's saving format and SGLang's loading expectations. The assistant could not have known this without inspecting the model's weight keys against SGLang's expected keys — which it does immediately after receiving these results.

A secondary issue is that the benchmark does not explicitly log acceptance rate alongside throughput. The assistant mentions wanting to "check if we get acceptance rate logging," implying the benchmark script may not capture this metric directly. In the subsequent message ([msg 3536]), the assistant has to grep the server logs separately to find accept len: 1.00, accept rate: 0.20. A more informative benchmark would have included acceptance rate in its output.

Conclusion

Message [msg 3535] is a moment of diagnostic clarity in a complex engineering project. The stark number — 24.8 tok/s — cuts through weeks of incremental progress and forces a fundamental reassessment. It is a testament to the value of measurement: no amount of reasoning about data scaling, grokking, or training improvements could substitute for the simple act of running the model and measuring its output. The assistant's disciplined approach — verify, benchmark, diagnose, fix — turns a disappointing result into a precise diagnostic signal that ultimately leads to the discovery of the weight key name mismatch and the hidden state dimensionality problem. In the end, the benchmark did its job: it revealed the truth, however uncomfortable, and pointed the way forward.