The 81.7 tok/s Benchmark: A Pivotal Measurement in Speculative Decoding Optimization

The Message

[assistant] [bash] ssh root@10.1.230.174 '~/ml-env/bin/python3 /tmp/eagle3_bench.py'
=== Single-stream EAGLE3 benchmark ===
  Prompt 1: 940 tokens in 12.1s = 77.5 tok/s (prompt: 19 tok)
  Prompt 2: 1024 tokens in 10.6s = 96.5 tok/s (prompt: 20 tok)
  Prompt 3: 1024 tokens in 12.9s = 79.4 tok/s (prompt: 24 tok)
  Prompt 4: 1008 tokens in 12.9s = 77.9 tok/s (prompt: 18 tok)
  Prompt 5: 1024 tokens in 12.9s = 79.5 tok/s (prompt: 18 tok)

  Average: 81.7 tok/s (5020 tokens / 61.5s)

This message, appearing at index 3649 in the conversation, is a deceptively simple benchmark output. It reports an average single-stream throughput of 81.7 tokens per second for the EAGLE-3 speculative decoding pipeline running on a Kimi-K2.5 model with a custom-trained draft model. Yet beneath these five lines of numbers lies the culmination of an extensive debugging saga, a carefully orchestrated hyperparameter search, and a critical decision point that would redirect the entire project's trajectory.

The Context: Why This Message Was Written

To understand why this particular benchmark was run, one must trace the narrative backward through several rounds of increasingly refined experimentation. The assistant had been engaged in a multi-day effort to deploy EAGLE-3 speculative decoding — a technique where a smaller "draft" model proposes token sequences that a larger "target" model verifies in parallel — on a Kimi-K2.5 model running across 8 NVIDIA RTX PRO 6000 Blackwell GPUs.

The journey had been fraught. A critical bug had been discovered and fixed just hours earlier: the SGLang server had been started with --speculative-algorithm EAGLE instead of --speculative-algorithm EAGLE3. This seemingly trivial flag difference had catastrophic consequences. The is_eagle3() check in SGLang's codebase is strict — only the exact string EAGLE3 triggers the target model to capture and concatenate intermediate layer hidden states from layers [2, 30, 58]. With the wrong flag, the draft model received 7168-dimensional final-layer-only states instead of the expected 21504-dimensional concatenated states, causing the trained weights to be silently bypassed. The draft model's predictions were never actually being accepted (accept_len stuck at 1.0, meaning every draft token was rejected).

After fixing this bug, the assistant had been systematically benchmarking different configurations to find the optimal throughput. The sequence of tests tells a story of diminishing returns and careful trade-off analysis:

  1. EAGLE3 without CUDA graphs (msg 3622): 53.2 tok/s — worse than the 90 tok/s non-speculative baseline.
  2. AQ-MedAI drafter with EAGLE3 (msg 3628): 50.5 tok/s — even worse, confirming the custom K2.5-trained drafter was better.
  3. EAGLE3 with CUDA graphs, 16 draft tokens (msg 3635): 74.9 tok/s — a significant improvement, showing CUDA graphs dramatically reduced per-step overhead.
  4. EAGLE3 with CUDA graphs, 5 draft tokens (msg 3640): 82.3 tok/s — the best yet, demonstrating that fewer draft tokens reduced speculation overhead.
  5. EAGLE3 with CUDA graphs, 2 steps, 5 draft tokens (msg 3649, this message): 81.7 tok/s — slightly lower than the 3-step config. The message was written because the assistant needed to measure the throughput of the 2-step configuration to complete its hyperparameter sweep. The user had reported a server crash in msg 3644, which forced a restart. The assistant relaunched the server with --speculative-num-steps 2 in msg 3647, waited for it to load in msg 3648, and then ran the benchmark in this message.

The Thinking Process: What the Assistant Was Evaluating

The assistant's reasoning, visible across the surrounding messages, reveals a sophisticated understanding of speculative decoding dynamics. The core tension is between acceptance length (how many draft tokens are accepted per verification step) and speculation overhead (the computational cost of running the draft model and verifying its predictions).

With the 3-step, 5-draft-token configuration, the assistant had observed accept_len ~2.0-2.4 and accept_rate ~0.38-0.47. The average throughput was 82.3 tok/s — still below the 90 tok/s non-speculative baseline. The assistant's hypothesis was that reducing num_steps from 3 to 2 might reduce the overhead of the draft model's forward passes, potentially improving throughput even if the acceptance characteristics remained similar.

The 2-step configuration achieved 81.7 tok/s — essentially identical to the 3-step config within measurement noise. This was a valuable negative result: it confirmed that the number of speculation steps (within this range) was not the primary bottleneck. The limiting factor was the acceptance length itself.

Input Knowledge Required

To fully understand this message, one needs knowledge of several interconnected domains:

Speculative decoding fundamentals: The technique uses a fast draft model to propose multiple tokens, which the target model then verifies in a single forward pass. The key metric is "accept_len" — the average number of draft tokens accepted per verification step. If accept_len is too low, the overhead of running both models exceeds the benefit of generating multiple tokens in parallel.

CUDA graphs: A CUDA optimization technique that captures GPU kernel launches into a reusable graph, eliminating CPU-side launch overhead. The assistant's benchmarks showed CUDA graphs improved throughput from 53.2 tok/s to 74.9 tok/s — a 41% improvement — demonstrating that kernel launch overhead was a significant factor in the speculation pipeline.

EAGLE-3 architecture: The EAGLE-3 draft model uses hidden states from intermediate layers of the target model (layers 2, 30, and 58 for this model) concatenated into a 21504-dimensional vector. This is passed through a "fusion" layer (fc) before being fed into the draft transformer. The earlier bug had prevented this concatenation from happening, making the draft model effectively receive garbage inputs.

SGLang server configuration: The various flags — --speculative-num-steps, --speculative-num-draft-tokens, --speculative-eagle-topk, --num-continuous-decode-steps, --disable-custom-all-reduce — all interact in complex ways. The assistant had to understand how each parameter affected the trade-off between speculation quality and computational overhead.

NCCL tuning: The environment variables NCCL_PROTO=LL, NCCL_ALGO=Ring, NCCL_P2P_LEVEL=SYS, etc., were tuned earlier in the session to optimize inter-GPU communication across the 8-GPU tensor-parallel configuration.

Output Knowledge Created

This message created several important pieces of knowledge:

  1. A data point in the hyperparameter sweep: The 2-step configuration was benchmarked and found to be essentially equivalent to the 3-step configuration (81.7 vs 82.3 tok/s). This ruled out speculation depth as a lever for improvement.
  2. Confirmation of the acceptance length ceiling: The subsequent message (msg 3650) showed accept_len ~2.27 and accept_rate ~0.46 for this configuration, consistent with the 3-step results. This confirmed that the draft model's quality — not the speculation parameters — was the bottleneck.
  3. Evidence for the data scaling hypothesis: The assistant concluded that the fundamental problem was insufficient training data for the draft model. With only 10K training samples, the draft model achieved accept_len ~2.1, far below the 3-4+ needed for effective speculation. This directly motivated the massive data scaling effort that followed — launching an inference pipeline to generate responses for 83K prompts.
  4. A baseline for comparison: The 81.7 tok/s figure became the reference point against which future improvements would be measured. Any improvement to the draft model or speculation pipeline would need to exceed this to be worthwhile.

Assumptions and Potential Mistakes

The assistant made several assumptions in this message:

That the benchmark is representative: The benchmark used 5 prompts of ~1000 tokens each with short system prompts (18-24 tokens). This is a reasonable single-stream benchmark but may not reflect performance under different prompt lengths, batch sizes, or generation patterns.

That the server was fully warmed up: The assistant waited for the server to load but did not explicitly verify that CUDA graph capture had completed before running the benchmark. However, the log from msg 3634 showed CUDA graphs were captured successfully for the 3-step config, and the 2-step config used the same model and draft model.

That the 2-step config was properly started: The server had crashed in msg 3644 during a previous attempt to start the 2-step configuration. The assistant correctly diagnosed the cause (the compound kill command didn't complete properly) and restarted cleanly in msg 3647. The log output in msg 3648 confirmed the model loaded successfully.

That measurement noise is negligible: The five prompts showed variation from 77.5 to 96.5 tok/s, with a standard deviation of roughly 8 tok/s. The difference between 81.7 and 82.3 tok/s is well within this noise floor, so the assistant correctly concluded the two configurations were equivalent.

One subtle issue is that the benchmark script /tmp/eagle3_bench.py was not shown in the conversation, so we cannot verify its methodology. It appears to measure end-to-end generation time including both draft model speculation and target model verification, which is the correct metric. However, if the script measures only the generation phase after the first token, it might miss the initial prompt processing time.

The Broader Significance

This message sits at a pivotal moment in the project. The assistant had spent days building infrastructure: setting up the ML environment with NVIDIA drivers and CUDA, resolving flash-attn build issues, training the EAGLE-3 draft model, debugging the hidden state concatenation bug, and tuning server parameters. The benchmark results in this message represent the culmination of that effort — and the verdict was clear: the current approach was not working well enough.

The 81.7 tok/s result, while respectable, was still ~9% slower than the non-speculative baseline of 90 tok/s. The assistant could have continued tweaking hyperparameters indefinitely, but the data pointed to a more fundamental issue: the draft model needed more training data. This insight directly led to the decision to scale up the training dataset by 10×, launching an inference pipeline to generate responses for 83K prompts — a process expected to take 24-55 hours.

In this sense, the message is not just a benchmark result but a diagnostic instrument. It told the assistant that the speculation pipeline was working correctly (the hidden state bug was truly fixed, CUDA graphs were functioning, the draft model was making predictions that were being accepted) but that the quality of those predictions was insufficient. The path forward was clear: more data, better training, and a return to this benchmark to measure the improvement.

Conclusion

Message 3649 appears to be a routine benchmark output, but it represents the convergence of multiple threads of investigation: a critical bug fix, a systematic hyperparameter search, a nuanced understanding of speculative decoding dynamics, and a data-driven decision about project direction. The assistant's ability to interpret this result — to see beyond the raw numbers to the underlying cause — demonstrates the kind of diagnostic reasoning that separates effective engineering from mere experimentation. The 81.7 tok/s figure would become a baseline to beat, a benchmark that would drive the next phase of the project: scaling up training data to build a draft model that could finally make EAGLE-3 speculation worthwhile.