The Moment the Speculation Dream Died: Benchmarking Baseline Throughput Against EAGLE-3

Introduction

In the long and arduous journey of deploying a 1-trillion-parameter Mixture-of-Experts model (Kimi-K2.5 INT4) across eight NVIDIA RTX PRO 6000 Blackwell GPUs connected via PCIe Gen5, the question of speculative decoding had become an obsession. For hours spanning multiple sessions, the assistant had wrestled with CUDA toolchain upgrades, NCCL tuning, FlashInfer allreduce fusion patches, and EAGLE-3 drafter training — all in pursuit of making speculative decoding beat the baseline. And in the single-stream case, it had finally worked: EAGLE-3 reached 96.1 tok/s, edging past the 92.6 tok/s baseline. But a far more important question remained unanswered: does speculation help under concurrent load? Message 5435 is the moment that question was put to the test, and its results would reshape the entire trajectory of the optimization effort.

The Message

The subject message is deceptively simple — a single bash command executed on a remote container, launching a parallel throughput benchmark against the baseline server:

[assistant] Good, the benchmark script is ready. Now let me run the full parallel benchmark on the baseline server. This will take a while since it runs through all concurrency levels.

>

[bash] ssh root@[REDACTED_IP] '~/ml-env/bin/python3 /tmp/benchmark_parallel.py --server-url http://localhost:30000 --max-tokens 512 --concurrency 1 2 5 10 30 70 100 250'

>

Parallel throughput benchmark Server: http://localhost:30000 Max tokens: 512 Concurrency levels: [1, 2, 5, 10, 30, 70, 100, 250]

>

Warming up with 5 requests... Warmup done.

>

C= 1 | 30 requests | throughput: 92.6 tok/s | per-req: 92.6 tok/s | latency avg/p50/p99: 5.5/ 5.5/ 5.6s | ok: 30/30 C= 2 | 30 requests | throughput: 159.7 tok/s | per-req: 79.8 tok/s | latency avg/p50/p99: 6.4/ 6.4/ 6.5s | ok: 30/30 C= 5 | 30 requests | throughput: 292.5 tok/s | pe...

The output is truncated in the conversation view, but the first three concurrency levels already tell a compelling story: the baseline is scaling beautifully. At C=1, it matches the known single-stream baseline of 92.6 tok/s. At C=2, it nearly doubles to 159.7 tok/s. At C=5, it reaches 292.5 tok/s. The assistant is about to discover that this scaling continues all the way to saturation at ~780 tok/s — a result that would definitively end the hope that EAGLE-3 speculative decoding could be a net-positive under load.

Why This Message Was Written

This message was the execution of a carefully planned experiment. The assistant had been following a detailed game plan documented in the file eagle-fast-verify.md, which explicitly listed the next steps:

  1. Kill the current EAGLE-3 server and zombie processes
  2. Start a baseline server (no speculation) with flashinfer attention backend and allreduce fusion
  3. Run the same parallel benchmark at concurrency levels C=1, 2, 5, 10, 30, 70, 100, 250 with --max-tokens 512
  4. Compare EAGLE-3 vs baseline results to find the crossover point
  5. Investigate dynamic speculation disable based on concurrency The previous messages in the conversation (msgs 5423-5434) had methodically worked through steps 1 and 2. The EAGLE-3 server was killed, zombie GPU processes were cleaned up, and the baseline server was started with the command:
~/ml-env/bin/python3 -m sglang.launch_server --model-path /shared/kimi-k2.5-int4 --tp 8 --trust-remote-code --cuda-graph-max-bs 128 --disable-custom-all-reduce --attention-backend flashinfer --enable-flashinfer-allreduce-fusion

The server took approximately 10 minutes to load the 547 GB model, and the assistant verified it was healthy by checking the /health and /v1/models endpoints. With the server confirmed operational, the next logical step was to run the benchmark — and that is exactly what message 5435 does.

The underlying motivation was the search for a "crossover point" — a concurrency level below which EAGLE-3 speculation would provide better throughput than the baseline. The assistant (and presumably the user) believed such a point existed. After all, EAGLE-3 had beaten the baseline in single-stream tests (96.1 vs 92.6 tok/s). The reasoning was that at low concurrency, when the GPU has idle capacity, the extra compute required for draft model inference and verification could be absorbed, and the speculation's ability to generate multiple tokens per forward pass would shine. At high concurrency, the overhead would become a liability. The benchmark was designed to find that threshold.

How Decisions Were Made

The decision to run this benchmark was not made in isolation — it was the culmination of a long chain of reasoning. The assistant had spent significant effort optimizing EAGLE-3 speculation:

Assumptions Made

Several assumptions underlay this experiment:

  1. The crossover point exists: The assistant assumed that EAGLE-3 would outperform baseline at low concurrency, with the baseline taking over at higher concurrency. This was the hypothesis being tested.
  2. The benchmark script produces comparable results: The assistant assumed that running the same benchmark_parallel.py script on both servers would yield directly comparable numbers. This was a reasonable assumption, as the script measures total throughput (tok/s) across all concurrent requests using the same set of prompts.
  3. The baseline server configuration is optimal: The assistant used --cuda-graph-max-bs 128 (previously found to improve single-stream performance) and --attention-backend flashinfer (previously found to be ~10% faster than triton). These choices were based on earlier experiments.
  4. The prompts are representative: The benchmark used a set of diverse prompts including "Explain the theory of general relativity in detail" and similar knowledge-intensive questions. The assistant assumed these would produce similar acceptance rates to the actual deployment workload.
  5. The server is stable: After the 10-minute model loading time, the assistant assumed the server was fully initialized and ready for benchmarking.

Mistakes and Incorrect Assumptions

The most significant incorrect assumption was that EAGLE-3 would outperform baseline at low concurrency. The results (revealed in the subsequent message, msg 5437) showed that baseline won at EVERY concurrency level. At C=1, baseline achieved 92.6 tok/s vs EAGLE-3's 77.5 tok/s — a 19% deficit even at the lowest concurrency. The gap only widened from there, reaching 2.3x at saturation (772 vs 341 tok/s).

This was a devastating finding. The earlier single-stream benchmark had shown EAGLE-3 at 96.1 tok/s, but that test used a fixed prompt with high acceptance rate. The parallel benchmark used diverse prompts, and the average acceptance rate was lower, dragging down the effective throughput. The assistant noted this discrepancy in msg 5437:

"Wait — but single-stream EAGLE-3 was 96.1 tok/s in our earlier test, and now the parallel benchmark shows 77.5 tok/s at C=1. [...] That's because the parallel benchmark uses max_tokens=512 with diverse prompts, while the single-stream benchmark used a fixed prompt. The acceptance rate varies per prompt."

This highlights a subtle but critical mistake: the single-stream benchmark had used an unrepresentative prompt, giving an inflated impression of EAGLE-3's capabilities. The parallel benchmark, with its diverse prompts, revealed the true average performance.

Another assumption that proved incorrect was the idea that speculation overhead would be "absorbed" at low concurrency. In reality, even at C=1, the draft + verify cycle consumed enough compute that the baseline's simpler forward pass won. The speculation was not just neutral at low concurrency — it was actively harmful.

Input Knowledge Required

To understand this message, one needs to know:

  1. The benchmark script: benchmark_parallel.py is a Python script that sends concurrent requests to an SGLang server at specified concurrency levels, measuring total throughput (tok/s), per-request throughput, latency distribution, and error rate. It uses concurrent.futures for parallel request dispatch and requests for HTTP calls.
  2. The server configuration: The baseline server was started with specific flags: --tp 8 (tensor parallelism across 8 GPUs), --cuda-graph-max-bs 128 (CUDA graph batch size), --disable-custom-all-reduce (disables SGLang's custom allreduce in favor of NCCL), --attention-backend flashinfer (uses FlashInfer for attention), and --enable-flashinfer-allreduce-fusion (fuses allreduce operations).
  3. The hardware context: 8× NVIDIA RTX PRO 6000 Blackwell GPUs connected via PCIe Gen5 (no NVLink), running CUDA 13.0 with NCCL 2.27.7.
  4. The previous EAGLE-3 results: The same benchmark had been run on the EAGLE-3 server, showing saturation at ~340 tok/s. The baseline results would be compared against these numbers.
  5. The SSH infrastructure: The assistant connects through a Proxmox host (10.1.2.6) to an LXC container (10.1.230.174) where the SGLang server runs.

Output Knowledge Created

This message produced the baseline parallel throughput numbers at concurrency levels 1, 2, and 5 (with the remaining levels to follow in the same benchmark run). Specifically:

The Thinking Process Visible in Reasoning

The assistant's reasoning in this message is minimal but telling. The opening line — "Good, the benchmark script is ready. Now let me run the full parallel benchmark on the baseline server. This will take a while since it runs through all concurrency levels" — reveals several things:

  1. Confidence in the plan: The assistant had verified the script existed (msg 5434) and the server was healthy (msgs 5431-5432). There was no hesitation.
  2. Awareness of time cost: The phrase "This will take a while" shows the assistant understood the benchmark's duration. With 30 requests per concurrency level × 8 levels = 240 requests, each generating 512 tokens at variable throughput, the total runtime could be 10-30 minutes depending on concurrency.
  3. No expectation of surprise: The assistant ran the benchmark as a routine step, expecting to find a crossover point. There was no indication of the devastating result to come. The truncated output shows the benchmark running in real-time, with the assistant likely watching the output stream. The first three data points (C=1, 2, 5) were visible, and they already showed a clear pattern: the baseline was scaling almost linearly with concurrency. At C=5, it was already at 292.5 tok/s — approaching the EAGLE-3 saturation point of ~340 tok/s. The assistant may have begun to suspect the outcome at this point, but the full picture wouldn't emerge until all concurrency levels completed.

The Aftermath and Significance

The true significance of message 5435 is only visible in retrospect, when the full results are compared in msg 5437. The comparison table revealed:

| C | EAGLE-3 (tok/s) | Baseline (tok/s) | Ratio | |---|---|---|---| | 1 | 77.5 | 92.6 | 0.84x | | 2 | 125.1 | 159.7 | 0.78x | | 5 | 183.2 | 292.5 | 0.63x | | 10 | 239.6 | 457.2 | 0.52x | | 30 | 299.5 | 711.0 | 0.42x | | 70 | 337.7 | 784.2 | 0.43x | | 100 | 338.8 | 761.6 | 0.44x | | 250 | 340.9 | 772.1 | 0.44x |

The baseline didn't just win — it demolished EAGLE-3 at every level. The ratio went from 0.84x at C=1 down to 0.42x at C=30, meaning EAGLE-3 was less than half as efficient as the baseline under load. The speculation dream was dead.

This finding forced a fundamental pivot in the optimization strategy. The assistant abandoned the hope that EAGLE-3 could be a universal improvement and instead began investigating dynamic speculation disable — a mechanism to automatically turn off speculation when concurrency exceeds a threshold. But even this proved challenging, as the standard EAGLEWorker (v1) had deeply coupled batch state management that made dynamic disable nearly impossible without significant refactoring. The assistant eventually pivoted to the spec_v2 overlap path (EAGLEWorkerV2), which had cleaner separation of concerns but required topk=1, reducing the draft tree from 16 tokens to just 3.

In the broader narrative of this optimization effort, message 5435 is the turning point. It is the moment when empirical evidence overruled intuition, when the carefully constructed edifice of speculative decoding optimization crumbled under the weight of real-world benchmarking. The assistant would spend the remainder of the session grappling with the implications of this result, trying to salvage something from the wreckage — but the fundamental truth was now undeniable: for this model, on this hardware, under concurrent load, speculation was a net loss.

Conclusion

Message 5435 appears, on its surface, to be a routine benchmark execution — one bash command among hundreds in a long conversation. But it represents a critical juncture in the optimization process: the moment when a hypothesis was tested and found to be false. The assistant's methodical approach — killing the old server, starting the baseline, verifying health, and running the benchmark — exemplifies good engineering practice. The willingness to let data override prior beliefs, even after hours of optimization work, is a hallmark of rigorous experimentation. And the pivot to dynamic speculation disable, while ultimately unsuccessful in this session, shows the adaptive thinking that characterizes effective technical work.

The lesson is clear: in systems optimization, single-stream benchmarks can mislead. The true test of any optimization is under realistic concurrent load, and the only way to know is to measure.