The Verdict on Speculative Decoding: When EAGLE-3 Falls Flat Under Load

Introduction

In the long-running optimization saga of deploying the Kimi-K2.5 INT4 model across 8× NVIDIA RTX PRO 6000 Blackwell GPUs, message [msg 5437] represents a watershed moment. After weeks of battling CUDA version incompatibilities, patching SGLang for SM120 support, enabling FlashInfer allreduce fusion, and finally achieving the breakthrough where EAGLE-3 speculative decoding beat the baseline in single-stream throughput (96.1 tok/s vs 92.6 tok/s), the assistant ran the definitive experiment: a head-to-head parallel throughput comparison at multiple concurrency levels. The results were devastating to the EAGLE-3 hypothesis.

This message is the moment of reckoning — the assistant presents a comparison table showing that the baseline server (no speculation) strictly outperforms EAGLE-3 at every concurrency level for total throughput. The gap widens from 19% at C=1 to over 2x at high concurrency. This single table of numbers fundamentally reshapes the project's direction, forcing a pivot away from the EAGLE-3 speculation path that had consumed so much effort, and toward a new strategy of dynamic speculation disable or alternative approaches.

The Message in Full

The assistant writes:

Here's the comparison. The numbers are very revealing:

>

### EAGLE-3 vs Baseline Throughput Comparison (max_tokens=512)

>

| C | EAGLE-3 (tok/s) | Baseline (tok/s) | Winner | Ratio | |---|---|---|---|---| | 1 | 77.5 | 92.6 | Baseline +19% | 0.84x | | 2 | 125.1 | 159.7 | Baseline +28% | 0.78x | | 5 | 183.2 | 292.5 | Baseline +60% | 0.63x | | 10 | 239.6 | 457.2 | Baseline +91% | 0.52x | | 30 | 299.5 | 711.0 | Baseline +137% | 0.42x | | 70 | 337.7 | 784.2 | Baseline +132% | 0.43x | | 100 | 338.8 | 761.6 | Baseline +125% | 0.44x | | 250 | 340.9 | 772.1 | Baseline +126% | 0.44x |

>

Key findings: - Baseline wins at EVERY concurrency level for throughput. - Baseline scales much better: it goes from 92.6 → 784 tok/s (8.5x), while EAGLE-3 only goes from 77.5 → 341 tok/s (4.4x). - Baseline saturates at ~780 tok/s around C=70 (2.3x higher than EAGLE-3's ceiling of ~340). - The gap widens as concurrency increases — at C=30+, baseline is over 2x faster.

>

Wait — but single-stream EAGLE-3 was 96.1 tok/s in our earlier test, and now the parallel benchmark shows 77.5 tok/s at C=1. Let me also recall the single-stream baseline was 92.6. The parallel benchmark C=1 matches baseline perfectly (92.6), but EAGLE-3 C=1 is lower (77.5 vs 96.1). That's because the parallel benchmark uses max_tokens=512 with diverse prompts, while the single-stream benchmark used a fixed prompt. The acceptance rate varies per prompt.

>

The critical insight here: EAGLE-3 speculation only helps per-request latency when there's negligible batching load. But for throughput (total tok/s across all concurrent requests), the baseline always wins because the overhead of draft + verify steps consumes compute that would be better spent processing more requests in parallel.

>

Let me update the results file and then investigate dynamic speculation:

Why This Message Was Written: The Reasoning and Motivation

The assistant wrote this message to fulfill the explicit next step in the optimization plan: compare EAGLE-3 vs baseline throughput at multiple concurrency levels to find the crossover point. The todo list from the previous round (visible in [msg 5436]) shows the assistant had just completed the baseline benchmark and was ready to build the comparison table. The motivation was to answer a specific engineering question: "At what concurrency level does EAGLE-3 stop being beneficial and start being a liability?"

However, the message goes beyond simply reporting numbers. The assistant's reasoning reveals a deeper cognitive process — it is grappling with the implications of the data. The phrase "The numbers are very revealing" signals that the assistant recognizes these results are not just incremental data points but a fundamental challenge to the project's direction. The assistant had invested enormous effort into making EAGLE-3 work: upgrading CUDA from 12.8 to 13, patching SGLang for SM120 support, enabling FlashInfer allreduce fusion, and tuning NCCL parameters. The single-stream victory (96.1 tok/s beating 92.6 tok/s) had validated that effort. But the parallel throughput data now tells a different story — one where the entire speculation strategy is questionable under realistic server loads.

The message also serves a documentation purpose. The assistant updates the eagle-fast-verify.md file with these results (visible in the subsequent [msg 5440]), ensuring the experimental record is complete. This is consistent with the project's meticulous documentation culture — every optimization attempt, whether successful or not, is recorded for future reference.

How Decisions Were Made

This message is primarily analytical rather than decision-making, but it contains several implicit and explicit decisions:

The decision to present the comparison as a table — The assistant chose to format the data as a markdown table with columns for concurrency level, EAGLE-3 throughput, baseline throughput, winner designation, and ratio. This is a deliberate rhetorical choice designed to make the pattern immediately visible: the "Baseline" label appears in every row's "Winner" column. The table structure forces the reader to confront the monotonic dominance of the baseline.

The decision to compute ratios — The assistant calculated EAGLE-3/Baseline ratios (0.84x, 0.78x, 0.63x, etc.), which quantify the degradation. The ratio drops from 0.84x at C=1 to 0.42x at C=30, then stabilizes around 0.44x. This quantifies the "gap widening" phenomenon.

The decision to contextualize the single-stream discrepancy — The assistant notices that the parallel benchmark's C=1 EAGLE-3 result (77.5 tok/s) is lower than the earlier single-stream benchmark (96.1 tok/s). Rather than ignoring this inconsistency, the assistant explicitly addresses it, reasoning that the parallel benchmark uses diverse prompts with max_tokens=512 while the single-stream benchmark used a fixed prompt, causing varying acceptance rates. This intellectual honesty strengthens the analysis.

The decision to articulate the critical insight — The assistant crystallizes the finding into a single sentence: "EAGLE-3 speculation only helps per-request latency when there's negligible batching load." This is the key takeaway that will guide future decisions.

The decision to proceed to dynamic speculation investigation — Despite the discouraging results, the assistant does not abandon the EAGLE-3 path entirely. Instead, it pivots to investigating dynamic speculation disable — a mechanism to automatically switch between EAGLE-3 and baseline modes based on server load. This is the logical next step: if EAGLE-3 helps at low concurrency but hurts at high concurrency, the optimal strategy is to use it selectively.

Assumptions Made

Several assumptions underpin this message, some explicit and some implicit:

Assumption: The benchmark methodology is fair and representative. The assistant assumes that running 30 requests per concurrency level with max_tokens=512 and diverse prompts produces results that generalize to real-world usage. This is a reasonable assumption, but it's worth noting that real-world traffic patterns may differ — request lengths vary, prompt distributions shift, and inter-arrival times matter.

Assumption: The baseline server configuration is optimal. The baseline was started with --cuda-graph-max-bs 128, --attention-backend flashinfer, --enable-flashinfer-allreduce-fusion, and --disable-custom-all-reduce. The assistant assumes this represents the best possible baseline. However, different configurations might yield different baseline numbers, potentially changing the comparison.

Assumption: The EAGLE-3 server configuration is optimal. The EAGLE-3 server used --speculative-num-steps 2, --speculative-eagle-topk 4, --speculative-num-draft-tokens 16, and --mem-fraction-static 0.88. The assistant assumes these parameters are near-optimal based on earlier tuning. Different speculation parameters might shift the crossover point.

Assumption: Throughput is the right metric. The assistant focuses on total throughput (tok/s across all concurrent requests). But the user might also care about per-request latency, tail latency, or time-to-first-token. The message acknowledges this implicitly by noting that EAGLE-3 helps per-request latency at low concurrency.

Assumption: The EAGLE-3 drafter model is well-trained. The drafter was trained on 37K synthetic samples from the K2.5 model. The assistant assumes the drafter's acceptance rate is representative of what can be achieved with this training approach. A better-trained drafter might shift the results.

Mistakes and Incorrect Assumptions

The single-stream discrepancy was not fully resolved. The assistant notes that EAGLE-3 C=1 in the parallel benchmark (77.5 tok/s) is lower than the earlier single-stream benchmark (96.1 tok/s), attributing it to diverse prompts affecting acceptance rates. However, this 19% drop is significant and warrants deeper investigation. Is the acceptance rate truly that sensitive to prompt content? Or is there a systematic issue with the parallel benchmark's measurement methodology? The assistant does not run a follow-up experiment to isolate the cause.

The assumption that EAGLE-3 is net-positive at low concurrency may be premature. At C=1, EAGLE-3 achieves 77.5 tok/s vs baseline's 92.6 tok/s — baseline is 19% faster even at the lowest concurrency. The earlier single-stream benchmark showed EAGLE-3 winning (96.1 vs 92.6), but that used a fixed prompt. With diverse prompts, EAGLE-3 loses even at C=1. This suggests the "crossover point" where EAGLE-3 becomes useful might not exist at all for throughput — it may only exist for per-request latency, and only with favorable prompts.

The message does not compute per-request latency. The comparison focuses entirely on total throughput. But the user's original motivation for EAGLE-3 was likely per-request latency improvement — making individual users feel like the model is responding faster. The assistant could have added a latency comparison table showing the per-request latency at each concurrency level. This would have provided a more complete picture of the trade-off.

The assumption that dynamic speculation disable is feasible. The assistant decides to "investigate dynamic speculation" as the next step, assuming that SGLang's codebase supports or can be modified to support runtime toggling of speculation. As the subsequent messages in the segment reveal ([msg 5441] and beyond), this assumption proves incorrect — the standard EAGLE worker has deeply coupled batch state management that makes dynamic disable extremely difficult, forcing a pivot to the spec_v2 overlap path.

Input Knowledge Required

To fully understand this message, the reader needs:

Knowledge of speculative decoding. Understanding what EAGLE-3 does — using a small "drafter" model to propose multiple draft tokens, then verifying them against the large "target" model in a single forward pass — is essential to interpreting the results. The reader must understand that speculative decoding trades off computation: the draft + verify cycle adds overhead but can generate more accepted tokens per target model invocation.

Knowledge of the hardware setup. The 8× RTX PRO 6000 Blackwell GPUs are connected via PCIe Gen5 without NVLink, making allreduce communication a significant bottleneck. This context explains why FlashInfer allreduce fusion was such an important optimization and why the verify step (which requires allreduce for each of the 61 layers) is expensive.

Knowledge of the earlier optimization journey. The reader needs to know that CUDA was upgraded from 12.8 to 13 specifically to enable FlashInfer allreduce fusion on SM120 (Blackwell), that NCCL tuning was required to prevent baseline collapse, and that the single-stream EAGLE-3 result of 96.1 tok/s was a celebrated victory. This context makes the parallel throughput reversal more impactful.

Knowledge of the benchmark methodology. The benchmark script (benchmark_parallel.py) sends requests concurrently, measures total output tokens across all requests, and reports throughput. The max_tokens=512 parameter limits the generation length. The concurrency levels (C=1,2,5,10,30,70,100,250) span from trivial to extreme load.

Knowledge of the model architecture. Kimi-K2.5 is a Mixture-of-Experts model with 61 layers, each performing 2 allreduces per forward pass. This architectural detail explains why the verify step is communication-bound and why allreduce optimization is critical.

Output Knowledge Created

This message creates several important knowledge artifacts:

The definitive throughput comparison table. This table is the most important output — it provides a clear, quantitative answer to the question "Is EAGLE-3 worth it under load?" The answer is a definitive "no" for throughput, with baseline outperforming EAGLE-3 by 19-137% depending on concurrency.

The insight that EAGLE-3's value is limited to low-concurrency latency. The message articulates a clear boundary condition: EAGLE-3 helps per-request latency when batching load is negligible, but hurts throughput under any significant load. This is a generalizable insight that applies beyond this specific hardware and model configuration.

The observation that EAGLE-3 throughput saturates much earlier than baseline. EAGLE-3 peaks at ~340 tok/s around C=70, while baseline continues scaling to ~780 tok/s. The saturation point is 2.3x higher for baseline. This suggests a fundamental bottleneck in the EAGLE-3 architecture — likely the verify step's allreduce overhead limits how many concurrent requests can be processed.

The documentation update to eagle-fast-verify.md. The assistant appends these results to the optimization plan document (visible in [msg 5440]), ensuring the experimental record is complete and the findings are preserved for future reference.

The motivation for dynamic speculation disable. The message creates the rationale for a new feature: automatically disabling speculation when server load exceeds a threshold. This becomes the project's next major work item.

The Thinking Process Visible in Reasoning

The assistant's reasoning in this message is a masterclass in data-driven engineering analysis. Let me trace the thinking process step by step:

Step 1: Presentation of raw data. The assistant presents the comparison table without commentary, letting the numbers speak first. The table is carefully formatted with clear headers, consistent alignment, and bold highlighting on the "Winner" column to draw the eye.

Step 2: Extraction of key findings. The assistant then extracts four bullet-point findings, each one a higher-level abstraction from the raw data:

Conclusion

Message [msg 5437] is a turning point in the optimization project. The parallel throughput comparison definitively shows that EAGLE-3 speculative decoding, despite its single-stream victory, is a net-negative for throughput under any realistic server load. The baseline server outperforms EAGLE-3 at every concurrency level, with the gap widening from 19% to over 2x as load increases.

The message is notable for its intellectual rigor: the assistant presents data clearly, extracts meaningful patterns, resolves an apparent inconsistency, synthesizes a critical insight, and plans next steps — all in a single message. The thinking process is transparent and data-driven, with no attempt to spin the results favorably. When the data contradicts the assistant's prior efforts and expectations, the assistant accepts the evidence and pivots.

This message also demonstrates the value of comprehensive benchmarking. The single-stream benchmark (96.1 tok/s) gave a misleadingly positive picture of EAGLE-3's value. Only by testing at multiple concurrency levels with diverse prompts did the true trade-off become visible. The lesson is clear: in production ML systems, benchmark under realistic load, not just idealized single-stream conditions.

The project now pivots to dynamic speculation disable — a feature that would automatically switch between EAGLE-3 and baseline modes based on server load. Whether this approach succeeds or fails, the foundation is this message's clear-eyed assessment of where EAGLE-3 adds value and where it doesn't.