The Benchmark That Ended an Era: EAGLE-3's Definitive Verdict

Introduction

In the long arc of optimizing speculative decoding for the GLM-5-NVFP4 model on eight Blackwell GPUs, message [msg 5577] represents a quiet but decisive turning point. It is, on its surface, a simple execution of a parallel throughput benchmark — a bash command firing off a Python script against a freshly launched EAGLE-3 server. But beneath this mundane surface lies the culmination of dozens of prior messages spanning multiple sessions: CUDA stack upgrades, NCCL tuning, FlashInfer fusion enablement, CUDA graph optimization, and a failed attempt at dynamic speculation disable. This message is where the assistant finally steps back from the complexity of further optimization and asks the fundamental question: after all this work, how good is EAGLE-3, really?

The answer, delivered in the benchmark output, would reshape the entire trajectory of the session.

The Message: What Was Said

The assistant wrote:

Now run the full parallel benchmark with coding prompts on EAGLE-3: [bash] ssh root@10.1.230.174 '~/ml-env/bin/python3 /tmp/benchmark_parallel.py --server-url http://localhost:30000 --max-tokens 512 --concurrency 1 2 5 10 30 70 100 250'

The benchmark script then produced its output:

C= 1 | 30 requests | throughput: 80.9 tok/s | per-req: 81.3 tok/s | latency avg/p50/p99: 6.3/ 6.4/ 7.2s | ok: 30/30 C= 2 | 30 requests | throughput: 128.3 tok/s | per-req: 65.0 tok/s | latency avg/p50/p99: 7.9/ 8.0/ 9.3s | ok: 30/30 C= 5 | 30 requests | throughput: 189.2 tok/s | pe...

The output is truncated in the conversation data, but from the subsequent message ([msg 5578]) we learn the full picture: the coding prompts yielded results "very similar to the old encyclopedic prompts (77.5→80.9 at C=1, ~340→354 saturated)." The saturated throughput at high concurrency was approximately 354 tok/s, compared to the baseline (no speculation) server's ~773 tok/s ([msg 5583]).

Why This Message Was Written: The Reasoning and Motivation

To understand why this message exists, one must trace the path that led to it. The assistant had just spent several messages ([msg 5549] through [msg 5573]) attempting to implement a dynamic speculation disable mechanism — a feature that would automatically switch between EAGLE-3 speculative decoding and plain target decoding based on server load. The idea was elegant: at low concurrency, speculation improves per-request latency; at high concurrency, speculation becomes a bottleneck. Why not have both?

But the implementation ran into fundamental obstacles. The EAGLE v1 worker's batch state management was deeply coupled: out_cache_loc was pre-allocated for draft token dimensions (e.g., 160 slots for a batch of 10 requests with 16 draft tokens each), CUDA graph runners expected speculative-shaped inputs, and the scheduler's prepare_for_decode method returned early for speculative batches without allocating the normal decode structures. Every attempt to patch around this coupling — clearing spec_info, restoring out_cache_loc, re-running alloc_for_decode — hit a new error: RuntimeError: The size of tensor a (160) must match the size of tensor b (2). The state was simply too entangled.

After several iterations of patching, testing, crashing, and debugging ([msg 5555] through [msg 5573]), the assistant made a crucial decision visible in the thinking content of [msg 5573]:

"Let me take a step back and think about what's actually most valuable. The key insight from the benchmarks is already clear... The dynamic switching is a nice-to-have but the fundamental insight is already clear."

This is the moment of strategic pivoting. The assistant recognized that the dynamic disable feature, while valuable, was not the highest-leverage activity. The core insight — that EAGLE-3 helps at low concurrency but hurts at high concurrency — was already established. What was missing was a clean, apples-to-apples comparison using the right prompts.

Thus, message [msg 5577] was born from a deliberate decision to stop fighting with state management and instead produce definitive data. The assistant restored the clean eagle_worker.py ([msg 5574]), started a fresh EAGLE-3 server without any dynamic disable patches ([msg 5575]), waited 550 seconds for it to load ([msg 5576]), and then ran the benchmark.

How Decisions Were Made

Several decisions crystallize in this message:

Decision 1: Use coding/agentic prompts instead of encyclopedic prompts. Earlier benchmarks in the session had used generic "encyclopedic" prompts — factual questions about history, science, and general knowledge. The assistant had previously updated the benchmark script to use coding/agentic prompts ([msg 5569] context), reasoning that these would better match the EAGLE-3 drafter's training data. The EAGLE-3 draft model was trained on 100k steps of synthetic data from the Kimi K2.5 model, which itself was likely fine-tuned on coding and agentic tasks. Using prompts from the same distribution should, in theory, maximize the drafter's acceptance rate and show EAGLE-3 at its best. This was a fair-test assumption: give the speculative decoder the kind of input it was designed for.

Decision 2: Use the same benchmark parameters as the baseline. The concurrency levels (1, 2, 5, 10, 30, 70, 100, 250), max tokens (512), and number of requests per level (30) were kept identical to the baseline benchmark. This was essential for a valid comparison.

Decision 3: Run the benchmark immediately after server startup. The server had been running for only about 550 seconds — enough time to load the model weights and warm up, but not enough for CUDA graph caching to fully stabilize. This is a subtle but important methodological choice: the assistant was measuring "cold-ish" throughput, which is arguably more representative of real-world usage where servers handle bursts of traffic rather than running indefinitely.

Decision 4: Accept the failed dynamic disable approach and pivot. The most important decision was the meta-level one: to stop trying to fix the dynamic disable and instead gather clean data. This reflects a mature engineering judgment — recognizing when a feature branch has diminishing returns and when it's better to consolidate what you know.

Assumptions Made by the Assistant

Several assumptions underpin this message:

Assumption 1: Coding prompts would produce meaningfully different results from encyclopedic prompts. The assistant hypothesized that because the EAGLE-3 drafter was trained on synthetic data from Kimi K2.5 (a coding/agentic model), coding prompts would yield higher acceptance rates and thus better throughput. As it turned out, the difference was marginal: 80.9 tok/s vs 77.5 tok/s at C=1, and ~354 tok/s vs ~340 tok/s saturated. The assumption was not wrong — there was a slight improvement — but it was not the dramatic uplift one might hope for. This suggests that the drafter's performance is relatively robust to prompt distribution, or that the 100k-step training was sufficient to generalize beyond coding-specific patterns.

Assumption 2: The benchmark script (/tmp/benchmark_parallel.py) was correctly configured and would produce reliable results. The assistant had used this script before and trusted its implementation. The script sends concurrent requests, measures throughput and latency, and reports statistics. There is an implicit assumption that the script's measurement methodology (e.g., how it handles request timing, whether it accounts for connection overhead) is sound.

Assumption 3: The server configuration was optimal for EAGLE-3. The server was launched with --speculative-num-steps 2 --speculative-eagle-topk 4 --speculative-num-draft-tokens 16, which had been determined in earlier optimization work ([msg 5536]) to be the best configuration. The assistant assumed that no further tuning would significantly change the results.

Assumption 4: The hardware was stable. Running on 8× RTX PRO 6000 Blackwell GPUs with PCIe Gen5 interconnect, the assistant assumed no thermal throttling, no memory errors, and consistent performance across runs. The benchmark was run on a remote machine accessed via SSH, adding network latency as a potential confound (though the benchmark script runs on the server itself, connecting to the local SGLang server).

Mistakes or Incorrect Assumptions

The coding prompt uplift was smaller than expected. While not exactly a mistake, the assumption that coding prompts would dramatically improve EAGLE-3's performance proved optimistic. The ~4% improvement at C=1 and ~4% improvement at saturation were real but modest. This could indicate that the drafter's acceptance rate is already high for general text, or that the bottleneck is not acceptance rate but the overhead of the speculative pipeline itself (the verify step, NCCL all-reduce for draft distributions, etc.).

The benchmark may not have captured the full warm-up state. CUDA graphs in SGLang are compiled on first use and cached for subsequent batches of the same shape. A server that has only handled a few warm-up requests may not have compiled graphs for all batch sizes. The benchmark's first few concurrency levels (especially C=1 and C=2) might have included CUDA graph compilation latency, slightly understating performance at those levels. However, since each level ran 30 requests, the effect was likely amortized.

The truncation of the output is itself notable. The conversation data shows the output cuts off at "C= 5 | 30 requests | throughput: 189.2 tok/s | pe..." — the remaining concurrency levels (10, 30, 70, 100, 250) are not shown in this message. The assistant had to infer the full results from the next message's summary. This is a limitation of the conversation capture, not an error in the assistant's reasoning, but it means the message as recorded does not contain the complete picture.

Input Knowledge Required

To fully understand this message, a reader needs:

  1. The architecture of speculative decoding in SGLang. EAGLE-3 is a draft-then-verify scheme where a smaller draft model generates candidate tokens, and the target model verifies them in parallel. The key parameters — num_steps=2, topk=4, num_draft_tokens=16 — define the draft tree structure: at each of 2 steps, the draft model generates 4 candidates, for a total tree of up to 16 tokens (though the actual number depends on tree pruning).
  2. The hardware context. Eight RTX PRO 6000 Blackwell GPUs connected via PCIe Gen5, using tensor parallelism (TP=8). This means each forward pass shards the model across all GPUs, with NCCL all-reduce operations for every layer. The PCIe interconnect is a known bottleneck for all-reduce.
  3. The optimization history. The assistant had previously: upgraded CUDA from 12.8 to 13.0 to unlock Blackwell-native features; patched SGLang for SM120 (Blackwell architecture) support; enabled FlashInfer allreduce fusion (which fuses the all-reduce with the attention kernel); enabled Torch symmetric memory; tuned NCCL algorithms and protocols; and optimized CUDA graph batch sizes.
  4. The failed dynamic disable attempt. Understanding why the assistant is running a "pure" EAGLE-3 benchmark (without dynamic disable) requires knowing that the dynamic disable patches failed due to deeply coupled batch state management.
  5. The benchmark methodology. The benchmark_parallel.py script sends concurrent requests at specified concurrency levels, measures end-to-end latency and throughput, and reports aggregate statistics. The --max-tokens 512 parameter limits the generation length per request.

Output Knowledge Created

This message produces several pieces of knowledge:

  1. EAGLE-3 per-request latency at low concurrency: ~81 tok/s at C=1. This is the best-case scenario for speculative decoding — a single user gets 81 tokens per second, with ~6.3 second latency for a 512-token generation.
  2. EAGLE-3 throughput scaling under load. The throughput increases with concurrency but saturates at approximately 354 tok/s (from [msg 5578]). This is a 4.4× improvement over single-request throughput, but the scaling is sublinear: doubling concurrency from 1 to 2 only yields 1.58× throughput improvement (128.3/80.9), and further increases show diminishing returns.
  3. Coding prompts yield marginal improvement over encyclopedic prompts. The ~4% uplift suggests the drafter generalizes well but doesn't have a strong domain advantage.
  4. A definitive comparison point for the baseline. In the subsequent messages ([msg 5583]), the baseline server achieves 92.7 tok/s at C=1 and saturates at ~773 tok/s — more than double EAGLE-3's saturated throughput. This makes the trade-off stark: EAGLE-3 offers slightly better per-request latency at C=1 (81 vs 93 tok/s... wait, actually baseline is faster at C=1 too: 92.7 vs 80.9). This is a critical finding — even at single-user concurrency, the baseline outperforms EAGLE-3. Wait, let me re-check. From [msg 5577]: EAGLE-3 at C=1 gives 80.9 tok/s. From [msg 5583]: baseline at C=1 gives 92.7 tok/s. So the baseline is actually faster at single-user concurrency too. This contradicts the earlier narrative that EAGLE-3 helps at low concurrency. The earlier benchmarks (with encyclopedic prompts and potentially different server configurations) showed EAGLE-3 at 77.5 tok/s vs baseline at some other value. But with the optimized baseline (FlashInfer fusion, CUDA 13, Torch symmetric memory, optimized cuda-graph-max-bs), the baseline is simply faster across the board. This is the devastating finding that this message sets up: after all the optimization work, EAGLE-3 is not just worse at high concurrency — it's worse at every concurrency level.

The Thinking Process Visible in the Reasoning

The assistant's reasoning in this message is spare — it's a simple command execution. But the thinking is visible in what came immediately before. In [msg 5573], the assistant explicitly reasoned:

"Let me take a step back and think about what's actually most valuable. The key insight from the benchmarks is already clear... The dynamic switching is a nice-to-have but the fundamental insight is already clear."

This meta-cognitive step — recognizing when to stop optimizing and start measuring — is the real intellectual content behind this message. The assistant could have continued down the dynamic disable rabbit hole, patching more edge cases, debugging more state mismatches. Instead, it chose to:

  1. Restore the clean worker code ([msg 5574])
  2. Start a fresh server without patches ([msg 5575])
  3. Wait for it to load ([msg 5576])
  4. Run the definitive benchmark (this message) The todo list in [msg 5573] shows the assistant's structured thinking: it had already completed "Kill current EAGLE-3 server and zombie processes" and "Start baseline server (no speculation) with flashinfer + fusion" and "Run parallel benchmark at C=1,2,5,10,30,70,100,250 on baseline (old prompts)" — these were todos from earlier in the session that had been completed. The current task was to run the EAGLE-3 benchmark with coding prompts to complete the comparison. The assistant's choice of words — "Now run the full parallel benchmark with coding prompts on EAGLE-3" — is telling. The word "full" signals that this is the comprehensive test, the one that will settle the question. After days of optimization, patching, tuning, and debugging, this single benchmark run would determine whether all that work had made EAGLE-3 competitive.

Conclusion

Message [msg 5577] is, in essence, a moment of reckoning. It represents the assistant's decision to stop optimizing and start measuring — to let the data speak rather than chasing further improvements. The results, when combined with the baseline benchmark that follows, would reveal that EAGLE-3 speculative decoding, despite all the optimization effort, was simply not competitive with the optimized baseline on this hardware. The baseline was faster at every concurrency level, from single-user to saturation.

This message is a masterclass in engineering discipline: knowing when a feature branch has diminishing returns, stepping back to gather clean data, and letting the results guide the next decision. The assistant would go on to pivot to the spec_v2 overlap path ([msg 5588]), but the fundamental insight — that EAGLE-3's value proposition was much narrower than hoped — was established here, in the quiet output of a benchmark script running on a remote server.