The Verdict: Baseline 92.7 tok/s vs EAGLE-3 80.9 tok/s — A Benchmark That Changed the Trajectory

Introduction

In the long arc of optimizing speculative decoding for the GLM-5-NVFP4 model on 8× Blackwell GPUs, message <msg id=5583> appears at first glance as a routine benchmark execution: a bash command running a parallel throughput test against a baseline server. But this message is anything but routine. It represents the culmination of a multi-session effort spanning environment setup, CUDA upgrades, NCCL tuning, FlashInfer fusion, and a failed attempt at dynamic speculation disable. More importantly, it delivers the definitive quantitative verdict that reshapes the entire project's direction: the baseline server — running pure Kimi K2.5 without any speculative decoding — strictly outperforms the EAGLE-3 speculative decoding server at every concurrency level tested.

This article examines message <msg id=5583> in depth: why it was written, the decisions that led to it, the assumptions it validates and invalidates, and the knowledge it creates for the project going forward.

The Message: What It Contains

The message is a single tool call — a bash command that executes the benchmark_parallel.py script against a server at http://localhost:30000 with --max-tokens 512 and concurrency levels [1, 2, 5, 10, 30, 70, 100, 250]. The output shows the benchmark framework warming up with 5 requests, then proceeding through the concurrency sweep:

C=  1 | 30 requests | throughput:    92.7 tok/s | per-req:   92.7 tok/s | latency avg/p50/p99:    5.5/   5.5/   5.5s | ok: 30/30
C=  2 | 30 requests | throughput:   157.7 tok/s | per-req:   78.8 tok/s | latency avg/p50/p99:    6.5/   6.5/   6.6s | ok: 30/30
C=  5 | 30 requests | throughput:   290.8 tok/s | pe...

The output is truncated in the conversation (the pe... indicates the line was cut off), but from the chunk summary and subsequent messages we know the full sweep shows the baseline saturating at approximately 773 tok/s total throughput, compared to EAGLE-3's saturation at approximately 354 tok/s — a gap of over 2× at high concurrency.

The server configuration for this baseline run, visible in the preceding messages, includes: --model-path /shared/kimi-k2.5-int4 --tp 8 --trust-remote-code --cuda-graph-max-bs 128 --disable-custom-all-reduce --attention-backend flashinfer --enable-flashinfer-allreduce-fusion. Notably absent are any speculative decoding flags — this is pure, unadorned inference.

Why This Message Was Written: The Reasoning and Motivation

To understand why message <msg id=5583> exists, we must trace the reasoning chain back through the preceding dozen messages. The assistant had just spent a significant effort attempting to implement a dynamic speculation disable mechanism — a feature that would automatically switch between EAGLE-3 speculative decoding and normal decoding based on server load. The idea was elegant: at low concurrency, speculation provides per-request latency benefits; at high concurrency, speculation becomes a throughput liability, so the server should fall back to baseline mode.

This dynamic disable effort failed. The failure was not due to a simple bug but to fundamental architectural coupling in the EAGLE v1 worker path. As the assistant discovered through a series of debugging steps (messages <msg id=5555> through <msg id=5573>), the speculative batch state is deeply intertwined with the CUDA graph runner's shape expectations. The out_cache_loc tensor, pre-allocated for draft token dimensions (e.g., 160 slots for a batch of 10 with 16 draft tokens), cannot simply be replaced with a 2-slot tensor for normal decode. The prepare_for_decode method in the scheduler returns early for speculative batches, leaving out_cache_loc uninitialized for the normal decode path. The EAGLE worker handles its own decode setup, including input_ids, out_cache_loc, and penalties — all of which would need to be replicated in a fallback path.

After tracing through the code and understanding the full complexity, the assistant made a critical decision in message <msg id=5573>:

"Let me take a step back and think about what's actually most valuable. The key insight from the benchmarks is... The dynamic switching is a nice-to-have but the fundamental insight is already clear. Let me just run the baseline with the new coding prompts to get a fair apples-to-apples comparison, then document everything."

This decision represents a pivot from engineering (building a feature) to science (establishing facts). The assistant recognized that the dynamic disable mechanism, while desirable, was premature without first having a complete and fair comparison between EAGLE-3 and baseline performance. Message <msg id=5583> is the direct result of this pivot — it is the baseline benchmark that completes the comparison.

The Decisions Made in This Message

While message <msg id=5583> itself is "just" a benchmark execution, several decisions are embedded in its parameters:

  1. Using coding/agentic prompts: The benchmark uses prompts that match the EAGLE-3 drafter's training data. This was a deliberate change from earlier benchmarks that used encyclopedic prompts. The assistant had updated the benchmark prompts (visible in the chunk summary) to ensure the comparison was fair — if the drafter was trained on coding/agentic data, testing it on unrelated prompts would be an unfair evaluation.
  2. The concurrency sweep: The choice of concurrency levels [1, 2, 5, 10, 30, 70, 100, 250] is designed to capture the full range of behavior from single-user latency to saturation throughput. The fine granularity at low concurrency (1, 2, 5) helps identify the crossover point where speculation stops being beneficial, while the high-end values (100, 250) measure peak throughput.
  3. 30 requests per level: Using 30 requests per concurrency level provides statistical significance while keeping total benchmark runtime manageable.
  4. Identical server configuration: The baseline server uses the same core settings as the EAGLE-3 server (flashinfer attention, allreduce fusion, CUDA 13, cuda-graph-max-bs 128) — only the speculative decoding flags differ. This ensures the comparison isolates the effect of speculation.

Assumptions Made by the Assistant

Several assumptions underpin this message:

  1. The coding prompts are representative: The assistant assumes that coding/agentic prompts are a fair test for the EAGLE-3 drafter, which was trained on similar data. This assumption is validated by the results — the EAGLE-3 numbers with coding prompts (80.9 tok/s at C=1) are similar to the old encyclopedic prompts (77.5 tok/s), suggesting the drafter's performance is not highly sensitive to prompt style.
  2. The benchmark methodology is sound: The assistant assumes that running 30 requests per concurrency level with 512 max tokens provides a reliable measurement. The low variance in latency (p50 and p99 are nearly identical at C=1) supports this assumption.
  3. The baseline represents optimal performance: The assistant assumes that the baseline server configuration (flashinfer + allreduce fusion, CUDA 13, no custom all-reduce) is a reasonable optimal point for comparison. This is supported by the extensive optimization work in prior segments (segments 34-36) that established these settings as best practice.
  4. The comparison is fair: By running both servers on the same hardware, with the same model, the same prompt set, and the same core configuration, the assistant assumes that any performance difference is attributable to the speculative decoding mechanism itself.

Mistakes and Incorrect Assumptions

The most significant incorrect assumption — one that the assistant held before this benchmark — was that EAGLE-3 speculation might provide a throughput advantage at low concurrency. The earlier benchmarks (message <msg id=5577>) showed EAGLE-3 achieving 80.9 tok/s at C=1, which was already lower than the baseline's 92.7 tok/s revealed in this message. The assistant had previously believed that EAGLE-3's value lay in per-request latency at low concurrency, but this benchmark shows that even that narrow advantage does not exist: the baseline is faster at C=1 on both total throughput (92.7 vs 80.9 tok/s) and per-request latency (5.5s vs 6.3s average).

Another assumption that proved incorrect was the viability of the dynamic speculation disable approach. The assistant spent considerable effort (messages <msg id=5555> through <msg id=5573>) attempting to implement this feature on the EAGLE v1 worker path, only to discover fundamental state coupling issues. The assumption was that the speculative and non-speculative paths could be cleanly separated at the batch level, but the codebase's architecture proved otherwise — out_cache_loc allocation, CUDA graph shape expectations, and the EAGLE worker's self-contained decode setup all conspired against a clean separation.

Input Knowledge Required to Understand This Message

To fully grasp the significance of message <msg id=5583>, one needs:

  1. The EAGLE-3 speculative decoding architecture: Understanding that EAGLE-3 uses a draft model to predict multiple tokens per step, which are then verified by the target model. The key parameters are speculative-num-steps 2, speculative-eagle-topk 4, and speculative-num-draft-tokens 16, which together create a tree of up to 16 candidate tokens per forward pass.
  2. The prior optimization work: The assistant had spent segments 34-36 optimizing the verify step — enabling FlashInfer allreduce fusion, upgrading to CUDA 13, patching SGLang for SM120 support, and enabling Torch symmetric memory. These optimizations transformed EAGLE-3 from a net-negative 54.1 tok/s to a net-positive 96.1 tok/s at single-stream. The baseline in this message benefits from the same optimizations.
  3. The failed dynamic disable effort: The immediately preceding messages document the assistant's attempt to implement automatic speculation disabling, which failed due to state management complexity. This context explains why the assistant pivoted to running this benchmark instead of continuing to develop the feature.
  4. The PCIe Gen5 multi-GPU topology: The 8× RTX PRO 6000 Blackwell GPUs are connected via PCIe Gen5, which introduces communication overhead for tensor parallelism. This topology is the reason --disable-custom-all-reduce is used — Blackwell GPUs on PCIe cannot use NVLink-based custom all-reduce.
  5. The benchmark tool: The benchmark_parallel.py script measures total throughput (aggregate tokens/second across all requests) and per-request throughput, at varying concurrency levels. The distinction between total throughput and per-request throughput is critical for understanding the tradeoffs.

Output Knowledge Created by This Message

Message <msg id=5583> creates several pieces of actionable knowledge:

  1. Definitive baseline throughput curve: The project now has a complete throughput-vs-concurrency curve for the optimized baseline, serving as the reference point for all future optimization work.
  2. Quantified cost of speculation: The comparison shows that EAGLE-3 speculation costs approximately 2× total throughput at high concurrency (773 vs 354 tok/s), with no compensating benefit even at low concurrency.
  3. Validation of the pivot away from dynamic disable: The benchmark confirms that the dynamic disable mechanism, had it been successfully implemented, would have been valuable — but the results also show that the baseline is strictly better, meaning the "disable" case is the one you'd always want.
  4. A clear project direction: The assistant's subsequent actions (message <msg id=5584>: "Excellent! Now I have both datasets... Let me compile the final comparison") show that this benchmark completes the data needed for a go/no-go decision on EAGLE-3 speculation. The path forward shifts from optimizing speculation to either abandoning it or finding fundamentally different approaches (such as the spec_v2 overlap path mentioned in the chunk summary).
  5. Documentation update: The assistant updates the eagle-fast-verify.md document with these results (message <msg id=5586>), preserving the knowledge for the project.

The Thinking Process Visible in the Reasoning

The assistant's thinking process in the lead-up to this message reveals a mature engineering approach. When the dynamic disable implementation hit fundamental obstacles, the assistant did not double down or hack around the issues. Instead, it performed a meta-level assessment:

"Let me take a step back and think about what's actually most valuable."

This is a hallmark of effective technical work — recognizing when a subgoal (dynamic disable) has become a distraction from the primary goal (understanding and optimizing the system). The assistant correctly identified that the most valuable thing to do was to complete the comparison dataset, not to continue fighting with state management.

The assistant also showed disciplined experimental methodology. After the EAGLE-3 benchmark with coding prompts (message <msg id=5577>), it noted the similarity to old prompts and immediately pivoted to running the baseline with the same prompts — ensuring the comparison would be apples-to-apples. The cleanup steps (messages <msg id=5574>, <msg id=5579>) — killing zombie processes, restoring patched files, verifying GPU memory is freed — demonstrate attention to experimental hygiene.

Conclusion

Message <msg id=5583> is a turning point in the project. It delivers the quantitative evidence that EAGLE-3 speculative decoding, despite extensive optimization, is strictly inferior to baseline inference on this hardware configuration. The baseline achieves 92.7 tok/s at C=1 compared to EAGLE-3's 80.9 tok/s, and the gap widens to over 2× at high concurrency. This knowledge reshapes the project's priorities: rather than continuing to optimize speculative decoding, the effort shifts toward understanding whether speculation has any role in this deployment, or whether the project should pursue alternative approaches like the spec_v2 overlap path or simply accept baseline performance.

The message also exemplifies good engineering practice: when a feature implementation hits fundamental obstacles, step back, establish the facts, and let data guide the next decision. The benchmark in message <msg id=5583> is not just a number — it is a decision-making tool that saves the project from investing further effort in a losing proposition.