The Pivot: When Speculative Decoding Fails on Blackwell GPUs

In the high-stakes world of large language model serving, every millisecond counts. When you have eight NVIDIA RTX PRO 6000 Blackwell GPUs and a 547GB quantized model (Kimi-K2.5 INT4), the difference between a good serving configuration and a great one can mean hundreds of tokens per second in throughput. Message 3219 represents a quiet but significant moment in an intense optimization campaign: the moment when a promising but ultimately failing approach is abandoned in favor of the empirically best configuration.

The Message

The assistant writes:

Now let me restore the best-performing option — SGLang base with CUDA graphs:

>

```bash ssh root@10.1.230.174 'nohup /root/ml-env/bin/python3 -m sglang.launch_server \ --model-path /shared/kimi-k2.5-int4 \ --trust-remote-code \ --tp-size 8 \ --mem-fraction-static 0.85 \ --host 0.0.0.0 \ --port 8000 \ --log-level info \ > /data/eagle3/synth_10k/sglang_base_final.log 2>&1 &' echo "Launched SGLang base (best config)"

At first glance, this looks like a simple server restart. But this message is the culmination of hours of painstaking work: patching model files, training custom draft models, debugging API incompatibilities, and running benchmarks. It represents a data-driven decision to abandon speculative decoding and return to a simpler, faster configuration.

The Context: A Multi-Day EAGLE-3 Campaign

To understand why this message matters, we need to understand what led to it. The assistant had been on a multi-day journey to implement EAGLE-3 speculative decoding for Kimi-K2.5, a Mixture-of-Experts (MoE) model with Multi-head Latent Attention (MLA). The promise of EAGLE-3 is compelling: by using a lightweight "draft" model to predict multiple future tokens in parallel, the main model can verify them in a single forward pass, potentially achieving 2-3x speedup over autoregressive decoding.

The campaign had several phases:

  1. Infrastructure setup: Building the EAGLE-3 training pipeline, patching the speculators library for vLLM 0.16 compatibility, and creating custom worker code for the DeepseekV2-derived architecture (<msg id=3197-3201>).
  2. Training a custom drafter: Generating synthetic training data by capturing Kimi-K2.5's actual hidden states during inference, then fine-tuning an EAGLE-3 draft model on 10,000 samples.
  3. Testing with vLLM: The first attempt at EAGLE-3 inference with vLLM yielded a disastrous ~15% acceptance rate and only 0.66x throughput — slower than baseline (<msg id=3207 context>).
  4. Pivot to SGLang: Switching to SGLang, which loaded the model in 22 seconds but initially appeared to hang. The assistant discovered the "hang" was actually just the server taking 5-10 minutes to load the massive 547GB model across 8 GPUs.
  5. Baseline benchmarking: Before testing EAGLE-3 on SGLang, the assistant established a baseline: SGLang base with CUDA graphs achieved 63.6 tok/s single-stream and an impressive 2,370 tok/s peak throughput at concurrency 128 (<msg id=3207 context>).
  6. AQ-MedAI drafter test: Testing a pre-trained EAGLE-3 drafter from AQ-MedAI (trained for Kimi-K2, not K2.5) showed ~42% acceptance rate — better than vLLM's 15%, but still resulting in lower throughput (849 tok/s vs 2,370 tok/s baseline) due to SGLang's automatic max_running_requests=48 limit for speculative mode ([msg 3210]).
  7. Custom K2.5 drafter test: The custom-trained drafter performed even worse, with a 25% acceptance rate and accept length of exactly 1.0 — meaning it accepted zero draft tokens per step. The drafter was essentially proposing random tokens (<msg id=3217-3218>).

The Decision Point

Message 3219 is the moment of truth. After benchmarking both EAGLE-3 drafters and finding both produced worse throughput than the base SGLang configuration, the assistant makes a clean decision: kill the speculative decoding experiment and restore the best-performing option.

The decision is notable for what it doesn't include. There is no hand-wringing, no "maybe if we tweak X," no attempt to salvage the approach. The assistant has a clear comparison table in mind (presented in message 3218):

| Config | Single-stream tok/s | C=128 tok/s | Accept rate | |--------|---------------------|-------------|-------------| | vLLM baseline | 82.5 | 1,536 | N/A | | SGLang base (CG) | 63.6 | 2,370 | N/A | | SGLang + AQ-MedAI EAGLE-3 | 62.9 | 849 | ~42% | | SGLang + Custom K2.5 EAGLE-3 | 40.8 | 597 | 25% (broken) |

The data is unambiguous. SGLang base with CUDA graphs achieves 2,370 tok/s at C=128, nearly 2.8x the throughput of the best EAGLE-3 configuration. Even single-stream, where speculative decoding should theoretically shine, the base config matches or exceeds the EAGLE-3 variants (63.6 vs 62.9 tok/s).

Why EAGLE-3 Failed

The assistant's analysis in the preceding messages identifies several root causes:

1. Quantization mismatch: The EAGLE-3 drafter was trained on BF16/FP16 hidden states, but the serving model is INT4 quantized. The hidden states from the quantized model differ from those the drafter was trained on, causing the drafter to make poor predictions. This is a fundamental alignment issue between training and inference.

2. Concurrency limitations: SGLang automatically limits max_running_requests to 48 when speculative decoding is enabled, compared to 2048 for the base configuration. This severely constrains throughput under high concurrency, which is where SGLang's batching engine excels.

3. Draft model quality: The AQ-MedAI drafter was trained on Kimi-K2, not K2.5 — different model generations with potentially different behavior patterns. The custom K2.5-trained drafter was apparently broken, likely due to issues in the training data pipeline or hidden state extraction.

4. Overhead outweighs benefit: With ~42% acceptance rate (accepting ~1.65 out of 4 draft tokens), the cost of running the draft model and verifying its predictions exceeds the savings from parallel token generation. For speculative decoding to be worthwhile, acceptance rates typically need to be 70% or higher.

Assumptions and Lessons

The EAGLE-3 campaign was built on several assumptions that proved incorrect:

What Comes Next

The message launches a base SGLang server, but the assistant is not done optimizing. In the subsequent messages (outside this message but within the same segment), the assistant pivots to tuning SGLang's single-stream performance by applying NCCL environment variables (NCCL_PROTO=LL, NCCL_ALGO=Ring, NCCL_P2P_LEVEL=SYS, etc.) and launching with --attention-backend flashinfer --num-continuous-decode-steps 4 --disable-custom-all-reduce. The goal shifts from speculative decoding to matching vLLM's superior single-stream speed (82.5 tok/s vs SGLang's 63.6 tok/s).

The assistant also begins planning a new EAGLE-3 training pipeline using SGLang-based hidden state extraction, hoping that alignment between the extraction and inference frameworks will produce a better drafter. This is a more measured approach: instead of rushing to test, the assistant will first ensure the training data is properly aligned.

Conclusion

Message 3219 is a study in disciplined engineering decision-making. Faced with disappointing results from a sophisticated technique, the assistant does not double down or make excuses. Instead, the assistant:

  1. Collects comprehensive benchmark data across multiple configurations
  2. Analyzes the root causes of failure
  3. Makes a clean, data-driven decision to revert to the best-performing option
  4. Documents the results for future reference This message marks the end of one chapter and the beginning of another. The EAGLE-3 experiment on Kimi-K2.5 with INT4 quantization on Blackwell GPUs has been thoroughly evaluated and found wanting. But the knowledge gained — about quantization alignment, concurrency limits, and the importance of training-inference consistency — will inform future optimization efforts. Sometimes the most important decision an engineer can make is knowing when to abandon a failing approach and return to what works.