The Crossroads of Speculative Decoding: Evaluating SGLang vs. vLLM for EAGLE-3 on Kimi-K2.5

In the high-stakes world of large language model inference, few moments are as tense as the one captured in message 3077 of this opencode session. The assistant stands at a crossroads, having just received the results of a comprehensive research task into SGLang's EAGLE-3 speculative decoding support, while simultaneously watching the fourth attempt to load the same model under vLLM slowly grind through 64 checkpoint shards. This message is a synthesis—a moment of strategic evaluation where months of engineering effort (building an EAGLE-3 training pipeline from scratch, generating synthetic data, finetuning drafters, and patching vLLM's source code) must be weighed against the promise of a cleaner, more compatible alternative.

The Context That Led to This Message

To understand why message 3077 was written, we must trace the events that led to it. The session had been on a long and arduous journey to deploy speculative decoding for the Kimi-K2.5 INT4 model—a 1-trillion-parameter Mixture-of-Experts (MoE) language model running on 8x NVIDIA RTX PRO 6000 Blackwell (SM120) GPUs. The team had successfully built a complete EAGLE-3 training pipeline: synthetic data generation from 10,000 inference traces, hidden state extraction at 3,165 tokens per second producing 828 GB of training data, and a 5-epoch finetune from the AQ-MedAI/Kimi-K2-Instruct-eagle3 checkpoint that completed in 2.6 hours.

But then came the wall. When testing the trained drafter with vLLM's EAGLE-3 integration, the assistant had to apply three separate monkey patches just to get the model to load: adding DeepSeek V3 to the model whitelist, fixing image token handling, and implementing the SupportsEagle3 interface on both the DeepseekV3 and KimiK25ForConditionalGeneration classes. Even after all that effort, the acceptance rate was a dismal ~15%, yielding a throughput of just 0.66x—worse than running without speculation at all. The pre-trained AQ-MedAI baseline drafter showed the same behavior, confirming the issue was not training quality but a fundamental integration problem with vLLM's handling of Multi-Head Latent Attention (MLA) hidden state extraction during decode.

The user's response was blunt and decisive (see [msg 3063]): "Vllm is dead; Also consider sglang if eagle3 support there is significantly better." This directive triggered a parallel research task ([msg 3074]) to investigate SGLang's EAGLE-3 capabilities, the results of which arrived just as the assistant was launching vLLM attempt 4 with a critical syntax error fix applied.

What the Message Actually Says

The message opens with a concise situation summary that distills the research findings into actionable intelligence:

SGLang EAGLE-3 support is significantly better than vLLM for our use case: - SGLang has first-class EAGLE-3 support — it's their recommended spec decode method - AQ-MedAI/Kimi-K2-Instruct-eagle3 (the base we finetuned from) is explicitly tested with SGLang and gets ~1.8x throughput - SGLang supports custom EAGLE-3 drafters via --speculative-draft-model-path - vLLM requires 3+ monkey patches just to load (model whitelist, image token, SupportsEagle3 interface on both DeepseekV3 and KimiK25 classes), and we're on attempt 4

The assistant then honestly lists the caveats that temper the optimism:

However, there are caveats for SGLang: - MoE models are harder to tune for spec decode (may need lower batch sizes) - SM120/Blackwell support in SGLang is unknown (we've been running vLLM specifically because it worked on SM120) - Our model is Kimi-K2.5 (not K2), and the drafter was trained for K2.5 specifically

The message concludes with a pragmatic dual-track action: checking whether vLLM attempt 4 succeeds while simultaneously probing whether SGLang is even installed on the system. The bash command reveals the answer: "sglang not installed."

The Reasoning and Decision-Making Process

What makes this message fascinating is the thinking process it reveals. The assistant is performing a multi-dimensional trade-off analysis under uncertainty. On one axis is integration effort: vLLM has consumed enormous engineering time—three patches, four loading attempts, and the discovery of a fundamental MLA integration issue that may not be fixable without deep vLLM internals work. SGLang, by contrast, promises first-class support with no patches required. On another axis is performance certainty: the AQ-MedAI drafter is explicitly tested with SGLang at 1.8x throughput, while vLLM delivers a counterproductive 0.66x. On a third axis is hardware compatibility: the team chose vLLM specifically because it worked on SM120 Blackwell GPUs, and SGLang's support for this architecture is an unknown.

The assistant is also managing risk through parallelism. Rather than committing fully to either path, it keeps vLLM attempt 4 running while investigating SGLang. This is a classic strategy in systems engineering: never put all your eggs in one basket when both baskets are fragile. The message implicitly acknowledges that vLLM attempt 4 might still fail (as previous attempts did), and having SGLang as a fallback is essential.

Assumptions and Their Validity

Several assumptions underpin this message. The first is that SGLang's EAGLE-3 support will actually work for the Kimi-K2.5 architecture without patches. This assumption is reasonable—SGLang explicitly lists DeepSeek V3 and Kimi-K2 as supported models for EAGLE-3—but it is untested. The second assumption is that SM120 compatibility is the primary risk factor. This turns out to be correct, as later in the segment SGLang loads the model in 22 seconds (compared to vLLM's 25 minutes) but then deadlocks on SM120. The third assumption is that the drafter trained for K2.5 will work with SGLang's EAGLE-3 implementation. Since SGLang uses the same drafter interface as the Hugging Face ecosystem from which the training pipeline was derived, this is a reasonable but unverified assumption.

One implicit assumption worth examining is that the vLLM path is worth continuing at all. The assistant has already invested enormous effort in patching vLLM, and the sunk cost fallacy could easily justify continuing. The message resists this by honestly presenting the comparison: "vLLM requires 3+ monkey patches just to load... and we're on attempt 4." The contrast with SGLang's "first-class EAGLE-3 support" is stark.

Input Knowledge Required

To fully understand this message, one needs knowledge of several domains. First, the EAGLE-3 speculative decoding architecture: how draft models predict multiple future tokens in parallel, how acceptance rates determine throughput gains, and why MLA attention creates integration challenges. Second, the vLLM and SGLang inference engine ecosystems: their architecture, model registration systems, and extension points. Third, the SM120/Blackwell GPU architecture and why it introduces compatibility issues—NVIDIA's Blackwell generation uses a different compute architecture (SM120 vs. the SM90 in Hopper) that requires specific kernel support. Fourth, the Kimi-K2.5 model architecture: its MoE structure, MLA attention, and the relationship between the base model and the EAGLE-3 drafter.

Output Knowledge Created

This message creates several pieces of actionable knowledge. It establishes a clear comparison matrix between vLLM and SGLang for this specific use case, documenting the integration cost, performance expectations, and risk factors for each. It confirms that SGLang is not currently installed, creating an installation task. It sets up the decision framework that will guide the next several hours of work: attempt to get vLLM working (attempt 4 is loading), but prepare SGLang as the primary alternative. The message also implicitly documents the team's learning about the limitations of vLLM's EAGLE-3 integration with non-standard model architectures—a finding that could inform future infrastructure decisions.

The Broader Significance

Message 3077 is a microcosm of the challenges faced when deploying cutting-edge AI research in production. The gap between "this model supports EAGLE-3" and "this model supports EAGLE-3 with your specific hardware, your specific model variant, and your specific drafter checkpoint" can be enormous. The assistant's methodical approach—research, compare, test in parallel, document assumptions—is a template for how to navigate this uncertainty. The message also reveals a crucial truth about open-source AI infrastructure: the difference between "supported" and "works" is often measured in hours of debugging, patches, and false starts.

The message ends with the system in a state of productive tension. vLLM attempt 4 is loading weights, SGLang research is complete, and the next steps are clear. Whether the path forward leads through vLLM's patched internals or SGLang's first-class support remains to be seen, but the foundation for that decision has been laid with clarity and honesty.