The Third Attempt: Launching SGLang with EAGLE-3 Speculative Decoding

In the middle of a high-stakes optimization campaign for the Kimi-K2.5 INT4 model running on 8x NVIDIA Blackwell GPUs, a single bash command represents the culmination of a delicate debugging and patching effort. Message <msg id=3192> is deceptively simple — it launches an SGLang inference server with EAGLE-3 speculative decoding enabled — but the reasoning behind it reveals the intricate interplay between model architecture, inference engine compatibility, and the relentless pursuit of throughput.

The Context: A Bottleneck Hunt

The broader session had been a deep investigation into the inference performance of Kimi-K2.5, a Mixture-of-Experts (MoE) model built on a DeepSeek V2/V3 architecture. Earlier profiling (Segment 19) had identified AllReduce communication as the dominant bottleneck, consuming 51.5% of decode time. With hardware-level optimizations exhausted — the team was already using NVLink, CUDA graphs, and the Blackwell SM120 architecture — the assistant turned to software-only optimizations. Speculative decoding emerged as the most promising path: instead of generating one token at a time with the large target model, a small "drafter" model proposes several candidate tokens, and the target model verifies them in parallel. If the drafter is accurate, this yields significant speedups without quality degradation.

The assistant had already built a complete EAGLE-3 training pipeline (Segments 20–23), trained a custom drafter on 10,000 samples of Kimi-K2.5's own outputs, and attempted to integrate it with vLLM. That integration failed — the acceptance rate was only ~15%, yielding a net throughput regression (0.66×) rather than speedup. The pivot to SGLang was motivated by its different attention backend (triton) and potentially better EAGLE-3 support. But SGLang brought its own challenges.

The Problem: A Missing Interface Method

The immediate trigger for message <msg id=3192> was a crash. The assistant's first attempt to launch SGLang with EAGLE-3 (at <msg id=3178>) had failed because KimiK25ForConditionalGeneration — the model class in SGLang's codebase — did not implement the set_eagle3_layers_to_capture method. This method is part of SGLang's EAGLE-3 interface: it tells the target model which intermediate hidden states to capture during the forward pass, so the drafter can use them as conditioning signals.

The assistant diagnosed this crash in <msg id=3180> by searching SGLang's source code for references to the missing method. It found that DeepseekV2ForCausalLM (the parent class of the language model component) already had a complete implementation at line 2963 of deepseek_v2.py. The problem was that KimiK25ForConditionalGeneration wraps DeepseekV3ForCausalLM via self.language_model but never delegates the EAGLE-3 call. This is a common pattern in multi-modal model wrappers: the vision components are handled separately, and the language model's interface methods must be explicitly forwarded.

The Patch: A Five-Line Delegation

The fix was straightforward but required careful understanding of the codebase. The assistant examined how mllama4.py handled the same pattern (a multi-modal model delegating to its language model), found a simple delegation:

def set_eagle3_layers_to_capture(self, layer_ids: Optional[List[int]] = None):
    if hasattr(self.language_model, "set_eagle3_layers_to_capture"):
        self.language_model.set_eagle3_layers_to_capture(layer_ids)

This five-line method was injected into KimiK25ForConditionalGeneration just before the load_weights method. The assistant verified the patch succeeded by searching for "eagle3" in the modified file, confirming lines 739–742 contained the new delegation.

But this patch was not the only thing needed. The assistant also had to verify that the forward pass would correctly capture and return hidden states. By examining deepseek_v2.py lines 2712–2778, it confirmed that DeepseekV2Model already had logic to capture hidden states at specified layers and return them as aux_hidden_states. The set_eagle3_layers_to_capture method simply sets self.capture_aux_hidden_states = True and populates self.layers_to_capture with the layer IDs. Everything downstream was already wired.

The Command: What the Flags Reveal

With the patch applied and the Python cache cleared (<msg id=3190>), and all previous processes killed and GPU memory freed (<msg id=3191>), the assistant issued the launch command in <msg id=3192>:

nohup /root/ml-env/bin/python3 -m sglang.launch_server \
  --model-path /shared/kimi-k2.5-int4 \
  --trust-remote-code \
  --tp-size 8 \
  --mem-fraction-static 0.85 \
  --host 0.0.0.0 \
  --port 8000 \
  --speculative-algorithm EAGLE3 \
  --speculative-draft-model-path /data/eagle3/aq-medai-k2-drafter \
  --speculative-num-steps 3 \
  --speculative-eagle-topk 1 \
  --speculative-num-draft-tokens 4 \
  --log-level info \
  > /data/eagle3/synth_10k/sglang_eagle3_aqmedai_v3.log 2>&1 &

Every flag tells a story:

• --speculative-algorithm EAGLE3: Selects the EAGLE-3 algorithm, which uses a lightweight transformer drafter that conditions on intermediate hidden states from the target model. This is distinct from other speculative decoding approaches like Medusa or self-speculation.
• --speculative-draft-model-path /data/eagle3/aq-medai-k2-drafter: Points to a pre-trained drafter model from the AQ-MedAI team, which was trained specifically for Kimi-K2.5. The assistant had verified its config.json in <msg id=3177>, confirming it used LlamaForCausalLMEagle3 architecture with the correct eagle_config including eagle_aux_hidden_state_layer_ids: [2, 30, 58]. These layer IDs — early, middle, and near-final layers — are where the drafter conditions on the target model's hidden states.
• --speculative-num-steps 3: The drafter generates three draft tokens per step. Each step is a forward pass through the small drafter model, which is much cheaper than a pass through the full 547B target model.
• --speculative-eagle-topk 1: At each draft position, only the single most likely token is considered (greedy sampling). Higher values would allow tree-based exploration of multiple candidates, increasing the chance of acceptance but also increasing verification cost.
• --speculative-num-draft-tokens 4: Four draft tokens are generated per verification round. The target model then verifies all four in a single forward pass using a tree attention mask. If all four are accepted, the effective throughput is multiplied by 4.
• --tp-size 8: Tensor parallelism across all 8 GPUs, which is essential for fitting the 547GB model (INT4 quantized) into GPU memory.
• --mem-fraction-static 0.85: Reserves 85% of GPU memory for the model's static allocation, leaving 15% for KV cache and temporary buffers.
• --log-level info: Standard logging; the assistant would later monitor the log file for the "fired up" signal indicating server readiness.
• > /data/eagle3/synth_10k/sglang_eagle3_aqmedai_v3.log 2>&1 &: Output is redirected to a log file. The v3 suffix is telling — this is the third attempt to launch SGLang with EAGLE-3, after the first attempt crashed and the second attempt (if any) was preempted by the patching process.

Assumptions Embedded in the Launch

The command makes several implicit assumptions that deserve scrutiny:

1. The patch is sufficient: The assistant assumes that adding set_eagle3_layers_to_capture delegation is the only missing piece. But SGLang's EAGLE-3 integration might require other methods — get_eagle3_aux_hidden_states for extracting the captured states, or get_embed_and_head / set_embed_and_head for handling the embedding and output projection layers. The assistant had checked the model runner code and confirmed only set_eagle3_layers_to_capture was called, but the full EAGLE-3 pipeline involves more than just setting layer IDs.
2. The AQ-MedAI drafter is compatible: The drafter was presumably trained for use with vLLM or a specific SGLang version. Its config.json shows LlamaForCausalLMEagle3 architecture, but SGLang's EAGLE-3 implementation may have different assumptions about the drafter's forward pass, output format, or integration protocol.
3. CUDA graphs are compatible with speculative decoding: The previous successful base launch used CUDA graphs (which capture GPU operations for replay). Speculative decoding involves dynamic control flow — the number of draft tokens accepted varies per step — which may conflict with CUDA graph capture.
4. The model will load successfully: The 547GB model takes 5–10 minutes to load. The nohup and background execution indicate the assistant expects this wait, but any error during loading would require manual intervention.

The Output: Minimal but Meaningful

The command's output is simply:

Launched SGLang EAGLE-3 v3

This is the echo from the echo command in the SSH session. It confirms the shell command was submitted, but it says nothing about whether the server started successfully. The assistant would need to wait and monitor the log file — a process it had automated in previous attempts with a polling loop that checks for "fired up" in the logs.

The minimal output reflects a key reality of this work: launching a large model server is never instantaneous. The real outcome would only be known minutes later, after weight loading, CUDA graph capture (if applicable), and speculative decoding initialization complete.

What This Message Achieves

Message <msg id=3192> is the culmination of a focused debugging chain spanning messages <msg id=3178> through <msg id=3191>. In that chain, the assistant:

1. Identified the root cause of the crash (missing set_eagle3_layers_to_capture)
2. Located the existing implementation in the parent class
3. Found the correct delegation pattern from another model file
4. Patched kimi_k25.py with surgical precision
5. Cleared cached bytecode to force recompilation
6. Cleaned up GPU state from the previous failed attempt
7. Issued the corrected launch command This is a textbook example of debugging in the large-model serving domain: the error manifests as a crash at startup, but the root cause is a missing interface method in a model wrapper file. The fix requires understanding the model class hierarchy, the inference engine's plugin architecture, and the EAGLE-3 algorithm's requirements.

The Broader Significance

This message sits at a critical juncture in the optimization campaign. The assistant has already established that SGLang with CUDA graphs achieves 63.6 tok/s single-stream and 2,370 tok/s peak throughput — significantly outperforming vLLM's peak of 1,536 tok/s but lagging in single-stream latency (63.6 vs 82.5 tok/s). The EAGLE-3 experiment is the next frontier: can speculative decoding close the single-stream gap or push peak throughput even higher?

The answer, which would emerge in subsequent messages, was disappointing. The AQ-MedAI drafter achieved only ~42% acceptance rate, yielding no net speedup. The custom K2.5-trained drafter fared even worse at 25% acceptance. SGLang's automatic reduction of max_running_requests to 48 in speculative mode further limited throughput. The assistant would ultimately conclude that EAGLE-3 provided no benefit on this hardware and pivot back to tuning SGLang's single-stream performance through NCCL environment variables and attention backend selection.

But that outcome doesn't diminish the importance of message <msg id=3192>. It represents a well-executed debugging and deployment cycle: diagnose, patch, clean, launch. In the fast-paced world of ML inference optimization, where a single flag or missing method can cost hours of debugging, this disciplined approach is the difference between progress and stagnation.