The Pivot Point: Launching a Custom-Trained EAGLE-3 Drafter for Kimi-K2.5

Introduction

In the high-stakes world of large language model inference optimization, every millisecond counts. Message 3213 captures a pivotal moment in an extended investigation into speculative decoding for the Kimi-K2.5 INT4 model running on eight NVIDIA RTX PRO 6000 Blackwell GPUs. After weeks of building infrastructure, training pipelines, and debugging integration issues, the assistant reaches a critical decision point: having confirmed that the off-the-shelf AQ-MedAI EAGLE-3 drafter yields disappointing results, it now pivots to test the custom-trained drafter that was painstakingly built over the preceding sessions. This single message—a seemingly routine server launch command—embodies the culmination of an entire research arc and the moment of truth for a major optimization hypothesis.

The Message

The assistant issues a single bash command over SSH to the remote machine:

SGLANG_ALLOW_OVERWRITE_LONGER_CONTEXT_LEN=1 nohup /root/ml-env/bin/python3 -m sglang.launch_server \
  --model-path /shared/kimi-k2.5-int4 \
  --trust-remote-code \
  --tp-size 8 \
  --mem-fraction-static 0.85 \
  --host 0.0.0.0 \
  --port 8000 \
  --speculative-algorithm EAGLE3 \
  --speculative-draft-model-path /data/eagle3/output_10k/4 \
  --speculative-num-steps 3 \
  --speculative-eagle-topk 1 \
  --speculative-num-draft-tokens 4 \
  --log-level info \
  > /data/eagle3/synth_10k/sglang_eagle3_custom_v1.log 2>&1 &

The only substantive change from the previous launch is the --speculative-draft-model-path argument, which now points to /data/eagle3/output_10k/4—the custom-trained EAGLE-3 drafter—instead of the AQ-MedAI drafter at /data/eagle3/aq-medai-k2-drafter. The log confirms: "Launched SGLang EAGLE-3 with custom drafter."

Why This Message Was Written: The Reasoning and Motivation

To understand why this message exists, one must trace the investigative arc that led to it. The assistant had been engaged in a multi-session effort to deploy Kimi-K2.5 INT4—a massive 547GB mixture-of-experts model—on an 8-GPU Blackwell server. Initial profiling (Segment 19) revealed that AllReduce communication was the dominant bottleneck, consuming 51.5% of decode time. This led the assistant to explore speculative decoding as a software-only optimization path that could improve throughput without hardware changes.

The investigation progressed through several phases:

  1. Research phase: The assistant evaluated speculative decoding options for Kimi-K2.5, ruling out n-gram speculation as too slow and settling on EAGLE-3 as the most promising approach.
  2. Infrastructure phase: A complete EAGLE-3 training pipeline was built, requiring patching the speculators library for vLLM 0.16 API compatibility, resolving KV cache config mismatches, and fixing custom worker implementations for the DeepseekV2 architecture that Kimi-K2.5 inherits.
  3. Training phase: The pipeline was validated on 10 samples, scaled to 1,000 samples, and ultimately run on 10,000 samples of synthetic data generated by capturing Kimi-K2.5's actual reasoning outputs.
  4. Integration phase: When vLLM's EAGLE-3 integration with MLA attention yielded only ~15% acceptance rate (resulting in 0.66x throughput—worse than no speculation), the assistant pivoted to SGLang, which loaded the model in 22 seconds but initially appeared to hang.
  5. Debugging phase: The "hang" turned out to be a ~5-10 minute model load time for the 547GB model. Once resolved, the assistant benchmarked base SGLang at 63.6 tok/s single-stream and 2,370 tok/s peak throughput—significantly outperforming vLLM's peak of 1,536 tok/s.
  6. AQ-MedAI evaluation: The assistant then tested SGLang with the AQ-MedAI EAGLE-3 drafter, achieving ~42% acceptance rate but no speedup—in fact, peak throughput dropped to 849 tok/s versus 2,370 tok/s for the baseline. The bottleneck was SGLang's automatic max_running_requests=48 limit for speculative mode. Message 3213 represents the critical comparison test: having established that the AQ-MedAI drafter (trained for Kimi-K2, not K2.5) is ineffective, the assistant must now determine whether the custom-trained drafter—trained on actual K2.5 inference data—can deliver the speculative decoding speedup that has been the goal all along.

How Decisions Were Made

The decision to launch with the custom drafter was the result of a systematic elimination process. The assistant had already:

Assumptions Made

Several assumptions underpin this message:

  1. The custom drafter is properly trained: The assistant assumes that the training pipeline completed successfully and produced a functional drafter model. This is a reasonable assumption given that the pipeline was validated end-to-end on 10 samples and scaled to 1,000 samples before the full 10K run, but the quality of the trained drafter remains unknown until inference.
  2. Config compatibility: The assistant assumes that because the config structure matches (both use LlamaForCausalLMEagle3 architecture with similar eagle_config), the custom drafter will load without errors. This is a technical assumption based on the JSON inspection in the previous message.
  3. SGLang's EAGLE-3 implementation is correct: The assistant assumes that the SGLang EAGLE-3 worker will correctly interface with the custom drafter, just as it did with the AQ-MedAI drafter. Given that the AQ-MedAI test succeeded (server fired up, inference worked), this is a reasonable assumption.
  4. The training data alignment is sufficient: The custom drafter was trained on hidden states extracted from the vLLM server, but inference is now running on SGLang. The assistant implicitly assumes that the hidden state distributions are similar enough between the two serving frameworks for the drafter to generalize.
  5. The environment is clean: The assistant assumes that killing all Python processes and waiting for GPU memory to clear (done in the previous message) is sufficient to avoid resource conflicts. Given the explicit cleanup steps, this is well-founded.

Mistakes or Incorrect Assumptions

Several potential issues lurk beneath the surface of this message:

  1. The training/inference framework mismatch: The custom drafter was trained using hidden states extracted from vLLM, but it is now being evaluated on SGLang. These two frameworks use different attention backends, different CUDA graph capture strategies, and potentially different numerical precision paths. The hidden state distributions could differ enough to degrade the drafter's acceptance rate. This is a subtle but potentially critical oversight—the assistant later recognizes this and plans to retrain using SGLang-based extraction.
  2. The 10K sample size may be insufficient: Training an EAGLE-3 drafter typically requires hundreds of thousands to millions of tokens to achieve good acceptance rates. The 10K inference samples, even if each generates hundreds of tokens, may not provide enough diversity for the drafter to generalize well across the full distribution of K2.5's outputs.
  3. The drafter architecture may be suboptimal: The custom drafter uses the same LlamaForCausalLMEagle3 architecture as the AQ-MedAI drafter, with hidden size 7168 and 60 layers. But the optimal drafter architecture for a mixture-of-experts model like Kimi-K2.5 may differ from what works for dense models. The assistant inherits this architecture choice from the training pipeline without re-evaluating it.
  4. No baseline for comparison: The assistant doesn't have a ground-truth acceptance rate target. The AQ-MedAI drafter achieved ~42% on K2.5, but it was trained for K2. The custom drafter should theoretically do better since it was trained on K2.5 data, but "better" is undefined. Without knowing the theoretical maximum acceptance rate for this model/drafter combination, any result is difficult to contextualize.

Input Knowledge Required

To fully understand this message, one needs:

  1. The history of the EAGLE-3 investigation: Knowledge of the preceding sessions—the profiling that identified AllReduce as the bottleneck, the failed vLLM integration, the pivot to SGLang, and the AQ-MedAI evaluation—is essential to understand why this test matters.
  2. Speculative decoding fundamentals: Understanding how EAGLE-3 works—drafting multiple tokens per step, verifying them against the target model, and the trade-off between acceptance rate and overhead—is necessary to interpret the significance of this launch.
  3. SGLang architecture: Familiarity with SGLang's server launch parameters (--tp-size, --speculative-algorithm, --speculative-draft-model-path, etc.) and its CUDA graph capture mechanism helps parse the command.
  4. The hardware context: The 8x RTX PRO 6000 Blackwell GPU configuration with PCIe interconnect explains why AllReduce is a bottleneck and why speculative decoding is being explored as a software-only optimization.
  5. The model architecture: Kimi-K2.5's DeepseekV2-derived MLA (Multi-head Latent Attention) architecture explains why EAGLE-3 integration was challenging—the custom worker patches were necessary to handle the non-standard attention mechanism.

Output Knowledge Created

This message produces several forms of knowledge:

  1. A testable hypothesis: Whether the custom-trained EAGLE-3 drafter can achieve higher acceptance rates than the generic AQ-MedAI drafter on Kimi-K2.5. The result of this test (which the assistant discovers in subsequent messages) will either validate the entire training pipeline or force a fundamental rethinking of the approach.
  2. A benchmark comparison point: The performance of SGLang + custom EAGLE-3 drafter can be compared against both the SGLang baseline (63.6 tok/s single-stream, 2,370 tok/s peak) and the SGLang + AQ-MedAI configuration (~63 tok/s single-stream, 849 tok/s peak). This three-way comparison will reveal whether custom training provides meaningful improvement.
  3. A diagnostic signal: The acceptance rate achieved by the custom drafter will indicate whether the training data quality, quantity, and pipeline are adequate. A rate similar to AQ-MedAI (~42%) would suggest fundamental issues with the training approach, while a significantly higher rate (60-80%) would validate the methodology.
  4. A decision point for future work: The outcome will determine whether the assistant continues down the EAGLE-3 path (perhaps with larger training datasets or SGLang-based extraction) or pivots to other optimization strategies like NCCL tuning for single-stream performance.

The Thinking Process

The assistant's reasoning is visible in the preceding messages and in the structure of this command. The key insight is the recognition that the AQ-MedAI drafter's mediocre acceptance rate is likely due to model mismatch—it was trained for Kimi-K2, not K2.5. The assistant has invested significant effort in building a custom training pipeline specifically for K2.5, and this launch represents the payoff of that investment.

The assistant's thinking follows a clear pattern:

  1. Diagnose the failure mode: The AQ-MedAI drafter's ~42% acceptance rate is insufficient. Why? Because the drafter was trained on a different model's distribution.
  2. Formulate a hypothesis: A drafter trained on K2.5's actual output distribution will achieve higher acceptance rates.
  3. Test the hypothesis: Launch SGLang with the custom drafter and benchmark it.
  4. Prepare for the outcome: Based on the results, the assistant will either declare victory (high acceptance rate, throughput improvement) or pivot to alternative optimization strategies. The assistant also demonstrates a pragmatic engineering mindset: rather than trying to tune hyperparameters or increase max_running_requests for the AQ-MedAI drafter (which would be optimizing a fundamentally flawed approach), they cut directly to testing the custom drafter. This reflects an understanding that acceptance rate is the primary lever for speculative decoding performance—all other optimizations are secondary.

Conclusion

Message 3213 is deceptively simple. On its surface, it is a single bash command launching a server with a different model checkpoint. But in context, it represents the culmination of an extensive research and engineering effort spanning multiple sessions, dozens of tool calls, and countless debugging iterations. The assistant has built a complete EAGLE-3 training pipeline, resolved numerous integration issues, established baseline performance metrics, and now faces the moment of truth: does the custom-trained drafter justify the investment?

The outcome of this test will shape the direction of the entire optimization effort. If the custom drafter achieves high acceptance rates, the path forward involves scaling the training data and refining the pipeline. If it fails, the assistant must pivot to alternative optimization strategies—NCCL tuning, attention backend selection, or entirely different approaches. This single message, for all its brevity, carries the weight of an entire research arc.