The EAGLE-3 Integration Odyssey: From Training Triumph to Framework Pivot on Blackwell GPUs

Introduction

The deployment of speculative decoding for a 1-trillion-parameter language model across eight NVIDIA Blackwell GPUs is not a single engineering task — it is a saga. This article synthesizes the work captured in segment 23 of an opencode coding session, spanning the completion of an EAGLE-3 training pipeline, the surgical patching of vLLM's internals, the discovery of a fundamental integration failure, a decisive pivot to SGLang, the building of custom CUDA kernels for the SM120 architecture, and the beginning of a new debugging campaign. The narrative arc encapsulates everything that makes frontier ML infrastructure work simultaneously exhilarating and exhausting: the triumph of building something that works, the frustration of discovering it doesn't work well enough, and the resilience required to start over with a new framework.

The EAGLE-3 Training Pipeline: A Complete Success

Before the integration troubles began, the assistant achieved something remarkable: the complete EAGLE-3 training pipeline for Kimi-K2.5 ran to successful completion. This was no small feat. The pipeline required three major stages, each demanding careful engineering.

Stage 1: Fixing Synthetic Data Generation

The journey began with a critical bug in the synthetic data generation script (01b_generate_synthetic.py). The script was designed to query the Kimi-K2.5 INT4 model via vLLM's OpenAI-compatible API, capturing both the model's internal reasoning chain and its final response for use as training data. However, the assistant discovered that the reasoning field was being captured incorrectly.

The root cause was subtle but consequential. vLLM's API with --reasoning-parser kimi_k2 returns reasoning in a field called reasoning on the message object, accessible as msg.reasoning in the OpenAI Python client. But the script was checking for msg.reasoning_content — an attribute that doesn't exist in the OpenAI client's response object, silently returning None. This meant that 388 samples had been generated with empty reasoning fields, rendering them useless for training ([msg 2928]).

The fix required two changes. First, the attribute access was corrected from reasoning_content to reasoning. Second, and more critically, the tokenization reassembly logic needed to wrap the reasoning content with the special <think> and </think> tokens (token IDs 163606 and 163607) that Kimi-K2.5 uses as structural markers in its generation stream ([msg 2932]). Without these tokens, the training data would not faithfully represent the model's actual output distribution, and the drafter would learn incorrect transition probabilities between reasoning and content.

The assistant discarded the 388 corrupted samples and prepared to restart from scratch with a 10,000-sample target. Before launching, it performed a comprehensive "state of the union" documentation ([msg 2928]) — a 3,500-word artifact cataloging everything learned about the hardware, software, model architecture, API quirks, and operational procedures. This document would prove invaluable as the project grew in complexity.

Stage 2: The 10K Inference Run

The 10,000-sample inference run launched successfully and completed in approximately 5.3 hours with zero errors and 100% reasoning capture ([msg 2943]). The assistant monitored the run across multiple checkpoints, validating early outputs to confirm the reasoning fix was working. A single JSON validation at message 2947 confirmed that sample 5115 contained 360 tokens of coherent reasoning and 164 tokens of final answer — exactly the structure the training pipeline needed.

Stage 3: Hidden State Extraction

With the synthetic data in hand, the assistant proceeded to hidden state extraction — the most computationally expensive phase. The extraction script (02_extract_hidden_states.py) loaded the full 547 GB model across all 8 GPUs with tensor parallelism, then ran prefill-only inference on every training sample while capturing hidden states from four specific layers (layers 2, 30, 58, and 60).

The extraction ran at 3,165 tokens per second, processing the full dataset in approximately 91 minutes (plus 24.5 minutes for model loading). The output was staggering: 828 GB of FP16 hidden state tensors across 10,000 samples, representing 17.3 million tokens' worth of intermediate activations ([msg 2977]). The assistant monitored this process across ten separate checkpoints, watching the file count grow from 344 to 10,000 and the disk usage swell from 28 GB to 828 GB.

Stage 4: Vocabulary Mapping and Fine-Tuning

A vocabulary mapping step (03_build_vocab_mapping.py) established the correspondence between the verifier's 163,840-token vocabulary and the drafter's smaller 32,000-token vocabulary, achieving 98.3% token frequency coverage ([msg 2955]).

The fine-tuning itself launched from the AQ-MedAI K2 drafter checkpoint — a pre-trained EAGLE-3 drafter for the DeepSeek V3 architecture that shares the same underlying MLA mechanism as Kimi-K2.5. The training ran for 5 epochs on 9,000 samples per epoch (1,000 held out for validation), completing in exactly 155.2 minutes — 2.6 hours, almost precisely matching the assistant's estimate ([msg 2993]). The final checkpoint at /data/eagle3/output_10k/4/ contained a 4.5 GB model.safetensors file with the LlamaForCausalLMEagle3 architecture, already configured with a vLLM-compatible config.json.

The pipeline had worked flawlessly. Every component — data generation, extraction, mapping, training — had executed exactly as designed. The assistant had built a complete EAGLE-3 training pipeline from scratch and validated it end-to-end. But the real test was yet to come.

The Integration Crisis: vLLM's Three Barriers

When the assistant attempted to load the trained drafter into vLLM for production speculative decoding, it encountered not one but three separate barriers, each requiring a 25-minute model loading cycle to diagnose and fix.

Barrier 1: The Model Type Whitelist

The first launch attempt crashed immediately with a ValueError. vLLM's SpeculativeConfig class maintains a hardcoded whitelist of supported model types for EAGLE-3: ['llama', 'qwen', 'minicpm', 'gpt_oss', 'hunyuan_vl', 'hunyuan_v1_dense', 'afmoe']. Kimi-K2.5's model_type='kimi_k2' was not on this list ([msg 3005]).

The assistant patched the whitelist by adding "kimi_k2" and "deepseek_v3" to the array in the speculative config module, then cleared the Python bytecode cache and relaunched ([msg 3009]). The server progressed past the configuration validation stage, showing workers starting successfully — a promising sign.

Barrier 2: The Missing Image Token Index

After 25 minutes of model loading, the second attempt crashed with a new error: 'KimiK25Config' object has no attribute 'image_token_index'. The EAGLE-3 drafter loading code tried to access target_model.config.image_token_index, but Kimi-K2.5 uses a different multimodal configuration structure with media_placeholder_token_id (value 163605) instead ([msg 3021]).

The assistant inspected the source code, identified the branching pattern used by other multimodal models (Qwen3, Pixtral, etc.), and added a conditional branch for KimiK25ForConditionalGeneration that maps media_placeholder_token_id to image_token_index ([msg 3025]). After clearing caches and relaunching, the server progressed further than ever before.

Barrier 3: The Protocol Interface Gap

The third attempt revealed a fundamentally deeper problem. After the model loaded and the drafter initialized, vLLM crashed with: Model does not support EAGLE3 interface but aux_hidden_state_outputs was requested ([msg 3031]).

This error was not a configuration mismatch or a missing attribute — it was a statement about architectural capability. vLLM's EAGLE-3 implementation requires the target model to implement a Python protocol interface called SupportsEagle3, which mandates that the model can output auxiliary hidden states from intermediate layers during its forward pass. The DeepSeek V2 model class (which serves as the base for both DeepSeek V3 and Kimi-K2.5) only implemented SupportsEagle — the older interface for basic Eagle-style speculation that does not require intermediate hidden state extraction.

The assistant traced the error through vLLM's source code, discovering that supports_eagle3() performs a runtime isinstance check against the SupportsEagle3 protocol class ([msg 3036]). It studied how Qwen2 implements the interface ([msg 3037]), then examined the DeepSeek V2 model's forward method to understand where hidden state collection could be injected ([msg 3044]). The required changes were invasive: adding SupportsEagle3 to the class inheritance chain, implementing set_aux_hidden_state_layers() and get_eagle3_aux_hidden_state_layers() methods, and modifying the forward loop to collect hidden states at specified layer indices.

But then came the critical discovery. The assistant realized that Kimi-K2.5 uses a wrapper architecture: the outer KimiK25ForConditionalGeneration class contains an inner DeepseekV3ForCausalLM as self.language_model. And vLLM's gpu_model_runner.py calls supports_eagle3(self.get_model()) — where get_model() returns the outer wrapper, not the inner language model. This meant the patches to DeepseekV2ForCausalLM were necessary but insufficient. The outer wrapper also needed to implement SupportsEagle3 with delegation methods that forward calls to the inner model.

The assistant wrote a second patch for the Kimi-K2.5 model file, adding SupportsEagle3 to the class definition and implementing two thin delegation methods:

def set_aux_hidden_state_layers(self, layers: tuple[int, ...]) -> None:
    self.language_model.set_aux_hidden_state_layers(layers)

def get_eagle3_aux_hidden_state_layers(self) -> tuple[int, ...]:
    return self.language_model.get_eagle3_aux_hidden_state_layers()

This is the software equivalent of wiring a new switch plate onto an existing circuit — the complexity lives in the inner model, and the wrapper just needs to expose the same interface.

The Syntax Error That Killed a 547GB Server

With both patches applied, the assistant launched the vLLM server again. The server died instantly — within seconds, before even beginning to load the 547GB model weights. The log showed a traceback in multiproc_executor.py and core.py, but the root cause was buried in multiprocess worker output.

The assistant spent several messages methodically filtering log noise — excluding harmless Triton kernel import warnings, FutureWarning deprecation notices, and EngineCore_DP0 secondary errors — to isolate the real cause. The breakthrough came when the assistant pivoted from log forensics to source-code inspection, hypothesizing that the crash was an import-time error in the recently patched deepseek_v2.py.

The diagnosis was confirmed at message 3068: a SyntaxError at line 84, caused by a trailing comma in the import patch. The previous patch had mangled the import into an invalid form:

from .interfaces import MixtureOfExperts, SupportsEagle,
    SupportsEagle3, SupportsLoRA, SupportsPP

This is not valid Python. A multi-line import statement requires either parentheses (explicit continuation) or a backslash. Without either, the newline terminates the statement, and the indented continuation line is a syntax error.

The correct fix, applied in message 3070, wrapped the import in parentheses:

from .interfaces import (MixtureOfExperts, SupportsEagle, SupportsEagle3,
                          SupportsLoRA, SupportsPP)

This single character change — adding an opening parenthesis — unblocked the entire pipeline. The server loaded successfully.

The Devastating Discovery: 15% Acceptance Rate

When the assistant finally got EAGLE-3 speculative decoding running and measured the acceptance rate, the results were catastrophic. Both the newly trained drafter and the pre-trained AQ-MedAI baseline achieved only approximately 15% acceptance rate, resulting in 0.66× throughput — worse than running without speculation at all.

This was not a training quality issue. The fact that the pre-trained baseline showed identical behavior confirmed that the problem was architectural, baked into how vLLM's EAGLE-3 implementation interacted with Multi-Head Latent Attention (MLA), the attention mechanism used by DeepSeek V3 and Kimi-K2.5. The hidden state extraction that worked perfectly during training (at 3,165 tok/s) failed during inference because the draft model could not accurately predict the MLA-compressed hidden states.

The assistant ran a diagnostic test with the AQ-MedAI baseline drafter to confirm the finding. The results were unambiguous: the baseline achieved the same ~15% acceptance rate and 0.66x throughput. The problem was not the drafter — it was the integration.

The Pivot: "vLLM is Dead"

At message 3063, the user delivered a decisive verdict: "Vllm is dead; Also consider sglang if eagle3 support there is significantly better." This was not merely a status update — it was a strategic pivot directive. The user had observed the mounting complexity of the vLLM patches, the repeated crashes, and the fundamental throughput problem, and recognized that the vLLM path was a dead end.

The assistant's response was immediate and productive. Rather than continuing to fight with vLLM, the assistant pivoted to research SGLang's EAGLE-3 support. The findings were encouraging: SGLang has first-class EAGLE-3 support, lists it as the recommended speculative decoding method, and explicitly tests with Kimi-K2 drafters from AQ-MedAI. The assistant updated the todo list to formalize the transition, cleaned up the vLLM environment, and began building the SGLang deployment.## Building sgl-kernel for SM120: The Blackwell Compatibility Challenge

The pivot to SGLang was not without its own challenges. The Blackwell GPUs use the SM120 architecture, and SGLang's kernel compilation system needed to be built specifically for this target. The assistant discovered that the installed SGLang did not have SM120-compatible kernels and embarked on a multi-hour build process.

The build encountered several obstacles. First, the assistant discovered a pre-existing SGLang installation that was incomplete. Then came the SM120 feasibility check, which revealed that SGLang's build system could target SM120 but required building from source. The build failed initially due to missing dependencies — scikit-build-core was not installed. Then the build itself failed with cryptic errors, traced to a missing NUMA library.

The NUMA dependency was a particularly instructive moment. The build error pointed to a missing numa.h header, which is provided by the libnuma-dev package on Ubuntu. This package costs nothing and installs in seconds, but its absence blocked a multi-GPU kernel build for a 1-trillion-parameter model deployment. The assistant installed it with a single apt-get install command and the build proceeded.

The full sgl-kernel build took 48 minutes, compiling CUDA kernels for the SM120 architecture under memory constraints. But when the build completed, the assistant discovered that the compiled kernels were not being detected — a "phantom build" that produced nothing visible. This turned out to be an "editable install" trap: the package had been installed in development mode, so the build artifacts went to a different location than expected.

After diagnosing the architecture detection code and verifying the shared library paths, the assistant confirmed that the SM120 kernels were actually present and functional. The build had succeeded — it was just the discovery mechanism that needed adjustment.

The 22-Second Miracle and the Silent Deadlock

With the kernels built, the assistant launched SGLang with the Kimi-K2.5 model. The result was astonishing: SGLang loaded the 547GB model in just 22 seconds ([msg 3150]), compared to vLLM's 25 minutes — a 68× speedup. This was a genuine triumph, validating the pivot and demonstrating SGLang's superior model loading architecture. The difference stems from SGLang's avoidance of NCCL initialization overhead during loading — each worker loads its own shard independently without the collective synchronization that dominates vLLM's startup time.

But the triumph was short-lived. After loading, the server processes deadlocked with zero CPU and GPU utilization. The server was alive — the model was in GPU memory — but no requests could be processed. The assistant launched an extensive debugging campaign, checking process states, GPU memory allocation, CUDA graphs, and verbose logging.

The deadlock was specific to the SM120 architecture. The assistant tried disabling CUDA graphs, enabling verbose logging, and adjusting server configurations, but the deadlock persisted. The assistant isolated the problem to the interaction between SGLang's initialization sequence and the Blackwell GPU's SM120 architecture — a compatibility issue at the intersection of cutting-edge hardware and a rapidly evolving inference framework.

The breakthrough came when the assistant discovered that the deadlock was related to the trtllm_mla attention backend being selected by default for DeepSeek V3 models. On SM100 (Hopper), trtllm_mla is the optimal backend, but on SM120 (Blackwell), it appears to hang during initialization. The fix was to explicitly specify --attention-backend flashinfer to avoid the SM100-specific kernel path. With this change, the server loaded and served successfully.

Benchmarking SGLang: 90 tok/s Single-Stream

With SGLang serving successfully, the assistant conducted a comprehensive benchmark campaign. The results were impressive:

| Configuration | Single-Stream (tok/s) | Peak Throughput (tok/s) | |---|---|---| | vLLM baseline | 82.5 | 1,536 (C=128) | | SGLang base (default) | 63.6 | — | | SGLang tuned (NCCL + decode steps) | 90.0 | 2,370 (C=128) |

The tuning that unlocked SGLang's performance was a combination of NCCL environment variables and server parameters:

SGLang EAGLE-3: A Tale of Two Drafters

With base serving working, the assistant tested SGLang's EAGLE-3 speculative decoding with two drafters: the pre-trained AQ-MedAI baseline (trained for Kimi-K2, not K2.5) and the custom-trained drafter from the vLLM pipeline.

The AQ-MedAI drafter achieved an acceptance rate of approximately 40-48% — significantly better than vLLM's 15%, but still below the expected 60-80% for well-tuned EAGLE-3. The single-stream throughput was roughly even with baseline (~63 tok/s), but at high concurrency, the speculative decoding was actually worse: 849 tok/s vs 2,370 tok/s for base SGLang.

The custom-trained K2.5 drafter performed even worse, achieving only ~25% acceptance rate. This confirmed that the drafter trained on vLLM-extracted hidden states was fundamentally incompatible with SGLang's inference path. The INT4 dequantization path in vLLM produces different hidden state values than SGLang's, meaning the drafter had learned to predict a distribution that didn't match the serving environment.

The throughput bottleneck was further exacerbated by SGLang's automatic reduction of max_running_requests from 2048 to just 48 when speculative decoding was enabled — a safety limit that dramatically constrained concurrency.

The Hidden State Dump Patch: Extracting Compatible Data

The solution was clear: extract hidden states using SGLang itself, so the training data would match the serving environment. The assistant designed a novel approach: a non-invasive patch to SGLang's DeepseekV2Model.forward() that dumps hidden states to /dev/shm/sglang_hs/req_N/ during prefill, controlled by an environment variable (SGLANG_HS_DUMP_DIR).

The patch was elegantly minimal — approximately 20 lines added to the model's __init__ and forward methods. It captured hidden states at layers [3, 31, 59] (the +1 offset accounts for a difference in indexing convention between vLLM and SGLang) and saved them as individual .pt files alongside a meta.json with token counts and layer information.

The patch worked perfectly on the first try. A test with a 13-token prompt produced correctly shaped tensors: [13, 7168] for each of the three auxiliary layers plus the final hidden state. The extraction script was updated to track dump counters and handle the server-side naming convention.

The SGLang Extraction Pipeline

With the dump patch verified, the assistant launched the full extraction pipeline. The old 828 GB of vLLM-extracted hidden states were deleted to free disk space, and the SGLang server was restarted with the dump patch active, CUDA graphs disabled, and radix cache disabled to ensure every token was processed.

The extraction ran at approximately 1.5 samples per second, with each sample producing about 97 MB of hidden state data. At this rate, the full 10,000-sample dataset would complete in roughly 105 minutes and produce approximately 970 GB of data — comparable to the earlier vLLM extraction but now guaranteed to be compatible with SGLang's inference path.

The assistant monitored the extraction progress across multiple checkpoints, verifying that the dump files had correct shapes and that the counter was incrementing properly. The extraction completed successfully, and the assistant then launched a fresh training run — this time training from scratch (not finetuning from AQ-MedAI) on the SGLang-compatible hidden states.

The Debugging Philosophy: Methodical Investigation Under Pressure

What unifies this entire segment — from the training pipeline through the vLLM patches to the SGLang pivot and deadlock debugging — is a consistent debugging philosophy. The assistant never guesses. Every assumption is verified against source code. Every error is traced to its root cause. Every fix is verified before proceeding.

This is visible in the grep commands that trace get_model() through vLLM's codebase, in the sed invocations that read specific line ranges of critical files, in the systematic filtering of log noise to find buried errors, and in the methodical isolation of the SM120 deadlock. The assistant treats the codebase as a primary source of truth, reading it directly rather than relying on documentation or assumptions.

The approach is particularly valuable when working with complex, multi-layered inference engines where model loading involves registries, wrappers, inheritance hierarchies, and conditional dispatch. A single incorrect assumption about object identity can waste hours of debugging time. By verifying each link in the chain, the assistant ensures that its understanding of the system is grounded in code, not speculation.

Themes and Lessons

Several themes emerge from this segment that are broadly applicable to ML infrastructure engineering:

The fragility of runtime patching. Modifying installed Python packages at runtime is a common practice in ML engineering, but it creates a maintenance burden — every framework update risks overwriting the patches, and the patches themselves are fragile, depending on exact code structure and anchor points. The syntax error at line 84 of deepseek_v2.py is a perfect example: a single misplaced comma in an import statement brought down an 8-GPU server.

The hidden complexity of model wrappers. The Kimi-K2.5 model is not a single monolithic class but a hierarchy: a multimodal wrapper containing a language model, which itself contains transformer layers, MoE modules, and attention mechanisms. Adding a new capability like EAGLE-3 requires understanding this entire hierarchy and ensuring that every layer in the delegation chain is properly patched.

The importance of knowing when to pivot. The assistant invested significant effort in patching vLLM for EAGLE-3 support — writing comprehensive patches, debugging syntax errors, and verifying the integration. But when the benchmark revealed a fundamental 15% acceptance rate wall, the assistant pivoted decisively to SGLang. The user's intervention at message 3063 was critical, but the assistant's willingness to abandon invested work and start fresh with a new framework was equally important.

The bleeding edge is sharp. Building sgl-kernel for SM120, debugging NUMA dependencies, and diagnosing silent deadlocks on Blackwell GPUs — these are not problems that have well-documented solutions. The assistant was operating at the frontier of both hardware (Blackwell SM120) and software (SGLang's EAGLE-3 support), where every problem is a new discovery.

Data compatibility across frameworks is a hidden tax. The discovery that vLLM-extracted hidden states were incompatible with SGLang's inference path — despite both frameworks loading the same model weights — is a sobering reminder that "the same model" can produce different intermediate representations depending on the serving framework. This has profound implications for any workflow that involves training on one framework and deploying on another.

Conclusion

Segment 23 of this opencode session captures a complete arc of ML infrastructure work: from the triumph of a successfully trained EAGLE-3 drafter, through the pain of integration failures and syntax errors, to the strategic pivot to SGLang, the building of custom CUDA kernels, and the beginning of a new extraction and training pipeline. The assistant demonstrated deep knowledge of vLLM's internals, Python's import grammar, CUDA kernel compilation, and distributed system debugging — all in service of deploying a single 1-trillion-parameter model on eight cutting-edge GPUs.

The story is not over. The SGLang-based training pipeline was still running at the end of this segment, and the ultimate test — whether the SGLang-compatible drafter achieves a useful acceptance rate — remains to be determined. But the foundation has been laid: SGLang serves the model at 90 tok/s (surpassing vLLM), the hidden state extraction patch works correctly, and the training pipeline has been validated end-to-end.

In the end, this segment is a testament to the reality of frontier ML engineering: success is not a single event but a continuous process of building, testing, failing, learning, and pivoting. The assistant's systematic approach — diagnose, study, craft, verify, pivot — is a model for how to navigate this landscape, even when the path leads through dead ends and deadlocks. The 68× improvement in model loading time from vLLM to SGLang, the successful SM120 kernel compilation, and the elegant hidden state dump patch are all genuine engineering achievements that will serve as foundations for whatever comes next.