The Inference Runner: Scaling EAGLE-3 Training Data by 10×

In the long arc of debugging and optimizing EAGLE-3 speculative decoding for the Kimi-K2.5 model, message [msg 3698] marks a quiet but crucial inflection point. It is not a dramatic breakthrough — no bug fixed, no performance record set. Instead, it is the moment where the assistant transitions from preparation to production, from gathering raw materials to actually running the pipeline that will generate the training data needed to improve the draft model. This message, in which the assistant reports that all 10 datasets are prepped and then writes an inference runner script, represents the bridge between two phases of the project: the data acquisition phase and the large-scale inference phase.

The Context: Why This Message Was Written

To understand why this message exists, one must understand the debugging ordeal that preceded it. For several rounds, the assistant had been chasing a baffling problem: the EAGLE-3 draft model, despite being trained on real Kimi-K2.5 data, was achieving a zero acceptance rate when deployed on SGLang. Every prediction from the draft model was being rejected by the target model, rendering speculative decoding useless. The root cause, traced in [msg 3696] and surrounding messages, turned out to be a single incorrect flag: the SGLang server was started with --speculative-algorithm EAGLE instead of EAGLE3. The is_eagle3() check in SGLang's codebase is strict — only the exact string EAGLE3 triggers the target model to capture and concatenate intermediate layer hidden states from layers [2, 30, 58]. With EAGLE, the draft model received 7168-dimensional final-layer-only states instead of the expected 21504-dimensional concatenated states. The trained fc fusion layer was silently bypassed, and all the training effort was for nothing.

After fixing the flag, the acceptance rate improved but remained modest — an accept_len of approximately 2.1 tokens, meaning the draft model's predictions were accepted for only about two tokens on average before the target model rejected them. Benchmarking showed the best EAGLE-3 configuration achieved 82.3 tok/s (with CUDA graphs and 5 draft tokens) versus the 90 tok/s non-speculative baseline — still about 9% slower. The EAGLE-3 paper's scaling curves suggested that more training data was the primary lever for improving acceptance rates. The draft model was data-limited, not architecture-limited.

This diagnosis led to a strategic pivot: scale the training dataset by 10×. The assistant dispatched ten parallel agents to search for agentic coding, reasoning, and general chat datasets suitable for fine-tuning Kimi-K2.5. Ten datasets were selected, downloaded, and prepared, totaling 88,088 samples. But there was a catch: only 4,800 of those samples were already in Kimi-K2.5's native token distribution (datasets A1 and A2, which came from DeepSeek and Kimi-K2.5 itself). The remaining 83,288 samples — prompts from datasets like Glaive, OpenCodeInstruct, Magicoder, MixtureThoughts, OpenThoughts, UltraChat, ShareGPT, and SWE-agent — needed to be run through the Kimi-K2.5 model to generate responses that match the target model's token distribution. This is what message [msg 3698] sets in motion.

The Dataset Landscape

The assistant's opening line — "All 10 datasets prepped. 88,088 total (4,800 tokenized + 83,288 prompts needing inference)" — is a status report that encapsulates days of work. The ten datasets are:

• A1_deepswekimi: 2,800 tokenized records — multi-turn agentic conversations from DeepSeek, already in Kimi format
• A2_kimik25: 2,000 tokenized records — native Kimi-K2.5 outputs
• B1_glaive: 10,000 prompts — general instruction-following data
• B2_opencodeinstruct: 14,714 prompts — coding instruction data from a 5M-record scan
• B3_magicoder: 10,000 prompts — code generation data
• B4_mixturethoughts: 10,002 prompts — mixed reasoning traces
• B5_openthoughts: 10,000 prompts — open-domain reasoning
• B6_ultrachat: 15,000 prompts — large-scale chat data
• B7_sharegpt: 10,000 prompts — user-shared ChatGPT conversations
• B8_sweagent: 3,572 prompts — SWE-agent trajectories for long-context agentic coding The "A" datasets (tokenized) are already usable for training. The "B" datasets (prompts) need inference — each prompt must be fed to the Kimi-K2.5 model running on the SGLang server, which generates a response. The response is then combined with the prompt to form a complete training sample. This is necessary because the EAGLE-3 drafter must learn to predict Kimi-K2.5's specific token distribution, not the distribution of the original dataset's model. Using prompts from diverse sources but responses from Kimi-K2.5 ensures the training data is on-distribution for the target model.

The Inference Runner Design

The core of this message is the assistant's decision to write a new script, run_inference.py, by adapting the existing 01b_generate_synthetic.py. The message lays out three requirements:

1. Read from prompts.jsonl files instead of extracting from HF datasets: The existing script extracted prompts directly from HuggingFace dataset objects. The new script must read pre-formatted message lists from JSONL files, which are the output of the preparation pipeline.
2. Handle two partitions: The datasets are split into "B-short" (datasets B1-B7, with context length C=150-200 and max_tokens=10240) and "B-long" (dataset B8, the SWE-agent trajectories, with C=32 and max_tokens=16384). The SWE-agent dataset contains much longer conversations — full trajectories of agentic coding sessions — so it needs more output tokens and fewer concurrent samples to fit in GPU memory.
3. Output tokenized_data.jsonl per dataset: Each dataset's inference results must be written to a tokenized_data.jsonl file in the same format as the already-tokenized A datasets, so the training pipeline can consume them uniformly. The parameter choices reveal the assistant's reasoning about memory and throughput. The "C" parameter (context length multiplier) determines how many samples are batched together. For the short datasets, C=150-200 means the server processes 150-200 prompts concurrently, amortizing the overhead of model loading and attention computation. For the long-context B8 dataset, C=32 is more conservative — each sample is much longer, so fewer fit in the same memory budget. The max_tokens parameter (10240 vs 16384) similarly reflects the expected response length: SWE-agent trajectories require longer generations because they involve multi-step reasoning and code execution traces.

Assumptions and Knowledge

The assistant makes several assumptions in this message. First, it assumes the SGLang server is running and healthy. This is a non-trivial assumption — the server had been crashing and hanging throughout the session, and the assistant had just spent significant effort getting it to load the model reliably on the SM120 GPU architecture. Second, it assumes the prompts.jsonl format is consistent across all eight B datasets. Given that each dataset had unique quirks (the DeepSeek dataset had 84-message conversations, the SWE-agent dataset had trajectory-style records), this is a bet that the preparation script normalized them to a common format. Third, it assumes the existing 01b_generate_synthetic.py is a suitable template — that its API call patterns, error handling, and output formatting are close enough to what's needed that adaptation is faster than writing from scratch.

The knowledge required to understand this message is substantial. One must understand the EAGLE-3 speculative decoding architecture — how the draft model predicts target model hidden states, how the acceptance check works, and why on-distribution training data matters. One must understand the SGLang inference server — its API, its batching mechanism, and the C and max_tokens parameters. One must understand the dataset landscape — the distinction between "tokenized" (already in Kimi format) and "prompts" (need inference), and why the assistant chose these particular ten datasets. And one must understand the broader context of the debugging journey — the flag mismatch, the zero acceptance rate, the pivot from vLLM to SGLang, and the conclusion that more data is the primary lever.

The Thinking Process Visible in the Reasoning

The assistant's reasoning in this message is concise but reveals a structured thought process. The opening line is a status summary that immediately establishes the current state and the next action. The assistant then explicitly lists the three adaptations needed, showing that it has analyzed the gap between the existing code and the new requirements. The partitioning into B-short and B-long with specific parameters shows a memory-conscious design — the assistant is thinking about GPU memory constraints and throughput optimization.

The decision to write a new file rather than modify the existing script suggests the assistant values keeping the original script intact as a reference, or perhaps the adaptations are substantial enough that a fresh file is cleaner. The fact that the assistant writes the file in the same message (using the [write] tool) rather than planning it for a future message shows a bias toward immediate action — once the requirements are clear, the implementation follows without delay.

The LSP errors shown at the end of the message are from a different file (server_args_sm120.py) and are irrelevant to the current task. The assistant includes them because the tool output is appended to the message, but they don't affect the inference runner. This is a reminder of the messy reality of working with a complex codebase — errors in unrelated files are constant background noise.

Conclusion

Message [msg 3698] is a transitional message — it closes the chapter on dataset preparation and opens the chapter on large-scale inference. The 88,088 samples represent a 10× scale-up in training data, driven by the diagnosis that the EAGLE-3 draft model is data-limited. The inference runner script is the engine that will convert 83,288 prompts into on-distribution training samples, completing the dataset for the next round of training. Whether this scale-up will be enough to push the acceptance rate from 2.1 to something that makes speculative decoding worthwhile is an open question — but the assistant has built the machinery to find out.