The Raw Token Revelation: How One Debugging Session Fixed an EAGLE-3 Training Pipeline

"No parsing at all — prompt_ids + output_ids is the exact token sequence."

In a single, deceptively concise message ([msg 3812]), an AI assistant summarized the resolution of a critical bug that had been silently corrupting an EAGLE-3 speculative decoding training pipeline. The message, just a few paragraphs long, distills hours of investigative work into a clean diagnosis and fix. But behind this summary lies a fascinating detective story about tokenization, server endpoints, and the subtle ways that high-level APIs can destroy the very data fidelity that machine learning training demands.

The Message Itself

The assistant wrote:

Fixed. Here's what was wrong and what changed:

>

Problem: run_inference.py used the OpenAI /v1/chat/completions endpoint. SGLang's reasoning parser wasn't configured, so reasoning content ended up in message.content with thinking stripped and reasoning_content: null. Even with --reasoning-parser kimi_k2, the parser would interfere with tool call tokens too.

>

Fix: Rewrote to use SGLang's /generate endpoint with input_ids directly: - Pre-tokenize prompts via apply_chat_template(add_generation_prompt=True) — this appends thinking (token 163606) - /generate returns raw output_ids — the model's exact tokens including response (163607), tool call tokens (<|tool_calls_section_begin|>, etc.), and <|im_end|> - No parsing at all — prompt_ids + output_ids is the exact token sequence - Switched from openai client to aiohttp for direct HTTP

>

Verified: The output correctly has reasoning tokens → response → content/tool-call tokens, all as raw IDs. Server is running without --reasoning-parser, inference restarted from scratch on B1 at ~1.3 req/s.

This message serves as a status update, a postmortem, and a design document all at once. It was written because the assistant had just completed a significant refactor of the inference pipeline and needed to communicate both what changed and why it was necessary.

The Context: Building Training Data for EAGLE-3

To understand why this message matters, we need to step back. The broader project involved training an EAGLE-3 speculative decoding drafter for the Kimi-K2.5 model — a large language model running across 8 NVIDIA RTX PRO 6000 Blackwell GPUs. EAGLE-3 is a sophisticated technique where a lightweight "drafter" model predicts the base model's hidden states, enabling faster autoregressive generation through speculative decoding.

The training pipeline required generating synthetic data: feeding prompts to the base model and capturing its complete output token sequences, including reasoning traces and tool calls. These sequences would become the training targets for the drafter. Data fidelity was paramount — if the captured tokens didn't exactly match what the model generated, the drafter would learn incorrect patterns and its acceptance rate would suffer.

This is where the bug emerged.

The Problem: An API That Strips Information

The original run_inference.py used the OpenAI-compatible /v1/chat/completions endpoint provided by SGLang. This is the natural choice — it's the standard API, well-documented, and supported by the openai Python client library. But it has a critical flaw for this use case: it was designed for consumption, not training data collection.

The Kimi-K2.5 model uses a reasoning structure where the model first generates a thinking token (163606), then produces reasoning tokens, then a response token (163607), and finally the actual content (which may include tool calls). This structure is fundamental to how the model works. But the OpenAI chat completions API, when used without SGLang's --reasoning-parser flag, treats the entire output as plain message.content — the thinking token gets stripped during decoding, and the reasoning_content field is left null.

Even when the --reasoning-parser kimi_k2 flag was added, the parser would attempt to split reasoning from content, but it would also interfere with tool call tokens like <|tool_calls_section_begin|> and <|tool_call_begin|>. These special tokens are essential for the EAGLE-3 training data — the drafter needs to learn when and how to generate tool calls — but the parser had no concept of them and would mangle the output.

The result was training data where:

  1. The thinking token was missing from the start of generation
  2. The response token was stripped
  3. Tool call tokens were corrupted or lost
  4. The text-based round-trip (decode to string, then re-tokenize) introduced subtle differences from the original token sequence For EAGLE-3 training, which operates entirely in token space, these corruptions were catastrophic. The drafter would be trained on sequences that didn't match what the model actually produced, guaranteeing poor acceptance rates.

The Investigation: Following the Token Trail

The discovery of this bug didn't happen in the message itself — it unfolded across the preceding dozen messages ([msg 3792] through [msg 3811]). The assistant's reasoning process is visible in this chain, and it's a textbook example of systematic debugging.

Step 1: Testing the /generate endpoint. The assistant first tested SGLang's lower-level /generate endpoint, which returns raw output_ids. The initial test (msg 3792) used the text parameter rather than input_ids, and the output showed that the thinking token was missing from output_ids. The text started with "The user is asking..." — the reasoning content was present, but the opening thinking token was gone.

Step 2: Checking token identities. The assistant decoded tokens 163606 and 163607, confirming they mapped to thinking and response respectively (msg 3794). This confirmed the special tokens existed and were recognized by the tokenizer.

Step 3: The key insight — thinking is prompt-side. The critical discovery came in msg 3795. The assistant ran apply_chat_template on a simple user message and found that the template appends thinking (token 163606) as the last token of the prompt. The model never generates thinking — it's part of the input, not the output. This meant that the missing thinking in output_ids wasn't a bug in the /generate endpoint; it was correct behavior. The thinking token belongs to the prompt, and the model's generation starts after it.

This insight changed everything. The correct approach wasn't to try to force the server to include thinking in the output — it was to concatenate prompt_ids + output_ids on the client side.

Step 4: Proving the approach works. The assistant restarted the server without --reasoning-parser and tested the new approach (msg 3803). The results were clean: prompt_ids ended with token 163606 ( thinking), output_ids contained token 163607 ( response) naturally at position 61, and the full sequence prompt_ids + output_ids gave the complete, faithful token sequence.

Step 5: Verifying on real data. After rewriting run_inference.py and restarting inference, the assistant verified the output on actual training samples (msg 3810). The data showed exactly the expected structure: reasoning tokens → response → content with tool calls, all as raw token IDs with no parsing artifacts.

The Fix: A Clean Architectural Change

The fix described in msg 3812 represents a significant architectural shift. The original design used a high-level API (OpenAI chat completions) and relied on text-based serialization (JSON with string fields). The new design uses a low-level API (SGLang /generate) and operates entirely in token ID space.

The key changes were:

  1. Endpoint swap: From /v1/chat/completions to /generate. The latter is designed for programmatic access and returns raw token IDs without any parsing or post-processing.
  2. Client library swap: From the openai Python client to raw aiohttp HTTP requests. This eliminated a layer of abstraction that could introduce assumptions about response format.
  3. Prompt pre-tokenization: Instead of sending text and letting the server tokenize it, the client now calls apply_chat_template locally to get exact prompt_ids, including the appended thinking token. This ensures the prompt boundary is precisely controlled.
  4. Raw token storage: The output is stored as prompt_ids + output_ids — a flat list of integers. No text decoding, no re-tokenization, no parsing. The data is exactly what the model produced.
  5. Server simplification: The --reasoning-parser flag was removed from the server entirely. All reasoning structure is handled client-side by simply looking for token 163607 ( response) in the output_ids.

Assumptions Made and Broken

Several assumptions were challenged during this debugging session:

Assumption: The OpenAI API preserves all output tokens. This was false. The chat completions API is designed for human-readable responses, not for machine learning training data. It strips special tokens and applies post-processing that destroys information.

Assumption: Text-based round-trip is lossless. The original pipeline decoded tokens to text, then re-tokenized for storage. This is lossy for models with special tokens — the re-tokenization may not produce the same token IDs, especially for tool call tokens that the tokenizer may segment differently.

Assumption: thinking is a generation token. The assistant initially assumed that thinking was generated by the model, like response. Discovering that it's actually part of the prompt (appended by the chat template) was the critical insight that unlocked the correct approach.

Assumption: The reasoning parser is harmless when not configured. Even without --reasoning-parser, the OpenAI endpoint still processes the output through its response formatting pipeline, which strips special tokens. The /generate endpoint bypasses this entirely.

Input Knowledge Required

To understand this message, one needs:

Output Knowledge Created

This message creates several pieces of actionable knowledge:

  1. A verified approach for capturing faithful token sequences from SGLang: Pre-tokenize prompts with apply_chat_template, send input_ids to /generate, concatenate prompt_ids + output_ids. This is now a documented pattern that can be reused.
  2. Documentation of Kimi-K2.5's token structure: Token 163606 = thinking (prompt-side), token 163607 = response (generation-side). The thinking is part of the chat template, not model output.
  3. A warning about the OpenAI API's data fidelity: For training data collection, the OpenAI-compatible endpoint is insufficient. The /generate endpoint is required for lossless token capture.
  4. Confirmation that the approach works at scale: The inference was running at ~1.3 requests per second on the B1_glaive partition, producing correctly structured data.

The Thinking Process

The reasoning visible in the surrounding messages reveals a methodical approach:

The assistant started with a hypothesis ("the /generate endpoint returns raw tokens") and tested it immediately. When the initial test showed missing tokens, instead of assuming a bug in the endpoint, the assistant dug deeper — checking token identities, examining the chat template output, and ultimately discovering that thinking is prompt-side.

This is a pattern of following the data rather than following assumptions. Each test informed the next, and the assistant was willing to abandon initial hypotheses (like "we need the reasoning parser") when evidence contradicted them.

The final approach — no reasoning parser, raw token IDs, client-side concatenation — is elegant precisely because it removes complexity rather than adding it. The original design added a reasoning parser to handle the thinking/ response structure, but this created more problems than it solved. The fix removes the parser entirely and handles the structure through simple token ID arithmetic.

Conclusion

Message [msg 3812] appears to be a simple status update, but it represents the culmination of a deep debugging session that revealed fundamental truths about how language model serving APIs interact with training data pipelines. The lesson is both specific and general: when building training data for speculative decoding, every layer of abstraction between the model's raw token output and the training dataset is a potential source of corruption. The cleanest solution is to eliminate those layers entirely and work directly with token IDs.

The message also demonstrates a valuable debugging methodology: when a high-level API produces unexpected results, drop down to the lowest available level and verify your assumptions about the data. The /generate endpoint, the tokenizer's apply_chat_template, and a simple Python script were all that was needed to trace the problem to its source and design a correct solution.