The Pivot to Raw Token Sequences: Why SGLang's Reasoning Parser Was Dropped for EAGLE-3 Training Data

Message 3798: "Good. Now restart without --reasoning-parser"

In a single, deceptively simple command, the assistant executed a strategic pivot that resolved a fundamental misunderstanding about how Kimi-K2.5 generates its reasoning traces. The message — a bash command restarting the SGLang inference server without the --reasoning-parser flag — was the culmination of a deep investigation into why the EAGLE-3 training data pipeline was producing corrupted samples. What makes this message significant is not the command itself, but the chain of reasoning that led to it: a journey through tokenizer internals, chat template behavior, and the subtle distinction between what a model generates versus what its prompt provides.

The Context: Building Training Data for EAGLE-3

The broader session was focused on training an EAGLE-3 speculative decoding drafter for the Kimi-K2.5 model. EAGLE-3 requires high-quality training data consisting of the exact token sequences the model produces during inference — including reasoning tokens, content tokens, and special structural tokens. The pipeline involved running inference on approximately 88,000 prompts using SGLang serving the Kimi-K2.5 model across 8 RTX PRO 6000 Blackwell GPUs.

The problem emerged when the assistant discovered that the reasoning content was being captured incorrectly. SGLang's OpenAI-compatible chat completions endpoint was returning reasoning_content: null and embedding the thinking text directly in message.content, with the opening thinking tag stripped. This meant the training data was missing the crucial structural boundary between reasoning and response phases.

The Investigation: Uncovering the Root Cause

The investigation unfolded over several messages (msg 3761–3797). Initially, the assistant assumed the fix was to enable SGLang's --reasoning-parser kimi_k2 option, which would properly split the output into reasoning_content and content fields. The server was restarted with this flag (msg 3779), and the assistant began testing.

But the user's question about tool calls (msg 3781–3782) prompted a deeper reflection. The assistant realized that for EAGLE-3 training, what matters is not the semantic separation of reasoning from content, but the exact token sequence the model produces. The OpenAI chat completions API, even with a reasoning parser, restructures the output in ways that could lose information.

The critical breakthrough came when the assistant tested SGLang's /generate endpoint (msg 3792), which returns raw output_ids alongside the decoded text. This revealed a startling fact: the output_ids array started at token 1008 (decoding to "The"), completely missing token 163606. Further investigation using the tokenizer (msg 3794–3795) confirmed that token 163606 corresponds to thinking, and token 163607 corresponds to response.

The Key Insight: The Chat Template Owns the Thinking Token

The most important discovery came in msg 3795–3796. By examining the chat template's output directly, the assistant found that apply_chat_template appends thinking (token 163606) as the last token of the prompt. The model's generation begins after this token — the model never generates thinking itself. This is a fundamental architectural choice by the Kimi-K2.5 designers: the thinking token is a prompt-side instruction that tells the model to begin its reasoning process, not a token the model emits.

This insight transformed the problem. If thinking is part of the prompt, then:

  1. The reasoning parser is harmful — it was stripping thinking from both text and token IDs, but since thinking was never generated, the parser was removing information that should be preserved as part of the prompt context.
  2. No parsing is needed — the raw token sequence (prompt_ids + output_ids) naturally contains the thinking token at the boundary between prompt and generation, and the response token (163607) appears naturally in output_ids when the model transitions from reasoning to answering.
  3. The /generate endpoint is ideal — it returns raw output_ids without any restructuring, giving the exact token sequence the model produced.

The Decision: Restart Without Reasoning Parser

Message 3798 executes this decision. The assistant kills the server that was running with --reasoning-parser kimi_k2 and restarts it without the flag. The command preserves all other server configuration: tensor parallelism across 8 GPUs (--tp-size 8), memory fraction at 0.85, continuous decoding steps, and the NCCL performance tuning parameters established earlier in the session.

The decision represents a correction of an earlier assumption. In msg 3769–3779, the assistant had assumed that the reasoning parser was the solution — that enabling it would fix the data capture. But deeper investigation revealed that the reasoning parser was actually part of the problem, because it was designed for the OpenAI chat completions API's semantic output format, not for the raw token sequence extraction needed by EAGLE-3 training.

Assumptions Made and Corrected

Several assumptions were challenged during this investigation:

Assumption 1: The thinking token is generated by the model. Corrected: It's appended by the chat template as the final prompt token.

Assumption 2: A reasoning parser is needed to properly separate reasoning from content. Corrected: For EAGLE-3 training, no separation is needed — the raw token sequence with structural tokens preserved is the correct training target.

Assumption 3: The OpenAI chat completions API is the right interface for data generation. Corrected: The /generate endpoint, which returns raw token IDs, is more appropriate for training data extraction.

Assumption 4: The reasoning parser only affects the text output, not token IDs. Corrected: The parser was stripping thinking from both text and output_ids in the /generate response, corrupting the token sequence.

Input Knowledge Required

To understand this message, one needs knowledge of:

Output Knowledge Created

This message created actionable knowledge:

  1. The correct server configuration for EAGLE-3 data extraction: no --reasoning-parser, use /generate endpoint.
  2. The data extraction protocol: Pre-tokenize prompts via apply_chat_template (which includes thinking), send to /generate, concatenate prompt_ids + output_ids for the full sequence.
  3. The structural token boundary: response (163607) will appear naturally in output_ids and serves as the delimiter between reasoning and content in the training data.

The Thinking Process Visible in Reasoning

The assistant's reasoning process, visible across the preceding messages, shows a methodical approach to debugging:

  1. Observation: Reasoning content is not properly separated in API output.
  2. Hypothesis 1: The --reasoning-parser flag is missing. Action: Enable it.
  3. Re-evaluation: The user's question about tool calls triggers a deeper analysis of whether parsing is even desirable.
  4. Hypothesis 2: The /generate endpoint might bypass parsing entirely. Action: Test it.
  5. Discovery: output_ids is missing the thinking token.
  6. Hypothesis 3: The chat template owns the thinking token. Action: Verify with tokenizer.
  7. Conclusion: No parser needed — use raw token IDs. Action: Restart server without --reasoning-parser. This progression from "fix the parser" to "eliminate the parser" demonstrates a willingness to question initial assumptions when new evidence emerges. The assistant didn't just fix the surface-level symptom (missing reasoning_content) but dug into the fundamental data flow to find the root cause.

Implications for the Pipeline

The decision in message 3798 had immediate downstream effects. The run_inference.py script would need to be rewritten to use the /generate endpoint instead of the OpenAI chat completions API. The data format would change from parsed JSON with reasoning_content/content fields to raw token ID sequences. And crucially, the training data would now faithfully represent what the model actually produces, including the response structural token that marks the transition from reasoning to answering.

This pivot also simplified the pipeline: no need to configure or debug reasoning parsers, no risk of parser bugs corrupting the data, and no need to handle tool call parsing separately. The raw token approach is both more robust and more correct for the EAGLE-3 training objective.

Conclusion

Message 3798 is a textbook example of a strategic pivot driven by deep technical investigation. What began as a quest to fix a reasoning parser configuration ended with the realization that the parser itself was unnecessary — and indeed harmful — for the actual task at hand. The assistant's willingness to question its own assumptions, verify them against tokenizer behavior, and choose the simpler, more correct approach over the more complex one is the hallmark of effective debugging. The command to restart without --reasoning-parser is not just a server configuration change; it represents a corrected mental model of how Kimi-K2.5 generates its output and what EAGLE-3 training truly requires.