Rewriting the Pipeline: How Raw Token IDs Replaced the OpenAI API for EAGLE-3 Training Data

In the course of building a training dataset for EAGLE-3 speculative decoding on the Kimi-K2.5 model, a single message marked a decisive architectural pivot. The assistant, having spent several rounds debugging why the model's reasoning content was not being captured correctly, arrived at a clean solution: abandon the OpenAI-compatible chat completions API entirely and use SGLang's lower-level /generate endpoint to obtain raw token IDs. Message [msg 3800] captures this moment — a short, declarative summary of key changes followed by a file write operation. But behind those few lines lies a chain of reasoning that fundamentally reshaped the data pipeline.

The Problem: Reasoning Content Was Being Lost

The story begins with the user's observation in [msg 3763]: "Pretty sure we're running sglang with wrong reasoning parser(?)". The assistant investigated and discovered that SGLang offered a --reasoning-parser flag with a kimi_k2 option that mapped to the Qwen3Detector class ([msg 3774]). This detector was designed to split model output into reasoning_content and content fields by recognizing the thinking and response special tokens that Kimi-K2.5 uses.

The server was initially started without this flag, so all output — reasoning included — was dumped into a single content field with reasoning_content: null. The assistant restarted the server with --reasoning-parser kimi_k2 ([msg 3779]), expecting this would fix the data capture. But the user raised a crucial follow-up question in [msg 3786]: "We want either correct parsers or we want no parsing at all (special token-direct endpoint?)"

This question set off a deeper investigation. The assistant tested SGLang's /generate endpoint ([msg 3792]) and discovered something alarming: even with the reasoning parser enabled, the output_ids field was missing the thinking token (token ID 163606). The reasoning parser was stripping it from both the text output AND the raw token IDs. This meant that any approach relying on the OpenAI-compatible API — even with correct parsers — would produce incomplete token sequences.

The Discovery That Changed Everything

The critical insight came in <msg id=3795-3796>. The assistant inspected the chat template's output:

Chat template output:
'<|im_user|>user<|im_middle|>What is 2+2?<|im_end|><|im_assistant|>assistant<|im_middle|> thinking'

Last 10 token IDs: [220, 17, 10, 17, 30, 163586, 163588, 69702, 163601, 163606]
Last 10 decoded: [' ', '2', '+', '2', '?', '<|im_end|>', '<|im_assistant|>', 'assistant', '<|im_middle|>', ' thinking']

The thinking token (163606) was already being appended by apply_chat_template as the last token of the prompt. The model never generates it — it's part of the input. The model's generation begins after thinking, and when it finishes reasoning, it naturally emits the response token (163607) to transition to its final answer.

This meant two things. First, the reasoning parser was unnecessary: since thinking was in the prompt (not the generation), there was nothing for the parser to extract. Second, the parser was actively harmful: by stripping tokens from output_ids, it was corrupting the exact token sequence needed for EAGLE-3 training.

The Decision: Raw Token IDs via /generate

Message [msg 3800] is the culmination of this investigation. The assistant writes:

Now rewrite run_inference.py. The key changes: - Use /generate endpoint via raw HTTP (not OpenAI client) to get output_ids - Pre-tokenize prompts with apply_chat_template to get prompt_ids (includes thinking) - Store prompt_ids + output_ids directly — no text-based tokenization needed - This eliminates any parsing ambiguity

This is a clean architectural decision with several layers of reasoning beneath it.

Why /generate instead of the OpenAI client? The OpenAI-compatible /v1/chat/completions endpoint applies post-processing: it runs the reasoning parser, it may apply a tool-call parser, and it returns text strings rather than token IDs. Every layer of processing is a potential source of data corruption. The /generate endpoint is SGLang's native interface — it returns the raw output_ids array directly, with no parsing, no restructuring, no text conversion.

Why pre-tokenize with apply_chat_template? The Hugging Face tokenizer's apply_chat_template method handles all the special token formatting — the &lt;|im_start|&gt; and &lt;|im_end|&gt; markers, the role tags, and crucially the thinking token that signals the model to begin reasoning. By running this client-side, the assistant gains full control over the prompt construction. The resulting prompt_ids array is a faithful representation of exactly what the model sees.

Why concatenate prompt_ids + output_ids? For EAGLE-3 training, the model needs to learn the full token sequence — both the prompt and the generation. The response token (163607) that separates reasoning from final answer, the tool-call special tokens, the &lt;|im_end|&gt; stop token — all of these are naturally present in the raw output_ids. By concatenating the two arrays, the assistant produces a complete training example with zero ambiguity about token boundaries.

Assumptions and Their Validity

The approach makes several assumptions, all of which proved correct in subsequent testing.

The first assumption is that apply_chat_template correctly includes the thinking token. This was verified in [msg 3795] where the last token of the prompt was confirmed to be 163606 ( thinking). The second assumption is that the model naturally generates the response token when transitioning from reasoning to final answer. This was confirmed in [msg 3803] where the test showed response (163607) appearing at position 61 of the output. The third assumption is that the /generate endpoint does not apply any parsing or transformation to output_ids when the server runs without --reasoning-parser. This was the entire motivation for restarting the server without that flag in [msg 3798].

One assumption that deserves scrutiny is that the chat template's behavior is consistent across all prompt formats. The Kimi-K2.5 model uses a custom chat template that appends thinking for the assistant turn. If a different template or model were used, this behavior might differ. But for the specific model being deployed, the assumption held.

The Input Knowledge Required

To understand this message, one needs to know several things about the SGLang inference engine and the Kimi-K2.5 model architecture. The distinction between the OpenAI-compatible API (/v1/chat/completions) and SGLang's native /generate endpoint is crucial — the former applies post-processing, the latter returns raw outputs. The concept of a "reasoning parser" that splits model output into reasoning and content segments is specific to models that use special tokens like thinking/ response to demarcate their chain-of-thought. The role of apply_chat_template in Hugging Face's tokenizer — how it inserts special tokens, role markers, and generation prompts — is also essential background.

One also needs to understand why EAGLE-3 training requires exact token sequences. EAGLE-3 is a speculative decoding technique where a lightweight "drafter" model predicts the next several tokens of the base model, and the base model verifies them in parallel. The drafter must learn the exact token distribution of the base model, including special tokens like response and tool-call markers. Any discrepancy between the training data and the actual model output would cause the drafter to produce invalid token sequences, resulting in zero acceptance rate — exactly the problem the assistant had been debugging in earlier segments.

The Output Knowledge Created

This message produced a rewritten run_inference.py that fundamentally changed the data pipeline. The old pipeline used the OpenAI client library, sent messages as structured chat objects, received parsed text responses, and then re-tokenized the text to obtain token IDs — a roundabout process with multiple opportunities for tokenization mismatches. The new pipeline sends pre-tokenized input_ids to /generate, receives raw output_ids, and concatenates them directly — no re-tokenization, no parsing, no ambiguity.

The file write operation in this message was the first step. Subsequent messages show the script being copied to the remote machine ([msg 3805]), tested ([msg 3803]), and then used to restart the inference pipeline. The LSP errors shown at the bottom of the message are irrelevant to the actual deployment — they come from a different file (server_args_sm120.py) on the local development machine and reflect missing Python packages in the local environment, not issues with the rewritten script.

The Thinking Process Visible in This Message

What makes this message interesting is what it doesn't say. There is no lengthy justification, no exploration of alternatives, no hedging. The assistant has already done the investigation in previous messages and arrived at a conclusion. The message is purely declarative: "Now rewrite run_inference.py. The key changes: ..." followed by the file write.

This terseness reflects a moment of clarity after a long debugging session. The assistant had tried the reasoning parser approach (it stripped tokens), considered keeping the parser and reconstructing the sequence (complex and error-prone), and finally arrived at the simplest possible solution: bypass all parsing by using the raw token endpoint. The four bullet points read like a checklist of lessons learned:

  1. Use /generate — because the OpenAI API adds processing layers
  2. Pre-tokenize with apply_chat_template — because the prompt needs thinking
  3. Store prompt_ids + output_ids — because concatenation is simpler than reconstruction
  4. Eliminates parsing ambiguity — the ultimate goal The message also shows the assistant's comfort with the tooling. The [write] tool call writes a file directly, and the assistant doesn't need to show the file contents — it summarizes the key changes and trusts that the implementation details are correct. This is a sign of mature engineering: the assistant knows what needs to change, communicates the high-level design, and executes without second-guessing.

Conclusion

Message [msg 3800] represents a turning point in the EAGLE-3 data pipeline. After discovering that SGLang's reasoning parser was corrupting token sequences, and after realizing that the thinking token was already part of the prompt (not the generation), the assistant made a clean break from the OpenAI-compatible API. The decision to use raw token IDs via /generate eliminated an entire class of parsing bugs and produced training data that faithfully represents the model's actual token distribution. It's a textbook example of how understanding the underlying data format — in this case, the exact token IDs flowing through the model — can lead to simpler, more robust solutions than layering on more parsers and post-processing.