Debugging Reasoning Capture and Token Reconstruction for Kimi-K2.5 EAGLE-3 Training Data

Introduction

In the ongoing effort to build a high-quality synthetic training dataset for EAGLE-3 speculative decoding on the Kimi-K2.5 model, a critical debugging moment occurred at message 2923. The assistant had been running a large-scale data generation pipeline—feeding 25,000 questions from the mlabonne/open-perfectblend dataset through a vLLM inference server running Kimi-K2.5 INT4 on 8x Blackwell GPUs—when the user identified two interrelated problems. First, the captured responses showed empty reasoning fields, meaning the model's rich internal chain-of-thought was being discarded. Second, the user flagged that when reconstructing the full token sequences for hidden state extraction and EAGLE-3 training, the correct special tokens (the "think" tokens) needed to be wrapped around the reasoning content. This message represents the assistant's diagnostic response: identifying the root cause of the reasoning capture bug and investigating the tokenizer to understand how to properly reconstruct sequences.

The Context: Why Reasoning Matters for EAGLE-3

To understand why this debugging moment matters, one must appreciate the architecture being built. EAGLE-3 is a speculative decoding framework that uses a lightweight draft model to predict the next several tokens, which are then verified by the full target model. For this to work well, the draft model needs to learn the target model's generation patterns—including its internal reasoning process. Kimi-K2.5, like many modern large language models, produces a two-phase output: first a reasoning/thinking phase where it works through the problem internally, then a response phase where it delivers the final answer. The reasoning phase is typically wrapped in special tokens (e.g., thinking... response) that demarcate the boundary. Capturing this reasoning is essential because it constitutes a significant portion of the model's output and contains the structured problem-solving patterns that the EAGLE-3 draft model needs to learn. Without it, the training data would be missing the most informative part of the model's generation.

The Bug: reasoning_content vs reasoning

The assistant's first action in this message is to identify the bug with characteristic precision: "It's msg.reasoning, not msg.reasoning_content. The script was checking the wrong attribute." This single sentence encapsulates the entire debugging session that preceded it. In the previous message ([msg 2922]), the assistant had tested the OpenAI Python client against the vLLM server and discovered that while the raw REST API returns the reasoning under the field name reasoning, the OpenAI Python client library exposes it as msg.reasoning (a direct attribute access), not msg.reasoning_content (which returned NOT_FOUND). The original script (01b_generate_synthetic.py) was likely written with an assumption about the attribute name—perhaps based on documentation for other models or APIs that use the reasoning_content field name (common in some OpenAI-compatible endpoints). This is a classic integration bug: the vLLM server's OpenAI-compatible API maps the reasoning field differently than expected, and the client library's attribute naming doesn't match the raw JSON key.

The mistake is understandable. The landscape of LLM API conventions is fragmented. Some providers use reasoning_content (e.g., some OpenAI-compatible endpoints), others use reasoning (the raw vLLM response), and the Python client library may map the raw JSON field to a Python attribute with a different name. The assistant's systematic approach—first testing the raw curl response, then testing the Python client, then comparing the two—demonstrates proper debugging methodology. The curl test in [msg 2921] showed the raw API returns "reasoning": "...", and the Python test in [msg 2922] confirmed that msg.reasoning works while msg.reasoning_content does not.

The Tokenizer Investigation: Finding the Think Tokens

Having resolved the reasoning capture issue, the assistant immediately pivots to the second problem: reconstructing the full token sequence with the correct special tokens. The user had noted "when reassembling we should append with correct Think tokens." This is a crucial detail for the EAGLE-3 training pipeline. The training data must contain the exact token sequence that the model would see during inference, including the special delimiter tokens that separate reasoning from response.

The assistant runs a bash command that probes the Kimi-K2.5 tokenizer for thinking-related tokens. The results are illuminating:

Assumptions and Mistakes

Several assumptions are visible in this message and its immediate context:

Assumption 1: The OpenAI client attribute name matches the raw API field. The original script assumed reasoning_content was the correct attribute, likely based on patterns from other API implementations. This was incorrect for the vLLM + OpenAI client combination being used.

Assumption 2: The model would automatically produce reasoning. The empty reasoning fields in the captured data suggest the model wasn't being prompted to reason. The assistant later realizes this and notes "The model might not be reasoning because we're not telling it to." This is a separate issue from the attribute name bug—even with the correct attribute, if the model isn't instructed to think, the reasoning field will be empty.

Assumption 3: The tokenizer would use standard think-token patterns. The assistant checked for common patterns like <|think|>, <|/think|>, <|startofthinking|>, <|endofthinking|>, but none of these are single tokens in Kimi-K2.5's vocabulary. Instead, the model uses the custom single tokens thinking (163606) and response (163607). This is a model-specific design choice that required empirical investigation.

Assumption 4: The special tokens would be in the standard tokenizer attributes. The assistant checked special_tokens_map and added_tokens to find think-related tokens. The tokens were found by iterating through a range of token IDs (163500–163840) and decoding each one—a brute-force approach that worked but suggests the tokens weren't easily discoverable through standard API methods.

Input Knowledge Required

To fully understand this message, one needs:

  1. Understanding of speculative decoding and EAGLE-3: Why reasoning data matters for training a draft model
  2. Knowledge of the OpenAI API and Python client: How raw API fields map to Python object attributes
  3. Familiarity with vLLM's OpenAI-compatible endpoint: How vLLM exposes reasoning fields in its chat completions API
  4. Understanding of tokenizer internals: How special tokens are stored, how to probe for them, and why single-token delimiters matter
  5. Knowledge of Kimi-K2.5's architecture: That it uses a two-phase generation (reasoning + response) with custom delimiter tokens
  6. The broader pipeline context: That this data will feed into hidden state extraction and EAGLE-3 training

Output Knowledge Created

This message produces several valuable outputs:

  1. The bug fix: The script should use msg.reasoning instead of msg.reasoning_content
  2. The tokenizer map: Token 163606 = thinking, token 163607 = response
  3. Confirmation of non-standard tokens: Standard think-token patterns like <|think|> are not single tokens in this vocabulary
  4. A methodology for investigating similar issues: Test raw API → test Python client → probe tokenizer → fix script
  5. Documentation of Kimi-K2.5's special token structure: The special tokens list including chat template markers

The Thinking Process

The assistant's reasoning in this message follows a clear diagnostic pattern. First, it states the discovered bug directly and authoritatively: "It's msg.reasoning, not msg.reasoning_content." This is stated as a fact, not a hypothesis, because it was verified in the previous message's Python test. The assistant then immediately pivots to the next problem—token reconstruction—without waiting for confirmation. This shows confidence in the diagnosis and a desire to move efficiently.

The bash command is carefully constructed. It loads the tokenizer from the model path, then probes for think tokens in three ways: (1) encoding common think-token strings to see if they map to single IDs, (2) checking the special tokens map, and (3) brute-force scanning a range of token IDs for anything containing "think" or "reason" in the decoded text. This multi-pronged approach ensures no stone is left unturned. The grep at the end filters out noisy log messages ("Calling super") that transformers library sometimes emits.

The results are presented cleanly, showing that thinking → [163606] and response → [163607] are the correct single-token delimiters. The assistant doesn't comment on the results—they speak for themselves. The message ends with the tokenizer output, implicitly saying "this is what we needed to know."

Broader Significance

This debugging moment, while small in isolation, illustrates several important principles in AI engineering. First, API integration bugs are common and require systematic testing—never assume that field names in documentation match actual behavior. Second, tokenizer probing is an essential skill when working with non-standard model architectures; the tokenizer is the ground truth for how text maps to tokens. Third, the two-phase generation pattern (reasoning + response) is becoming increasingly common in modern LLMs, and understanding how to properly capture and reconstruct these sequences is critical for fine-tuning and speculative decoding pipelines.

The discovery that Kimi-K2.5 uses custom single tokens (163606 and 163607) for reasoning/response demarcation rather than multi-token textual markers is a significant architectural detail. It means the model has dedicated vocabulary entries for these functions, which likely allows for more efficient generation and more precise control over the reasoning/response boundary. For the EAGLE-3 training pipeline, this means the reconstruction logic must insert these specific token IDs rather than textual strings—a detail that, if missed, would corrupt the training data and render the draft model ineffective.

In the broader arc of the session, this message represents a turning point where the data generation pipeline shifts from blind collection to informed, precise capture. The reasoning data will now be properly extracted, and the token sequences will be correctly reconstructed. The 388 samples already collected (as noted in [msg 2920]) were captured with the buggy attribute check, meaning their reasoning fields are empty—they will need to be re-collected or the reasoning will need to be extracted retroactively from the content field. The assistant's next steps would be to fix the script, restart the generation with the corrected attribute and proper thinking prompts, and ensure the reconstruction logic uses tokens 163606 and 163607 correctly.