Verifying Tool Call Token Fidelity in OpenRouter-Based EAGLE-3 Training Data Generation

In the sprawling landscape of modern machine learning infrastructure, the gap between a working prototype and a production-ready data pipeline is often bridged by a single, meticulously verified assumption. Message [msg 4045] in this extended coding session represents exactly such a moment: the assistant, prompted by a user's request to "look at tool call correctness too," pivots from a high-level confidence about data reconstruction to a deep, empirical verification of how OpenRouter's API handles tool-calling special tokens. What emerges is a textbook example of defensive engineering — a deliberate, methodical check that the reconstruction formula for converting OpenRouter's text responses back into token IDs preserves the exact structure of tool calls, which is essential for training the EAGLE-3 speculative decoding drafter.

The Context: Building Training Data at Scale

To understand why this message matters, we must first understand the broader mission. The team is building a large-scale training dataset for an EAGLE-3 drafter — a speculative decoding model that accelerates inference by predicting multiple tokens in parallel. The training data consists of hundreds of thousands of prompt-response pairs, where each response must be captured as exact token IDs from the Kimi-K2.5 model. These token IDs are used to train the drafter to predict hidden states, so any corruption in the token sequence — especially around special tokens like tool call markers — would introduce noise into the training signal.

The team had originally been generating responses using local GPU inference with SGLang, but the computational demands were enormous. A strategic pivot was made to use OpenRouter's API, which provides access to Kimi-K2.5 through multiple provider backends. This brought a new challenge: OpenRouter returns responses as text strings, not token IDs. The assistant had built a reconstruction pipeline that concatenates reasoning + " response" + content + "<|im_end|>" and then tokenizes the result. But this reconstruction depends on a critical assumption — that the tool call special tokens embedded in the model's output survive the roundtrip through OpenRouter's API without being parsed, stripped, or transformed.

The User's Prompt: A Catalyst for Rigor

The immediate trigger for message [msg 4045] is the user's simple request in [msg 4041]: "Look at tool call correctness too." This one-line instruction reveals the user's deep understanding of the pipeline's fragility. The assistant had been confidently running the OpenRouter inference script, reporting that "everything works" and that the reconstruction code handles the format correctly. But the user recognized that tool calls represent a special case — they involve structured output that OpenRouter might handle differently than plain text.

The assistant's response in [msg 4042] shows an immediate shift in thinking. The reasoning block begins with "Good call. Let me think about tool call correctness more carefully," and then lays out two distinct scenarios:

  1. When tools is not sent in the request: The model may still generate tool call special tokens, but they appear as raw text in the content field.
  2. When OpenRouter parses tool calls: They appear in a structured message.tool_calls array, with the text between response and the tool call section going into content. The assistant correctly identifies that the B1_glaive dataset — which contains function-calling prompts — uses a system message to describe available functions rather than the OpenAI-compatible tools parameter. This means OpenRouter would not know to parse tool calls structurally, and the special tokens should appear as raw text. But this is an assumption that needs verification.

The Empirical Investigation

What follows is a series of three test scripts, each narrowing in on the critical question. The first attempt ([msg 4042]) tries to run three tests simultaneously and times out after 120 seconds — a provider was likely too slow. The second attempt ([msg 4043]) simplifies to a single test with a random B1_glaive prompt, but the model correctly identifies it can't book flights and doesn't generate tool calls. The third attempt ([msg 4044]) is more sophisticated: it searches the existing local SGLang responses for a sample that actually contains tool call tokens, finds sample 241 (a price discount calculation), and sends that exact prompt to OpenRouter.

The output from [msg 4044] reveals the critical finding. The OpenRouter response for sample 241 shows:

Message [msg 4045]: The Verification

Message [msg 4045] is the culmination of this investigation. It begins by summarizing the three findings:

  1. Tool call special tokens appear as raw text in content — <|tool_calls_section_begin|>, <|tool_call_begin|>, etc. are all present
  2. message.tool_calls is None — OpenRouter does not parse them into structured format
  3. No <|im_end|> in content — it's stripped by the provider The reconstruction formula is therefore: reasoning + " response" + content + "<|im_end|>". But the assistant doesn't stop there. The critical insight is the phrase "since we verified they roundtrip through the tokenizer" — but this verification hasn't actually been done yet. The assistant has only confirmed that the text contains the right strings. The crucial next step is to verify that when this reconstructed text is tokenized, the special token strings actually map to the correct token IDs. This is the heart of message [msg 4045]. The assistant writes a Python script (test_tc_encode.py) that:
  4. Takes the actual content and reasoning strings returned by OpenRouter for sample 241
  5. Reconstructs the full text as reasoning + " response" + content + "<|im_end|>"
  6. Tokenizes it using the Kimi-K2.5 tokenizer
  7. Checks that every expected special token ID is present in the output The script defines the known special token IDs for the Kimi-K2.5 model: - response = 163607 - <|tool_calls_section_begin|> = 163595 - <|tool_call_begin|> = 163597 - <|tool_call_argument_begin|> = 163598 - <|tool_call_end|> = 163599 - <|tool_calls_section_end|> = 163596 - <|im_end|> = 163586 The output confirms that all seven special tokens are present in the tokenized output, at the expected positions. The tool call section spans positions 79 through 105 in the 106-token sequence, with the correct ordering: section begin, call begin, argument tokens, call end, section end, followed by the <|im_end|> terminator.

Why This Matters: The EAGLE-3 Training Pipeline

The seemingly narrow verification in message [msg 4045] has profound implications for the entire EAGLE-3 training pipeline. The training data for the speculative decoding drafter must consist of exact token ID sequences that the model actually generated. Any deviation — a missing special token, a mis-encoded tool call boundary, an incorrect <|im_end|> placement — would cause the drafter to learn incorrect patterns.

Consider what would happen if the reconstruction were wrong. If OpenRouter had parsed the tool calls into structured format and stripped the special tokens from content, the reconstruction would produce a sequence missing the tool call markers entirely. The drafter would learn that after reasoning, the model simply stops — it would never learn to generate tool calls. Conversely, if the <|im_end|> token were missing, the drafter might learn to continue generating past the intended end of the response.

The verification in message [msg 4045] also validates the broader approach of using OpenRouter for training data generation. The team had already completed all B-datasets (B3-B8) in approximately 33 minutes at a cost of roughly $86 — a dramatic improvement over local GPU inference. But this speed and cost efficiency would be worthless if the data quality were compromised. The tool call verification is one of several quality checks that collectively ensure the OpenRouter-generated data is equivalent to what local inference would produce.

Assumptions and Their Verification

Message [msg 4045] reveals several assumptions that the assistant is consciously making and then verifying:

Assumption 1: OpenRouter does not parse tool calls when tools is not sent. This was verified empirically in [msg 4044] by sending a prompt known to trigger tool calls and observing that message.tool_calls was None while the raw special tokens appeared in content.

Assumption 2: The tokenizer correctly maps the special token strings back to their original IDs. This is the core verification in message [msg 4045]. The script confirms that every special token string in the reconstructed text maps to the expected token ID.

Assumption 3: The <|im_end|> token is stripped by the provider and must be appended. This was observed empirically — the content string from OpenRouter does not end with <|im_end|>, but the model's actual output always includes it. The assistant correctly appends it during reconstruction.

Assumption 4: The BPE tokenizer handles the boundary between response and the content correctly. This is a subtle but important point. The tokenizer uses Byte-Pair Encoding, which can merge tokens across boundaries if the text flows naturally. By reconstructing as reasoning + " response" + content, the assistant ensures that the response token (ID 163607) is encoded as a single special token, not split into subwords. The verification confirms this — response appears at position 66 as a single token.

The Thinking Process: From Confidence to Verification

What makes message [msg 4045] particularly instructive is the thinking process it reveals. The assistant moves through several cognitive stages:

  1. Initial confidence: In earlier messages, the assistant states that "everything works" and the reconstruction code handles the format correctly. There's a sense of momentum — the OpenRouter pipeline is running, datasets are completing, and the focus is on throughput.
  2. Receptivity to feedback: When the user says "Look at tool call correctness too," the assistant immediately pivots. The response begins with "Good call" — an acknowledgment that this is a legitimate concern worth investigating.
  3. Scenario analysis: The assistant systematically enumerates the possible behaviors OpenRouter could exhibit with tool calls, considering both the case where tools is sent and where it isn't.
  4. Empirical testing: Rather than relying on documentation or assumptions, the assistant writes test scripts to observe actual behavior. The first two attempts encounter obstacles (timeout, non-tool-calling sample), but the third succeeds.
  5. Verification of the reconstruction: Even after confirming that tool call tokens appear as raw text, the assistant doesn't stop. The critical insight is that "since we verified they roundtrip through the tokenizer" is an unverified claim — and message [msg 4045] is where that verification happens.
  6. Documentation of results: The output is presented clearly, showing each special token, its ID, and its position in the token sequence. The tool call section is decoded and displayed for human inspection. This thinking process exemplifies a key principle in ML engineering: trust but verify. The assistant trusts that OpenRouter returns text faithfully, but verifies that the tokenizer can reconstruct the exact token IDs. The assistant trusts that the reconstruction formula is correct, but verifies it with actual data from a tool-calling sample.

The Broader Implications for Data Pipeline Design

Message [msg 4045] offers lessons that extend far beyond this specific session. When building data pipelines that involve API intermediaries — whether for training data generation, model evaluation, or inference serving — the fidelity of structured data through the API is never guaranteed. OpenRouter, like many API providers, may transform model outputs in ways that are convenient for chat applications but destructive for training data.

The specific pattern demonstrated here — reconstructing token IDs from text responses — is increasingly relevant as more teams use API-based model access for data generation. The key insight is that the reconstruction must account for:

Conclusion

Message [msg 4045] is a masterclass in defensive data engineering. It demonstrates that the difference between a working pipeline and a broken one often comes down to a single verified assumption — in this case, that tool call special tokens survive the roundtrip through OpenRouter's API and can be faithfully reconstructed by the tokenizer. The assistant's methodical approach — from scenario analysis to empirical testing to verification of the reconstruction — provides a template for anyone building data pipelines that depend on API intermediaries.

The message also reveals the collaborative dynamic that makes this session productive. The user's simple prompt — "Look at tool call correctness too" — triggers a chain of investigation that uncovers and verifies a critical assumption. The assistant, in turn, responds not with defensive justification but with rigorous empirical testing. This partnership between human oversight and automated verification is what separates robust ML infrastructure from fragile prototypes.

In the end, the verification succeeds: all seven special tokens are present at their expected positions, the tool call section is intact, and the reconstruction formula is validated. The pipeline can proceed to the next phases with confidence. But the deeper lesson is that confidence must be earned through verification — and message [msg 4045] shows exactly how that work is done.