Verifying Data Integrity in an EAGLE-3 Training Pipeline

In the middle of a sprawling machine learning engineering session — one that had already spanned dozens of hours, multiple server reboots, kernel compilation failures, and a complete rewrite of the inference pipeline — a single question from the user triggered a precise, surgical verification. The question, at message index 3822, was deceptively simple: "Are we seeing correct toolcalling behavior? Is system/user prompt included?"

The answer came in message 3823, and it is the subject of this article. On its surface, the message is a straightforward bash command that runs a Python script to inspect a few training samples. But beneath that simplicity lies a critical moment of quality assurance — a checkpoint where weeks of work could have been invalidated by a subtle data corruption bug.

The Context: A Pipeline Built on Raw Token Fidelity

To understand why this verification mattered, we need to step back. The broader session (segments 23–28 of this opencode conversation) was dedicated to training an EAGLE-3 speculative decoding drafter for the Kimi-K2.5 model — a large language model deployed on a machine with 8 NVIDIA RTX PRO 6000 Blackwell GPUs. The drafter's job is to predict the next several tokens in a draft, which the main model then verifies in parallel, accelerating inference.

The training data for this drafter was being generated by running the Kimi-K2.5 model itself on 88,000 prompts drawn from eight different datasets (B1_glaive through B8_sweagent, plus two auxiliary sets A1 and A2). The model would generate responses, and those responses — complete with reasoning chains, tool calls, and final answers — would become the ground truth for the drafter to learn from.

But there had been a critical bug. Earlier in the session ([msg 3812]), the assistant discovered that the inference script was using OpenAI's /v1/chat/completions endpoint, which caused SGLang's reasoning parser to strip the <think> tags and embed reasoning content into message.content with reasoning_content: null. The fix was a complete rewrite: bypass the OpenAI-compatible API entirely and use SGLang's raw /generate endpoint, which returns output_ids — the model's exact token sequence including special tokens like <think> (token 163606), </think> (token 163607), tool call markers, and the end-of-turn token <|im_end|>.

This rewrite was the foundation. But the user, understandably, wanted proof that the new pipeline was producing correct data.

The Message: A Targeted Verification Script

The assistant's response in message 3823 begins with a simple acknowledgment: "Good questions. Let me check both — whether the tool call format is what the model should produce, and whether the prompt includes the system message with tool definitions."

Then comes the bash command — a Python script executed over SSH on the remote inference server. The script does three things:

  1. Loads a tool-calling sample from the raw responses file for the B1_glaive dataset (a function-calling dataset with tool definitions).
  2. Loads the corresponding prompt by matching sample_id to the line number in prompts.jsonl.
  3. Prints diagnostic information: the messages array (system + user), the full chat template text after tokenization, the prompt token IDs, and the decoded full sequence. The output shown in the message is truncated — we see the system message beginning with "SYSTEM: You are a helpful assistant with access to the following functions..." and the user message about a dress discount. But the structure is clear: the assistant is visually confirming that the system prompt (with tool definitions) is present in the data, and that the model's response format is correct.

Why This Verification Was Critical

The user's question touched on two distinct concerns, each with its own failure mode.

Concern 1: Tool calling behavior. The Kimi-K2.5 model uses a special token format for tool calls: <|tool_calls_section_begin|>, <|tool_call_begin|>, <|tool_call_argument_begin|>, and so on. If the raw token capture was somehow mangling these — for instance, if the tokenizer was decoding and re-encoding them incorrectly, or if the /generate endpoint was stripping them — then the training data would teach the drafter to produce malformed tool calls. The drafter would learn to predict broken sequences, and any downstream application relying on tool use would fail.

Concern 2: System/user prompt inclusion. The training data for EAGLE-3 consists of (prompt, response) pairs. The prompt is the full chat history including system messages, tool definitions, and user messages. If the prompt was being truncated or if the system message was being dropped, the drafter would learn to generate responses without the proper context. The model might still produce plausible-sounding text, but it would lack the grounding needed for tool use or instruction following.

The assistant's script directly addresses both concerns. By loading the raw prompt and printing its messages, it confirms that the system message with tool definitions is intact. By decoding the full sequence (prompt + output), it confirms that the model's response contains the expected tool call tokens.

The Assumptions Underlying the Check

Every verification rests on assumptions, and this one is no exception. The script assumes that:

What the Output Reveals

The truncated output shows the system message beginning with "SYSTEM: You are a helpful assistant with access to the following functions..." and a JSON tool definition for calculate_discounted_price. The user message asks about a dress discount. This confirms that:

  1. The prompt includes the system message. The tool definitions are present in the training data, so the drafter can learn to recognize when tool use is appropriate.
  2. The model's response format is preserved. Earlier checks ([msg 3819]) had shown that the output contains </think> at the correct position, followed by content and tool call tokens. The raw token IDs capture the exact sequence the model generated. The verification also implicitly confirms that the fix from [msg 3812] is working. The old pipeline (using /v1/chat/completions) would have returned reasoning content embedded in message.content with the <think> tags stripped. The new pipeline (using /generate) returns the raw token IDs, and the decoded output shows the reasoning before </think> and the content after it — exactly as the model produced it.

The Broader Significance

This message is a small but essential piece of a much larger puzzle. The EAGLE-3 training pipeline had already consumed enormous effort: building flash-attn from source, tuning CUDA toolkit versions, debugging SGLang server hangs on SM120 GPUs, optimizing KV cache settings for throughput, and fixing the reasoning capture bug. Each of these was a potential failure point. A single corrupted training sample could poison the drafter, causing it to learn incorrect patterns that would be difficult to diagnose later.

The user's question and the assistant's response represent a quality gate — a moment of deliberate verification before committing to a long-running computation. The inference pipeline was about to generate ~228 million tokens across 88K prompts, taking an estimated 17–26 hours. Starting that process with unverified data would have been reckless.

In software engineering terms, this is the difference between "it compiles" and "it's correct." The pipeline was running. Data was being produced. But until someone checked that the data was actually what the model generated — not a parsed, stripped, or corrupted version — the pipeline was producing garbage. The assistant's verification script was the lintel in the quality arch.

Conclusion

Message 3823 is a textbook example of defensive data engineering. Faced with a pointed question about data integrity, the assistant did not simply assert correctness — it wrote a targeted script that cross-referenced two data files, decoded token sequences, and displayed the results for human inspection. The verification confirmed that tool calling behavior was correct and that system/user prompts were included, giving the green light for the full inference run to proceed.

The assumptions were reasonable, the scope was appropriate, and the output was actionable. This is the kind of check that separates robust ML pipelines from fragile ones: not just building the machinery, but verifying that it produces what you think it produces.