The Dataset Format Detective: Fixing Multi-Turn Agentic Conversation Parsing for EAGLE-3 Training

Introduction

In the sprawling ecosystem of an EAGLE-3 speculative decoding training pipeline, data is the lifeblood. The entire enterprise—training a lightweight draft model that can accelerate inference on the massive Kimi-K2.5 language model—hinges on having high-quality training data that accurately represents the target model's output distribution. When the assistant in this opencode session set out to scale the training dataset by a factor of 10, it launched ten parallel agents to download and prepare diverse datasets spanning agentic coding, reasoning, and general chat domains. But as the results came back, a pattern of zeros emerged: three of the ten datasets produced zero usable records. The target message, message 3677, is the moment where the assistant diagnoses the root cause of these failures and implements a unified fix, revealing a deeper lesson about the complexity of real-world conversational data.

The Context: A Pipeline Under Construction

To understand message 3677, one must appreciate the broader arc of the session. The assistant had just resolved a critical bug where the EAGLE-3 speculative decoding server was started with --speculative-algorithm EAGLE instead of EAGLE3, causing hidden states to be passed as single-layer 7168-dimensional vectors instead of the expected multi-layer 21504-dimensional concatenated vectors. After fixing that flag, the draft model finally achieved some acceptance (accept_len ~2.1), but throughput remained at 82.3 tok/s—still below the 90 tok/s non-speculative baseline. The EAGLE-3 paper's scaling curves suggested that more training data was the primary lever for improvement. Hence the massive data scaling effort: ten datasets totaling 88,088 samples (4,800 tokenized Kimi-native + 83,288 prompts needing inference) were selected, and an inference pipeline was launched on the baseline SGLang server to regenerate responses for the prompt-only datasets.

In the messages immediately preceding 3677 (messages 3657 through 3676), the assistant wrote a comprehensive plan document (train_plan_v4.md), created a unified dataset preparation script (prep_all.py), and launched all ten dataset preps in parallel on a remote container. The script was designed to handle two categories of datasets: "A" datasets that already contained Kimi model outputs (requiring only tokenization) and "B" datasets that contained only prompts (requiring inference through Kimi-K2.5 to generate responses). When the assistant checked progress, it discovered that three datasets had produced zero records: A1 (DeepSWE-Kimi, 0 of 2809), B5 (OpenThoughts, 0 prompts), and B8 (SWE-agent, 0 prompts).

The Target Message: Diagnosis and Decision

Message 3677 is the assistant's response to these failures. It reads:

Now I see the problem with A1 — the messages have "content" that is empty/None for some turns (the system prompt has text: None style). And the format has 84 messages where many assistant turns contain <function=...> calls and user turns contain tool results. The last message may not be "assistant".

>

For A1, the best approach is to tokenize the full multi-turn conversation and mark ALL non-system content as trainable (since this is already Kimi output). Let me fix all three: [edit] /home/theuser/glm-kimi-sm120-rtx6000bw/eagle3-train/datasets/prep_all.py Edit applied successfully.

>

LSP errors detected in this file, please fix: ERROR [40:14] Import "transformers" could not be resolved ERROR [152:23] "load_dataset" is not a known attribute of "None" ...

This message is deceptively short but packs significant reasoning. Let us unpack it layer by layer.

The Reasoning Process

The assistant's thinking is revealed through the structure of the message. It begins with a synthesis statement: "Now I see the problem with A1." This "now" is important—it signals that the assistant has just completed the investigative work from the previous messages. In message 3672, it ran a debug task that inspected the A1 dataset's structure and found that records had 84 messages alternating between system, user, and assistant roles, with the last message not necessarily being an assistant turn. The original prep_all.py script assumed a simple two-turn structure: a prompt (all messages before the last assistant turn) and a response (the last assistant turn). This assumption failed catastrophically on multi-turn agentic conversations.

The assistant then articulates the specific data characteristics it discovered:

  1. Some turns have empty/None content (the system prompt has text: None)
  2. The format has 84 messages per conversation
  3. Assistant turns contain <function=...> calls (tool invocations)
  4. User turns contain tool results
  5. The last message may not be an assistant turn This is a remarkably rich description of the data format. The DeepSWE-Agent-Kimi-K2-Trajectories dataset contains real agentic coding sessions where an LLM-powered agent interacts with a software environment, issuing function calls and receiving tool outputs. Each conversation is a complete trace of an autonomous programming session, with the Kimi-K2.5 model generating all the assistant responses. The original parsing logic was designed for simple chat conversations (user asks, assistant answers), not for these complex multi-turn agent trajectories.

The Decision: A Shift in Strategy

The assistant's decision is stated clearly: "For A1, the best approach is to tokenize the full multi-turn conversation and mark ALL non-system content as trainable (since this is already Kimi output)." This is a significant strategic pivot. The original approach treated only the last assistant message as the training target (the response to predict). The new approach treats every non-system turn as trainable, effectively multiplying the training signal from each conversation. Since all assistant responses in the dataset were generated by Kimi-K2.5 (the same model being targeted for speculative decoding), every assistant turn is a valid sample of the target model's output distribution.

This decision reflects a deep understanding of the EAGLE-3 training objective. The draft model needs to learn to predict the target model's hidden states at every position, not just at the final response. By marking all assistant turns as trainable, the assistant maximizes the utility of each multi-turn conversation. The system prompt (which is not model output) and user turns (which are inputs, not predictions) are excluded via the loss mask.

The phrase "Let me fix all three" is also telling. The assistant has identified that A1, B5, and B8 all have format issues, but the fixes are different. For B5, the issue was a simple field name mismatch (from == "human" vs from == "user"). For B8, the data used a trajectory field instead of messages. For A1, the issue was the multi-turn structure itself. The assistant applies a single edit to prep_all.py that addresses all three, suggesting a unified refactoring of the parsing logic.

Assumptions Made

The assistant makes several assumptions in this message:

  1. All non-system content is trainable: The assumption that every assistant turn in the A1 dataset is valid Kimi-K2.5 output is reasonable given the dataset's provenance (it was generated by Kimi-K2.5), but it's worth noting that some assistant turns contain function calls like <function=...> which are structural rather than generative. The draft model would need to learn to predict these as well, which is appropriate for agentic use cases.
  2. The LSP errors are false positives: The diagnostic errors about transformers import resolution and load_dataset not being a known attribute are LSP (Language Server Protocol) errors from the editor, not runtime errors. The assistant correctly ignores them—these are import resolution issues in the static analysis tool, not actual bugs in the code.
  3. The fix is sufficient for all three datasets: The assistant assumes that a single edit can fix A1, B5, and B8 simultaneously. This is a reasonable assumption given that all three failures stem from the same root cause: the parsing logic was too rigid and didn't account for the diversity of real-world data formats.

Mistakes and Incorrect Assumptions

The original parsing logic contained several incorrect assumptions that this message corrects:

  1. The "last assistant turn" heuristic: The original code assumed that the last assistant message in a conversation was the response to predict. This fails for multi-turn conversations where the last message might be a user turn (e.g., a tool result) or where there are multiple assistant turns.
  2. Simple two-turn structure: The original code assumed conversations had a simple structure: a sequence of messages where the last assistant turn was the response and everything before was the prompt. This is valid for chat datasets but not for agentic trajectories.
  3. Uniform message format: The original code assumed all datasets used the same message format (OpenAI-style role/content dicts). The B8 dataset used a trajectory field with a different structure, and B5 used from/value keys instead of role/content.
  4. Non-empty content: The original code didn't handle turns with empty or None content, which caused parsing failures when encountering system prompts with text: None.

Input Knowledge Required

To fully understand message 3677, one needs:

  1. Knowledge of the EAGLE-3 training pipeline: Understanding that the goal is to train a draft model that predicts the target model's hidden states, and that training data consists of tokenized sequences with loss masks indicating which tokens are trainable.
  2. Knowledge of the dataset landscape: The ten datasets span different domains (agentic coding, reasoning, general chat) and different formats (OpenAI messages, ShareGPT conversations, trajectory logs).
  3. Knowledge of the prep_all.py script: The original parsing logic and its assumptions about conversation structure.
  4. Knowledge of the remote infrastructure: The container at 10.1.230.174 running the prep scripts, the SGLang server for inference, and the data directories.
  5. Knowledge of the tokenizer format: The Kimi-K2.5 tokenizer and how input_ids, loss_mask, seq_len, n_prompt_tokens, and n_response_tokens are structured in the tokenized_data.jsonl output.

Output Knowledge Created

This message creates several important outputs:

  1. A corrected dataset preparation script: The edit to prep_all.py fixes the parsing logic for all three broken datasets, enabling the pipeline to proceed.
  2. A reusable pattern for multi-turn conversations: The insight that "mark ALL non-system content as trainable" is a generalizable approach for any dataset where the target model generated all assistant turns.
  3. Documentation of data format diversity: The message implicitly documents that agentic coding datasets have fundamentally different structure from chat datasets, requiring different parsing strategies.
  4. A decision point for the pipeline: With this fix, the A1 dataset goes from 0 to potentially thousands of training samples, significantly increasing the training data for the EAGLE-3 drafter.

Broader Significance

Message 3677, while brief, represents a critical inflection point in the data pipeline. The assistant could have taken a different approach—it could have written separate parsers for each dataset, or it could have skipped the problematic datasets entirely. Instead, it identified the common thread (rigid parsing assumptions) and implemented a unified fix that makes the pipeline more robust to real-world data diversity.

This is a recurring theme in machine learning engineering: data is never clean, and format assumptions are always wrong. The most valuable skill is not writing perfect parsers on the first try, but quickly diagnosing failures and adapting. The assistant's debugging workflow—launch parallel preps, check results, inspect failed datasets, synthesize findings, apply fix—is a textbook example of how to build robust data pipelines at scale.

The fix also has implications for the EAGLE-3 training quality. By marking all assistant turns as trainable in multi-turn conversations, the draft model will learn to predict hidden states across a wider range of contexts, including tool use and function calling. This could improve the draft model's performance on agentic tasks, which is particularly valuable given that the Kimi-K2.5 model is being used for coding and reasoning applications.

Conclusion

Message 3677 is a masterclass in data pipeline debugging. In a few lines, the assistant synthesizes findings from multiple investigations, identifies the root cause of three separate failures, articulates a unified fix strategy, and implements it. The message reveals the assistant's deep understanding of both the data formats and the training objective, and it demonstrates the iterative nature of real-world ML engineering: launch, check, diagnose, fix, repeat. The LSP errors at the end serve as a reminder that even in a successful edit, the development environment will always find something to complain about—but the important thing is that the runtime logic is correct. With this fix, the data pipeline can proceed, and the EAGLE-3 drafter will have 10× more training data to learn from.