The Critical Edit: Fixing B8's Dataset Format in the EAGLE-3 Training Pipeline

Introduction

In the sprawling complexity of an 83,000-sample ML training pipeline, a single three-word phrase—"Now fix B8"—can represent the culmination of hours of debugging, the unblocking of a critical data path, and the difference between a pipeline that produces 10,000 usable training samples and one that silently yields zero. Message [msg 3679] in this opencode session is deceptively brief: the assistant states "Now fix B8 (SWE-agent trajectories):" and applies an edit to a Python script. But this message is the third in a chain of dataset format fixes, each one necessitated by the gap between the assumptions baked into the original data preparation script and the messy reality of real-world datasets sourced from diverse Hugging Face repositories. To understand why this message matters—and why it represents a pivotal moment in a much larger endeavor—requires zooming out to see the full pipeline, the debugging chain that preceded it, and the specific format mismatch that made B8's fix necessary.

The Broader Context: Scaling EAGLE-3 Training by 10×

The subject message sits within a massive data scaling operation. The team had been training an EAGLE-3 draft model for the Kimi-K2.5 large language model, but acceptance rates were stuck around 2.1 tokens, insufficient to overcome speculation overhead. The EAGLE-3 paper's scaling curves suggested that more training data was the primary lever for improvement. The response was ambitious: scale the training dataset by 10×, from roughly 8,000 samples to 88,000. This meant sourcing diverse datasets—agentic coding trajectories, reasoning chains, general chat conversations—and processing them through a pipeline that would regenerate responses using the target Kimi-K2.5 model, ensuring the training data matched the model's actual output distribution.

The plan, documented in train_plan_v4.md ([msg 3661]), called for ten parallel dataset preparation agents. Two datasets (A1: DeepSWE-Kimi trajectories, A2: KimiK2.5-2000x) already contained Kimi-native outputs and could be tokenized directly. Eight datasets (B1–B8) contained prompts only—questions, instructions, or conversation starters—that needed inference through the Kimi-K2.5 model to generate matching responses. The B datasets included Glaive function calling, OpenCodeInstruct, Magicoder, MixtureThoughts, OpenThoughts, UltraChat, ShareGPT, and SWE-agent trajectories. Together, they represented 83,288 prompts requiring inference, an estimated 24–55 hours of server time on the 8-GPU SGLang baseline.

The Dataset Format Problem

The unified preparation script prep_all.py was written to handle all ten datasets through a single entry point, with each dataset having its own handler function. The script assumed certain common data formats: either a messages list with role/content keys (for Kimi-native data) or a conversations list with from/value keys (for prompt extraction). These are reasonable assumptions—they match the most common Hugging Face dataset conventions. But as the assistant discovered when the preps completed, three of the ten datasets produced zero usable records.

The failures were:

The Debugging Chain: A1 → B5 → B8

The fixes followed a deliberate debugging chain. For A1, the assistant launched a task agent to inspect the data format ([msg 3672]), discovering the multi-turn structure with 84 messages. The fix required changing the logic to tokenize the full conversation and mark all non-system content as trainable, rather than searching for a single assistant response at the end.

For B5, another task agent revealed the from == "user" vs from == "human" mismatch ([msg 3672]). The fix was a simple string comparison change.

For B8, the assistant inspected the format directly via bash ([msg 3675]), finding the trajectory key with entries containing role, text, and system_prompt. The trajectory had 93 entries, starting with a system prompt (text: None), then alternating user and assistant turns representing the agent's interaction with a command-line environment. The prompt to extract was the first user turn with actual text content—typically a GitHub issue description embedded in the conversation.

Each fix built on the previous one. By the time the assistant reached B8, the pattern was clear: the original script had made assumptions about data formats that were too narrow. The fix for B8 required adding yet another handler function, this time for the SWE-agent's trajectory format.

The B8 Fix Specifically

Message [msg 3679] reads in its entirety:

Now fix B8 (SWE-agent trajectories): [edit] /home/theuser/glm-kimi-sm120-rtx6000bw/eagle3-train/datasets/prep_all.py Edit applied successfully.

>

LSP errors detected in this file, please fix: <diagnostics file="/home/theuser/glm-kimi-sm120-rtx6000bw/eagle3-train/datasets/prep_all.py"> ERROR [40:14] Import "transformers" could not be resolved ERROR [152:23] "load_dataset" is not a known attribute of "None" ERROR [152:23] "load_dataset" is not a known attribute of module "datasets" ERROR [251:23] "load_dataset" is not a known attribute of "None" ERROR [251:23] "load_dataset" is not a known attribute of module "da...

The LSP errors shown are red herrings—they stem from the local development environment lacking transformers and datasets packages, not from any actual code defect. The assistant correctly ignored them, as it had with the identical errors in the previous two fixes ([msg 3677], [msg 3678]).

The actual edit added a handler for the SWE-agent trajectory format. Based on the inspection results from [msg 3675], the handler needed to:

  1. Detect the trajectory key in the dataset record
  2. Find the first user turn with non-empty text content (skipping the initial system turn which had text: None)
  3. Extract that text as the prompt
  4. Handle the case where the trajectory might be empty or have no user turns This fix was the third and final in a series that would transform the pipeline's yield from 7 datasets (with A1, B5, B8 producing zero) to all 10 datasets contributing usable samples. The SWE-agent dataset alone contributed 10,000 prompts—a significant portion of the 83,288 total.## The Thinking Process: What the Assistant's Actions Reveal While message [msg 3679] itself is brief, the surrounding context reveals a rich thinking process. The assistant's debugging approach followed a clear pattern: launch first, inspect results, fix failures. This "fail fast, fix fast" methodology is well-suited to the parallel-agent architecture of opencode sessions, where ten dataset preps could run simultaneously and the assistant could inspect all their outputs at once. The assistant's reasoning is visible in the sequence of actions. After launching all ten preps ([msg 3669]), the assistant waited 30 seconds then checked all logs ([msg 3671]). This timing is deliberate—long enough for small datasets to complete, short enough to catch failures early before the inference server was fully booted. The assistant then prioritized the fixes by severity: A1 (Kimi-native data, 2,809 samples lost), B5 (114,000 samples lost), and B8 (10,000 samples lost). The order also reflects complexity—A1 required the most invasive fix (changing the tokenization logic for multi-turn conversations), B5 was a trivial string fix, and B8 required adding a new handler. The assistant's decision to use task agents for A1 and B5 but a direct bash command for B8 is itself revealing. Task agents are heavier—they spawn subagent sessions that can run arbitrary multi-round conversations. For A1, the complexity of the multi-turn format warranted a full investigation. For B5, the assistant wanted to confirm the field name mismatch. For B8, by the time the assistant reached it, the pattern was familiar enough that a quick bash inspection sufficed. This demonstrates adaptive problem-solving: the assistant learned from the first two debugging sessions and applied that learning to the third.

The Hidden Complexity of Dataset Format Heterogeneity

One of the key insights from this message chain is the sheer diversity of dataset formats in the Hugging Face ecosystem. The ten datasets in this pipeline used at least five distinct schema patterns:

  1. Standard messages format (role/content dicts): Used by KimiK2.5-2000x and several others
  2. conversations format (from/value dicts): Used by OpenThoughts and Glaive
  3. trajectory format (role/text/system_prompt dicts): Used by SWE-agent
  4. Multi-turn agent traces (84 alternating messages): Used by DeepSWE-Kimi
  5. Flat text fields: Used by UltraChat and ShareGPT Each format required a different extraction strategy. For the trajectory format specifically, the challenge was that the first entry was always a system prompt with text: None, and the actual user prompt (a GitHub issue description) appeared later in the conversation. The handler had to skip the system turn, find the first user turn with non-empty text, and extract that as the inference prompt. Additionally, the SWE-agent trajectories contain tool call responses interspersed with assistant messages, making them unsuitable for direct tokenization as Kimi-native data—they had to be treated as prompt-only and regenerated through inference. This heterogeneity is a fundamental challenge in ML data pipelines. Each dataset comes from a different research group with different conventions, and unifying them requires either a flexible schema-matching approach (which can miss edge cases) or per-dataset handlers (which are labor-intensive to write). The assistant chose the latter approach, adding handlers for each failing dataset as the failures were discovered. This is pragmatic but carries the risk of overfitting to the specific datasets in the current pipeline—if new datasets were added later, they might require yet more handlers.

The Broader Implications for the EAGLE-3 Training Pipeline

The B8 fix, while small in isolation, was critical to the overall data scaling strategy. The SWE-agent trajectories dataset represented 10,000 prompts of agentic coding interactions—exactly the kind of data most likely to improve the EAGLE-3 draft model's performance on coding tasks. Agentic coding trajectories are particularly valuable because they involve long chains of reasoning, tool use, and error recovery, which are challenging for draft models to predict. By including this data in the training mix, the team hoped to improve the draft model's ability to anticipate the target model's behavior in complex, multi-step scenarios.

Furthermore, the SWE-agent dataset's long-context nature (93 turns per trajectory) presented an opportunity for the inference pipeline. The assistant had noted in the plan document that longer-context samples might need special handling with lower parallelism ([msg 3657]). The SWE-agent trajectories would test this—they required the SGLang server to process prompts that could be thousands of tokens long, potentially hitting the 10,000-token context window limit.

Conclusion: The Weight of a Three-Word Message

Message [msg 3679] is a study in the power of brevity in complex systems. Three words—"Now fix B8"—encapsulate hours of debugging, three dataset format investigations, and the unblocking of a critical data pipeline. The message itself is just the final action in a chain that began with the assistant noticing zero-record outputs in the logs, proceeded through targeted inspections of each failing dataset, and culminated in three targeted edits to the preparation script.

What makes this message interesting is not what it says, but what it represents: the moment when the third and final dataset format bug was fixed, transforming a pipeline that would have produced only 7 datasets' worth of training data into one that would produce all 10. In the context of the larger EAGLE-3 training effort, where every additional 10,000 samples could meaningfully improve the draft model's acceptance rate, this fix was not a minor cleanup—it was a critical enabler of the entire data scaling strategy.

The LSP errors that follow the edit are a fitting coda: they are irrelevant, stemming from a local environment mismatch, and the assistant correctly ignores them. In a pipeline with dozens of moving parts—remote servers, parallel agents, GPU inference, hidden state extraction—the ability to distinguish real errors from noise is perhaps the most valuable skill an AI assistant can demonstrate. The B8 fix, like the A1 and B5 fixes before it, required exactly that discernment.