The 10 Million Token Question: How a Single Sentence Reshaped a Training Pipeline

Subject Message: [user] Maybe in each category instead doing the whhole thing just do 10M tokens per category? — Message 3920 in the conversation, a user intervention during an EAGLE-3 training data generation pipeline

Introduction

In the middle of a sprawling, multi-day coding session dedicated to training an EAGLE-3 speculative decoding drafter for the Kimi-K2.5 model, a single user message arrives that is deceptively simple. It reads: "Maybe in each category instead doing the whhole thing just do 10M tokens per category?" The typo ("whhole") and the casual phrasing belie the weight of this intervention. This is not a technical question or a bug report — it is a strategic redirection. In one sentence, the user redefines the scope of an entire data generation pipeline, imposing a hard token budget that fundamentally changes the trajectory of the work.

To understand why this message was written, what it assumes, and what it set in motion, we must reconstruct the context of the session that led to this moment.

The Context: A Pipeline Under Pressure

The session had been running for hours — possibly days — across multiple segments (23 through 28, as documented in the conversation analysis). The goal was ambitious: generate synthetic training data for an EAGLE-3 draft model by running inference on a massive set of prompts across seven categories (B1 through B7), totaling approximately 88,000 samples. Each prompt would be fed to the Kimi-K2.5 model (an 8×GPU deployment using SGLang), and the model's response — including its full token sequence with reasoning tokens — would be captured as training data.

The assistant had already overcome numerous obstacles: fixing a reasoning capture bug by rewriting run_inference.py to use SGLang's /generate endpoint instead of the OpenAI-compatible chat completions API ([msg 3900] context), optimizing server throughput through KV cache tuning, and battling GPU memory constraints. The throughput had improved from an initial ~400 tok/s to a peak of ~1,450 tok/s with FP8 KV cache quantization, before the user rejected FP8 as quality-degrading ([msg 3910]: "Uh don't do fp8 kv cache, that degrades the model noticeably"), forcing a rollback to bf16 KV cache at ~930 tok/s.

At the time of the subject message, the inference pipeline was running on the B2_opencodeinstruct category — the longest and most demanding dataset, with prompts averaging 4,000+ tokens. The server was generating approximately 930 tok/s with 72 concurrent requests and 79 in queue, token usage at 0.96 — hitting the GPU KV cache ceiling again ([msg 3919]). The assistant had just reported that B2 was progressing at roughly 0.4-0.5 requests per second, estimating that generating all 88K samples across all categories would take approximately 57 hours.

Why This Message Was Written

The user's message is a direct response to this time estimate. It reflects a pragmatic, resource-constrained mindset. The user recognizes that:

  1. Time is finite. Waiting 57+ hours for a complete dataset is not acceptable. The pipeline needs to deliver usable results sooner.
  2. Data has diminishing returns. Beyond a certain volume, additional training samples yield marginal improvements. The user implicitly understands that 10 million tokens per category — roughly 2,500 samples at 4K tokens each, or more for shorter categories — is likely sufficient for training a quality draft model.
  3. Tokens are the right unit of measurement. Rather than capping by sample count (which would be uneven across categories of varying lengths), the user proposes a token-based budget. This is a more principled constraint because training data is ultimately measured in tokens, and the model's learning capacity scales with token count, not sample count.
  4. Control over total runtime. At the current throughput (~930 tok/s), 10M tokens per category × 7 categories = 70M tokens, which at ~930 tok/s would take roughly 21 hours of generation time. This is a significant reduction from 57 hours. The user is implicitly doing this arithmetic. The message also reveals the user's role in the conversation. They are not a passive observer — they are actively directing the work, making strategic decisions about resource allocation. The casual phrasing ("Maybe... just do 10M tokens per category?") is characteristic of a high-level decision-maker who trusts the assistant to implement the details.

Assumptions Embedded in the Message

The user's proposal rests on several assumptions, some explicit and some implicit:

That 10M tokens per category is sufficient. This is the most critical assumption. The user believes that 10 million tokens of training data per category will produce a draft model of acceptable quality. This is an empirical question — it depends on the complexity of the task, the diversity of the prompts, and the capacity of the EAGLE-3 architecture. The user may be drawing on experience with similar projects, or simply making a reasonable guess.

That token-based capping is feasible to implement. The user assumes that the assistant can modify run_inference.py to track token output and stop when a per-category budget is reached. This is a reasonable assumption given the assistant's demonstrated ability to modify the codebase, but it requires changes to the data pipeline logic.

That categories are roughly comparable in value. By applying the same 10M token cap to every category, the user implicitly assumes that all categories contribute equally to model performance. If some categories are more valuable than others (e.g., because they contain more diverse reasoning patterns), this uniform cap might be suboptimal.

That the current throughput is representative. The user's time estimate assumes that the server maintains its current throughput (~930 tok/s) across all categories. In practice, shorter categories (B3-B7 with ~1.5K avg tokens) would run faster because more requests fit in the KV cache, potentially reducing total time further.

The Implementation That Followed

The assistant immediately understood the directive. In the very next round ([msg 3921]), the assistant responded: "Good idea. That bounds the total time and ensures we don't waste compute on diminishing returns. Let me add --max-tokens-per-dataset to run_inference.py."

The implementation involved:

  1. Adding a --max-tokens-per-dataset parameter to run_inference.py, defaulting to 10,000,000 tokens per category.
  2. Tracking cumulative output tokens per category during generation, checking after each completed response whether the budget was exceeded.
  3. Stopping generation for a category once its token budget was reached, then moving to the next category.
  4. Reporting token counts in the progress monitor so the user could see how close each category was to its budget. The assistant estimated that with the 10M token cap, total generation would take approximately 17-26 hours instead of 57+ — a 2-3x reduction in wall-clock time. The pipeline was restarted on B2_opencodeinstruct with the capped budget, and the assistant reported it running steadily.

What This Message Reveals About the Collaboration

This message is a textbook example of effective human-AI collaboration. The user provides high-level strategic direction ("cap at 10M tokens per category"), while the assistant handles the implementation details (modifying code, tracking budgets, updating monitoring). The user does not need to specify how to implement the cap — they trust the assistant to figure it out. The assistant does not need to ask why 10M tokens — they understand the reasoning and execute.

The message also demonstrates the value of domain expertise. The user's intuition about token-based budgeting and diminishing returns comes from experience with training large language models. This is not a generic suggestion — it reflects a nuanced understanding of what makes training data useful and how to allocate compute resources efficiently.

Conclusion

A single sentence — "Maybe in each category instead doing the whhole thing just do 10M tokens per category?" — redirected an entire data generation pipeline, saving dozens of hours of compute time while maintaining data quality. The message is remarkable not for its technical sophistication (it has none) but for its strategic clarity. It identifies the core bottleneck (time), proposes a principled solution (token-based capping), and trusts the assistant to execute.

In the broader narrative of the EAGLE-3 training session, this message marks the transition from "generate everything" to "generate enough." It is a moment of pragmatic optimization — a recognition that in machine learning engineering, the perfect is the enemy of the good, and that a well-capped dataset delivered today is worth more than a complete dataset delivered next week.