The Data Scaling Question: A Strategic Pivot Point in EAGLE-3 Training

"Don't we just have ~10-20M tokens of data for a 1B model? Do we want to get maybe 5-10x more data?"

This short question, posed by the user at message index 3486 in a long-running coding session, is one of those rare moments where a single sentence fundamentally reorients the trajectory of a machine learning project. On its surface, it is a simple factual check and a tentative suggestion. Beneath that surface, it reveals a sophisticated understanding of neural scaling laws, a strategic awareness of resource allocation, and a willingness to pause the current workflow to question whether the entire approach is built on an insufficient foundation.

The Moment of Intervention

To understand why this message was written, we must examine the context in which it arrived. The session had been running for hours, building an EAGLE-3 speculative decoding draft model for the Kimi-K2.5 large language model — a 1-trillion-parameter Mixture-of-Experts architecture running on eight NVIDIA RTX PRO 6000 Blackwell GPUs. The assistant had just completed the second round of training, this time using hidden states extracted from SGLang rather than vLLM, after the first round produced a draft model with a catastrophic 25% acceptance rate (effectively zero useful draft tokens).

In the message immediately preceding the user's question ([msg 3485]), the assistant reported that training was 74% through epoch 4 of 5, with validation metrics showing clear signs of diminishing returns:

| Epoch | Total Loss | Step 0 Acc | |-------|-----------|-----------| | 1 | 6.276 | 74.0% | | 2 | 6.138 | 74.4% | | 3 | 6.127 | 74.5% |

The loss had plateaued. The accuracy had stagnated. The assistant's plan was straightforward: wait for training to finish, kill the process, launch SGLang with the new checkpoint, and benchmark the acceptance rate. This is the standard engineer's approach — complete the current cycle, measure the result, then iterate.

The user's intervention breaks this cycle. Rather than waiting 20 minutes for training to finish and then spending more time benchmarking a potentially data-starved model, the user asks a more fundamental question: is the data itself the problem?

The Reasoning Behind the Question

The user's reasoning draws on a well-established principle in modern deep learning: the relationship between model size and training data quantity. The draft model has approximately 1.2 billion trainable parameters (2.6B total, with 1.4B frozen embedding and language modeling head layers). The training dataset consists of 10,000 samples, totaling roughly 21 million tokens.

The Chinchilla scaling law, published by Hoffmann et al. in 2022, established that for optimal training of large language models, the number of training tokens should be roughly 20 times the number of parameters. By this heuristic, a 1.2B parameter model would need approximately 24 billion tokens — over 1,000 times what the current dataset provides.

However, the user is not applying the Chinchilla heuristic naively. EAGLE-3 training is not language modeling from scratch; it is a much simpler task: given rich hidden state representations from the target model, predict the next token. The draft model receives a concatenation of hidden states from three intermediate layers of the target model (layers 2, 30, and 58 out of 61 total), which already encode enormous amounts of contextual information. This means the draft model needs far less data than a full language model trained from scratch.

The user's intuition is instead guided by the EAGLE-3 paper's own scaling experiments. The paper's Figure 1 shows a clear scaling law: acceptance rate improves monotonically as training data increases from 68,000 samples (ShareGPT only) up to 530,000 samples (ShareGPT + UltraChat), with no sign of saturation even at the maximum data quantity. The paper's best results used roughly 25–50 times more unique samples than the current 10,000.

The user's question — "Do we want to get maybe 5-10x more data?" — is therefore grounded in the paper's own empirical findings. If the EAGLE-3 authors saw gains all the way to 8× their baseline data quantity, and our baseline is already much smaller than theirs, then scaling up by 5–10× could yield substantial improvements.

Assumptions Embedded in the Question

The user makes several assumptions, some explicit and some implicit:

1. The model is data-limited, not capacity-limited. This is the central assumption. The user believes that the diminishing returns in validation loss are caused by the model repeatedly seeing the same data, not by the model reaching its capacity ceiling. If the model were already at its capacity limit, adding more data would not help — the model would simply memorize more patterns without improving generalization. The EAGLE-3 paper's scaling curves support this assumption, but it is not proven for this specific architecture and training setup.

2. Disk is not a constraint. The user explicitly states this assumption: "Assuming disk is not a constraint (it's not bc it's a VM with huge ceph behind)." This turns out to be partially incorrect. While the underlying Ceph storage is large, the hidden state data is enormous — approximately 57 KB per token at bf16 precision for four layers of 7168-dimensional vectors. The current 10,000 samples occupy 924 GB. Scaling to 50,000 samples would require approximately 4.6 TB, and 100,000 samples would need roughly 9.2 TB. The VM's /data volume has only 1.8 TB free. The assistant's subsequent analysis ([msg 3492]) identifies this tension and proposes workarounds.

3. More data will translate to higher acceptance rate. This is the critical causal link. The user assumes that the bottleneck in speculative decoding performance is the draft model's accuracy, and that accuracy is primarily limited by training data quantity. If the low acceptance rate has other root causes — such as the hidden state mismatch between training and inference that was later discovered — then adding more data would not help.

4. Generating more data is feasible within the project's time and resource budget. The assistant calculates that 5× more data would require approximately 10 hours of extraction time, and 10× would need about 20 hours. This assumes the SGLang server can run uninterrupted for that duration, which is reasonable but not guaranteed.

Knowledge Required to Understand This Message

A reader needs several pieces of knowledge to fully grasp the significance of this question:

Scaling laws for neural language models. The user is drawing on the now-established principle that model performance follows a power-law relationship with training data quantity, up to a saturation point. The Chinchilla paper is the canonical reference, but the user is applying a more nuanced version specific to the EAGLE-3 setting.

The EAGLE-3 architecture and training procedure. Understanding why a draft model needs less data than a full language model requires knowing that the draft model receives rich hidden state representations from the target model, which already encode substantial contextual information. The draft model's task — predict the next token given these features — is significantly simpler than learning language from raw text.

The practical constraints of hidden state extraction. Each token generates approximately 57 KB of hidden state data at bf16 precision. This means data quantity is not just about the number of samples but about disk space, I/O bandwidth, and extraction time. The user's assumption about disk being unconstrained is reasonable for the underlying storage system but misses the practical limitation of the VM's mounted volume.

The distinction between unique tokens and token-passes (epochs). The current training runs for 5 epochs, meaning the model sees each sample 5 times. This is 105 million token-passes total, but only 21 million unique tokens. The user is asking about unique data, not total training steps. This distinction is crucial: adding more epochs on the same data would likely produce diminishing returns (as the validation metrics already show), while adding more unique data could unlock new generalization.

Knowledge Created by This Message

The immediate consequence of this question is a detailed analysis by the assistant ([msg 3488] through [msg 3492]) that:

Quantifies the exact data situation. The assistant counts samples (10,000) and total tokens (21,033,536) precisely, confirming the user's estimate.
Researches the EAGLE-3 paper's data scaling. The assistant fetches and reads the paper, finding that the authors used 532,000 samples for their best results and observed scaling gains up to 8× their baseline.
Identifies the disk constraint. The assistant calculates that 5× more data would require ~4.6 TB, exceeding the available 1.8 TB free on /data.
Proposes concrete options for scaling. The assistant suggests three approaches: using SpecForge's online training mode (generating hidden states on-the-fly during training), extracting in batches with deletion between rounds, or reducing hidden state precision.
Recommends a pragmatic path forward. Rather than immediately scaling up, the assistant suggests finishing the current training run, benchmarking the checkpoint, and using the acceptance rate result to decide whether data quantity is indeed the bottleneck. This analysis transforms the user's intuitive question into actionable intelligence. The project now has a clear picture of the data scaling landscape: the costs, the constraints, and the expected benefits.

The Thinking Process Visible in the Question

The user's question reveals a particular mode of strategic thinking. Rather than asking "what's the next step?" — which would be the natural follow-up to the assistant's status report — the user steps back and asks "are we even on the right path?" This is the difference between tactical execution and strategic oversight.

The user is also thinking in terms of resource economics. The question implicitly weighs the cost of generating 5–10× more data (10–20 hours of extraction time, disk space, compute) against the expected benefit (higher acceptance rate, faster inference). The user is not demanding that more data be generated immediately; they are asking whether it should be, inviting a cost-benefit analysis.

There is also a subtle pedagogical element. The user could have simply said "generate more data." Instead, they pose it as a question, prompting the assistant to think through the problem independently. This is consistent with the session's collaborative dynamic, where the user often guides the assistant toward deeper analysis rather than issuing direct commands.

The Broader Significance

This message represents a critical juncture in the project. The decision about data scaling — whether to proceed with the current checkpoint or invest in generating more data — would determine the trajectory of the entire EAGLE-3 effort. As it turned out, the subsequent benchmarking revealed that the trained draft model achieved zero acceptance rate due to a fundamental hidden state dimensionality mismatch (the draft model received 7168-dimensional single-layer hidden states instead of the expected 21504-dimensional concatenation of three auxiliary layers). This deeper architectural issue meant that data quantity was not the primary bottleneck after all — the problem was in the inference pipeline's failure to properly activate the auxiliary hidden state capture mechanism for the KimiK25 model.

The user's question about data scaling, while ultimately addressing a secondary issue, was nonetheless correct in its diagnosis of diminishing returns. The validation metrics genuinely showed signs of data limitation. The fact that a more fundamental bug was also present does not diminish the value of the question — it simply means that fixing the data pipeline alone would not have solved the problem.

In the end, this message exemplifies what makes effective human-AI collaboration in research settings: the human brings strategic intuition about scaling laws and resource allocation, while the AI brings the ability to rapidly quantify constraints, research literature, and compute trade-offs. The question itself — brief, precise, and grounded in deep understanding — is a model of how to intervene productively in an ongoing technical workflow.