The Data Pivot: When Fine-Tuning Fails, Scale From Scratch

In the middle of a high-stakes optimization session for Kimi-K2.5 inference throughput, a single message from the AI assistant marks a critical inflection point. Message [msg 5009] is brief—barely a few paragraphs—but it encapsulates the culmination of hours of debugging, training, and analysis. It is the moment when a promising but ultimately failed approach is definitively abandoned, and a new, more ambitious strategy is adopted. The assistant writes:

OK so the K2 init is conclusively worse than from-scratch on this data. The path forward is clear: more training data. Our from-scratch drafter trained on 37K samples reached 75% cond_acc_0 / ~2.0 accept_len. AQ-MedAI's K2 drafter trained on 1.4M samples reached 3.2-3.5 accept_len. The gap is almost certainly data volume.

We need to scale from 37K → 200K+ samples and retrain from scratch. That means:

1. Launch the K2.5 base model (no speculation) for generation 2. Generate diverse samples (math, code, reasoning, multilingual) 3. Extract hidden states via the patched pipeline 4. Train from scratch on the larger dataset

Should I proceed with Phase 2 (scaling up training data), or do you want to explore a different direction?

This message is not merely a status update. It is a reasoned conclusion drawn from empirical evidence, a strategic pivot, and a request for human direction. To understand its significance, we must trace the threads that lead to it: the failed fine-tuning of a pre-trained drafter, the diagnostic journey that revealed a fundamental mismatch, and the quantitative reasoning that pointed to data volume as the decisive variable.

The Context: A Drafter That Wouldn't Converge

The session had been exploring speculative decoding for the Kimi-K2.5 large language model. Speculative decoding accelerates inference by using a small "draft" model to predict multiple tokens ahead, which the large "target" model then verifies in parallel. The key metric is acceptance length—how many draft tokens on average are accepted before the target model needs to intervene. Higher acceptance means faster generation.

The team had already trained a from-scratch EAGLE-3 drafter on 37K samples, achieving 74.7% validation accuracy and ~2.0 accept length. This was decent but far from the 3.2-3.5 accept length reported by AQ-MedAI for their K2 drafter trained on 1.4M samples. The natural next step was to try fine-tuning AQ-MedAI's pre-trained K2 drafter on K2.5 data—a classic transfer learning approach that could save the enormous cost of generating hundreds of thousands of training samples.

What followed was a multi-phase investigation spanning messages [msg 4982] through [msg 5008]. Phase 0 was a direct probe of the AQ-MedAI K2 drafter on K2.5, which achieved ~1.5 accept length and 52 tok/s—barely better than the from-scratch model. Phase 1 was the fine-tuning attempt, which initially produced random-looking loss values (~18-20). The assistant diagnosed this as a critical vocab mapping mismatch: only 252 out of 32,000 draft-to-target token positions matched between the AQ-MedAI and K2.5 vocabularies. Fixing this dropped the loss to ~9 and improved accuracy to ~24%.

But then the fine-tuning plateaued. By epoch 2, the validation accuracy had only reached ~38%—half of what the from-scratch model achieved at epoch 5. The assistant's analysis in [msg 5007] laid out the stark comparison: the K2 fine-tune's cond_acc_0 was stuck at 38.3% from epoch 0 to epoch 1, while cond_acc_1 languished at 3.5%, meaning the autoregressive steps after the first were essentially non-functional. The from-scratch model, by contrast, reached 74.7% cond_acc_0 and 67.0% cond_acc_1 at epoch 5.## The Reasoning: Why Fine-Tuning Failed

The assistant's conclusion that the K2 weights were "conclusively worse than from-scratch" was not a guess—it was the result of careful quantitative comparison. The key insight was that the K2 drafter's internal representations—the fc layer that transforms concatenated hidden states, and the midlayer attention/MLP blocks—had been optimized for K2's hidden state distribution. When presented with K2.5's different hidden states, these weights were not just unhelpful; they were actively harmful, creating a "bad basin" in the loss landscape that gradient descent could not escape.

This is a subtle but important point about transfer learning in neural networks. It is often assumed that pre-trained weights provide a better starting point than random initialization, even when the downstream task differs slightly. But this assumption breaks down when the input representation changes. The EAGLE-3 drafter takes as input the hidden states from the target model's penultimate layer. If K2 and K2.5 produce meaningfully different hidden state distributions (due to different training data, different model architecture details, or different quantization schemes), then the drafter's feature extractor—the fc layer—may be tuned to features that simply do not exist in the new data.

The assistant's diagnostic process in [msg 4995] through [msg 4996] revealed this pattern clearly. The from-scratch model at ~723 batches (17% through epoch 0) had loss_0 ~2.5 and cond_acc_0 ~50%. The K2 fine-tune at the same point had loss_0 ~3.5 and cond_acc_0 ~30%. Despite starting with "better" weights, the fine-tuned model was learning more slowly. By epoch 2, the gap had not closed—the fine-tune was stuck.

The Pivot: Data Volume as the Decisive Variable

The most striking feature of message [msg 5009] is the quantitative comparison that drives the decision. The assistant cites three data points:

From-scratch on 37K samples: 75% cond_acc_0, ~2.0 accept_len
AQ-MedAI K2 on 1.4M samples: 3.2-3.5 accept_len
K2 fine-tune on 37K samples: 38% cond_acc_0 (plateaued) The inference is that the gap between 2.0 and 3.2-3.5 accept_len is primarily driven by data volume, not architecture or initialization. This is a defensible hypothesis: EAGLE-3 drafters are relatively small models (a single transformer layer with an MLP), and their capacity is limited. The marginal benefit of better initialization is likely small compared to the benefit of seeing more diverse training examples. However, this reasoning contains an implicit assumption worth examining. The assistant assumes that the from-scratch model trained on 200K+ samples would approach the performance of AQ-MedAI's K2 drafter. But AQ-MedAI's drafter was trained on K2 data, not K2.5 data. If the fundamental issue is that K2.5's hidden states are harder to predict (perhaps because K2.5 is a more capable model with more diverse outputs), then even 1.4M K2.5 samples might not reach 3.5 accept_len. The data volume hypothesis is plausible but unproven—it is the best available explanation given the evidence.

The Plan: A Multi-Step Pipeline

The assistant proposes a four-step plan that reveals a sophisticated understanding of the full training pipeline:

Step 1: Launch the K2.5 base model without speculation. This is necessary because generating training data requires running the target model to produce both the output tokens and the hidden states. Without speculation, the model runs at full precision (or with standard KV-cache optimization), which is slower but produces the ground-truth hidden states needed for training.

Step 2: Generate diverse samples. The assistant suggests math, code, reasoning, and multilingual data. This diversity is critical for generalization. If the drafter only learns to predict tokens in one domain, it will perform poorly on others. The 37K dataset may have been too narrow, leading to overfitting.

Step 3: Extract hidden states via the patched pipeline. This refers to the infrastructure built in earlier sessions (segments 29-30) where the assistant modified the SGLang inference engine to capture the target model's hidden states during generation. This is a non-trivial engineering task—it requires instrumenting the model's forward pass without breaking the CUDA graph capture that makes SGLang fast.

Step 4: Train from scratch. Starting from random initialization, but on a much larger dataset. The assistant explicitly rejects further attempts at fine-tuning, having learned that the K2 weights are a liability, not an asset.## Assumptions and Potential Pitfalls

Message [msg 5009] rests on several assumptions that deserve scrutiny. The first is that data volume is the binding constraint. The assistant cites the gap between 37K and 1.4M samples as evidence, but this is a correlation, not a causal proof. AQ-MedAI's K2 drafter may have benefited from other factors: better hyperparameters, a different training schedule, or architectural details not captured in the public release. The assistant implicitly assumes that scaling data alone will close the gap, but the actual relationship between data volume and acceptance length for EAGLE-3 drafters is unknown.

The second assumption is that generating 200K+ samples is feasible within the available time and compute budget. The assistant's earlier analysis (in segment 29) showed that generating the B-datasets via OpenRouter cost $86 for 33 minutes of API calls. Scaling to 200K samples would require roughly 5-6x more data than the 37K dataset, which could cost $400-500 and take several hours. This is not prohibitive, but it is a non-trivial commitment of resources.

The third assumption is that the hidden state extraction pipeline can handle the volume. The assistant previously noted (in segment 29) that extracting hidden states for the A1 long sample took significantly longer than expected. Scaling to 200K samples could introduce new bottlenecks in the extraction pipeline, potentially requiring further engineering work.

The Thinking Process: What the Message Reveals

The assistant's reasoning in this message reveals a clear, data-driven decision-making process. Rather than continuing to tweak hyperparameters or try alternative fine-tuning strategies (higher LR, re-initializing layers, adding noise—all options that were discussed in [msg 4999]), the assistant steps back and asks a more fundamental question: what is the actual bottleneck?

This is a hallmark of effective engineering: knowing when to stop optimizing a dead end and pivot to a higher-leverage approach. The assistant had spent hours debugging the vocab mapping, monitoring training curves, and comparing convergence rates. The conclusion that the K2 init was "conclusively worse" was not reached lightly—it was the result of systematic comparison against a known baseline (the from-scratch model).

The message also shows intellectual honesty. The assistant does not try to spin the failure as a partial success or propose marginal improvements that might salvage the approach. Instead, it states the conclusion plainly and proposes a clean break: abandon fine-tuning, scale data, train from scratch. This is the kind of decisive action that separates productive optimization from sunk-cost fallacy.

Input and Output Knowledge

To fully understand this message, the reader needs several pieces of context:

What EAGLE-3 speculative decoding is: A technique where a small draft model predicts multiple tokens ahead, and the large target model verifies them in parallel. The key metrics are acceptance length and conditional accuracy.
The difference between K2 and K2.5: K2 is the base model from Kimi; K2.5 is an improved version. Their hidden state distributions differ enough that a drafter trained on one does not transfer well to the other.
The vocab mapping problem: Different model versions may assign different token IDs to the same text. The lm_head layer maps hidden states to token probabilities, so its rows must correspond to the correct tokens. A mismatch causes random loss.
The training pipeline: Data generation → hidden state extraction → drafter training → deployment with SGLang. Each step has its own infrastructure and failure modes. The message creates new knowledge: a clear decision to scale data, a concrete plan with four steps, and a documented failure mode (K2 weights as bad initialization for K2.5) that future efforts can learn from.

Conclusion

Message [msg 5009] is a turning point. It closes the book on a promising but ultimately unsuccessful approach—fine-tuning a pre-trained drafter from a related model—and opens a new chapter focused on the most fundamental lever in machine learning: more data. The assistant's reasoning is grounded in quantitative evidence, its plan is concrete and actionable, and its request for human direction respects the user's agency in deciding the next move.

In the broader arc of the conversation, this message represents the moment when the team accepts that there are no shortcuts. The K2 drafter's weights, despite being trained on 1.4M samples, could not be repurposed for K2.5. The only reliable path to better speculative decoding performance is to generate the data, extract the hidden states, and train from scratch—the same process that produced the from-scratch model, but scaled up by an order of magnitude. It is not the most elegant solution, but it is the one that the evidence supports.