The Verdict on Fine-Tuning: When Transfer Learning Fails

{
  "timestamp": "08:01:55",
  "epoch": 2,
  "loss_0": 3.211,
  "cond_acc_0": 0.289,
  "loss": 26.544
}

On its surface, message <msg id=5002> appears to be one of the most mundane actions in any machine learning workflow: a simple bash command to check training logs. The assistant runs grep on a remote server to pull the last five lines of metrics from a fine-tuning run. But this message is anything but mundane. It is the moment of reckoning—the point where weeks of effort, multiple debugging sessions, and a carefully reasoned transfer-learning strategy collide with an unforgiving empirical reality.

The message reads in full:

[assistant] [bash] ssh root@10.1.230.174 'grep "speculators.metrics" /data/eagle3/synth_100k/logs/train_finetune_aqmedai_v2.log | tail -5' 2>&1
08:01:55 [speculators.metrics] {'train': {'loss_0': 3.2114696502685547, 'full_acc_0': 0.28878647089004517, 'cond_acc_0': 0.28878647089004517, 'loss_1': 4.849974632263184, 'full_acc_1': 0.009216589853167534, 'cond_acc_1': 0.03191489353775978, 'loss_2': 5.734766960144043, 'full_acc_2': 0.0, 'cond_acc_2': 0.0, 'loss_3': 6.193548202514648, 'full_acc_3': 0.0, 'cond_acc_3': 0.0, 'loss_4': 6.554019451141357, 'full_acc_4': 0.0, 'cond_acc_4': 0.0, 'loss': 26.543779373168945}, 'epoch': 2, 'lr': 2.53549528...

This single data point—epoch 2, loss_0 of 3.21, conditional accuracy of 28.9%—would become the decisive evidence that led to abandoning the entire fine-tuning approach. To understand why, we need to trace the reasoning chain that brought the assistant to this command, and the devastating comparison it enabled.

The Context: A Carefully Constructed Transfer Learning Hypothesis

The session's trajectory had been building toward this moment for hours. The user and assistant had been attempting to deploy Kimi-K2.5 with speculative decoding using an EAGLE-3 drafter. After training a from-scratch drafter that achieved 74.7% validation accuracy and ~94 tok/s, they sought to improve further by fine-tuning an existing drafter: the AQ-MedAI K2 EAGLE-3 model, which had been trained for the earlier Kimi-K2 architecture.

The logic was sound. Transfer learning from a related model should, in theory, accelerate convergence. The K2 drafter already knew how to predict tokens—it just needed to adapt from K2's hidden state distribution to K2.5's. The assistant had invested significant effort in making this work: diagnosing a critical vocab mapping mismatch (only 252 out of 32,000 token positions matched between AQ-MedAI's mapping and the K2.5 mapping), building a permutation matrix to remap the lm_head weights, and launching a fine-tuning run with carefully chosen hyperparameters (learning rate 5e-5, cosine scheduler, 5 epochs).

The user's skepticism was already visible in the preceding messages. At <msg id=4997>, they asked: "If it converges too slow should we have some weight decay/initial randomness to bootstrap?" At <msg id=5001>, they stated flatly: "Look at progress so far, seems not amazing." The assistant had initially defended the approach, arguing at <msg id=5000> that "the K2 init might just need more time to unlearn K2-specific patterns—by epoch 2-3 it could surpass from-scratch since the midlayer and lm_head carry useful general 'how to predict next tokens' knowledge."

Message <msg id=5002> is the assistant's response to that user skepticism—a data-gathering operation to determine whether the defense was justified.## The Data That Killed the Hypothesis

The metrics returned at epoch 2 tell a stark story. The conditional accuracy for the first draft token (cond_acc_0) stands at 28.9%. For comparison, the from-scratch model—trained from random initialization with no prior knowledge—had reached approximately 50% accuracy at a similar point in epoch 0, and would go on to achieve 74.7% by epoch 5. The fine-tuned model, despite starting with a pre-trained drafter that already knew how to predict tokens for a related architecture, was not only slower to converge but was on track to plateau far below the from-scratch baseline.

Even more telling is the behavior of the higher-order draft tokens (positions 1 through 4). The conditional accuracy for position 1 is a mere 3.2%, and positions 2 through 4 show zero accuracy. This means the drafter can sometimes guess the first token correctly but fails entirely at multi-token prediction—the very capability that makes EAGLE-3 speculation valuable. The from-scratch model, by contrast, showed meaningful accuracy at higher positions by this stage of training.

The loss values reinforce the picture. loss_0 (the primary prediction loss) is 3.21, while the cumulative loss across all five draft positions is 26.54. The from-scratch model at epoch 2 had a total loss well below 20. The fine-tuned model is simply not learning as effectively.

Why Transfer Learning Failed: A Deeper Analysis

The assistant's earlier diagnosis had been precise: the vocab mapping mismatch meant that the AQ-MedAI lm_head weights were initially predicting the wrong tokens entirely. The fix—remapping 25,750 of the 32,000 rows via a permutation matrix—was elegant and correct. But it addressed only the surface-level problem. The deeper issue was that the entire internal representation of the K2 drafter—the feature projection layer (fc) and the midlayer transformer blocks—had been optimized for K2's hidden state distribution, not K2.5's.

The EAGLE-3 architecture works by taking the target model's hidden states (the internal representations from the last layer of the transformer) and using them as conditioning signals to predict multiple draft tokens in parallel. When the base model changes from K2 to K2.5, the hidden state distribution shifts. The fc layer, which maps a concatenation of three hidden state vectors (21,504 dimensions) down to the model dimension (7,168 dimensions), was trained to extract features from K2's hidden states. Feeding K2.5's hidden states through this layer produces a different feature distribution—one that the midlayer attention and MLP blocks were not trained to handle.

This is a classic problem in transfer learning for generative models: the "shallow" layers that process input features are often the most model-specific, while "deep" layers that perform abstract reasoning transfer better. In this case, the fc layer—the very first transformation applied to the hidden states—was the most K2-specific component. By keeping it frozen (or adapting it slowly with a low learning rate), the fine-tuning was essentially trying to force K2.5's hidden states through a K2-shaped bottleneck.

The Assumptions That Underpinned the Strategy

Several assumptions were baked into the fine-tuning approach, and message <msg id=5002> reveals that most of them were incorrect:

Assumption 1: The K2 drafter's internal representations are sufficiently general to transfer to K2.5. This proved false. The 28.9% accuracy at epoch 2, compared to ~50% for from-scratch at the same point, suggests the K2 representations were actively harmful—the model had to unlearn K2-specific patterns before it could learn K2.5-specific ones.

Assumption 2: Fine-tuning with a low learning rate (5e-5) would allow gradual adaptation without catastrophic forgetting. The data suggests the opposite problem: the learning rate was too low to escape the K2 basin. The model was stuck in a local optimum that was good for K2 but poor for K2.5.

Assumption 3: The midlayer (attention and MLP blocks) would transfer well even if the fc layer didn't. The near-zero accuracy at higher draft positions suggests that the midlayer was also producing K2-appropriate features that didn't generalize.

Assumption 4: More training data would eventually overcome the initialization mismatch. The user had previously expanded the dataset from 10K to 100K samples. But the fine-tuning plateau suggested that data volume alone couldn't compensate for a fundamentally misaligned initialization.

The Reasoning Process Visible in the Message

The assistant's thinking in this message is implicit but clear. By choosing to grep the last five lines at epoch 2, the assistant is performing a specific diagnostic: checking whether the model has converged to a stable plateau or is still improving. The learning rate at this point is 2.54e-5—roughly half the peak value, descending on a cosine schedule. If the model were going to "catch up" to from-scratch performance, it would need to show accelerating improvement during the peak LR window (around epoch 0.5-1.5). By epoch 2, the LR is already declining, and the model has had ample time to adapt.

The assistant is also implicitly comparing these numbers to the from-scratch baseline from the previous training run. The from-scratch model at epoch 2 had a cond_acc_0 well above 60% and was still climbing. The fine-tuned model at 28.9% with a declining learning rate is not going to close that gap.

The Output Knowledge Created

This message produces a single, decisive piece of knowledge: the AQ-MedAI K2 drafter is not a viable initialization for K2.5 fine-tuning. The 28.9% accuracy at epoch 2, combined with the trajectory (plateauing rather than accelerating), provides sufficient evidence to abandon this approach.

This conclusion has immediate practical consequences. It means the assistant must pivot to a fundamentally different strategy: either (a) train a from-scratch drafter with more data, (b) explore system-level optimizations to make the existing drafter faster, or (c) investigate alternative speculation methods like n-gram speculation. The session's subsequent trajectory confirms this pivot—the assistant would go on to abandon fine-tuning, test n-gram speculation (which also failed), and ultimately focus on reducing the NCCL all-reduce overhead in the verify step.

The Broader Lesson

Message <msg id=5002> is a case study in the importance of empirical validation in ML engineering. The assistant had constructed a compelling logical argument for why fine-tuning should work: the K2 drafter already knows how to predict tokens, the vocab mapping fix addresses the label mismatch, and the midlayer should transfer well. But the numbers told a different story. The 28.9% accuracy was not just a number—it was a signal that the underlying assumptions about representation transfer were wrong.

In the end, the most valuable skill demonstrated here is not the ability to build a clever permutation matrix or debug a vocab mismatch. It is the willingness to check the data, accept the verdict, and pivot. The assistant could have rationalized the poor numbers—"it needs more epochs," "the LR hasn't peaked yet," "the validation set might be noisy"—but instead, the message serves as the evidence-gathering step that enables a clear decision. Sometimes the most important thing a model (or a person) can do is admit that a hypothesis was wrong and move on.