The Architecture of Frozen Knowledge: Understanding Parameter Freezing in EAGLE-3 Draft Model Training

In the midst of a sprawling, multi-day machine learning engineering session — one that had already survived a VM crash, a disk migration, 4.6 terabytes of hidden state extraction, and countless debugging cycles — a single question from the user stopped the momentum cold: "Before starting train - are we correctly training the whole model, embeddings etc or is anything frozen?" The question arrived at message index 4231, just as the assistant was preparing to launch a 10-hour training run on 8 NVIDIA RTX PRO 6000 Blackwell GPUs. It was the kind of question that, if left unasked, could have wasted days of compute on a fundamentally broken training configuration.

The assistant's response at <msg id=4233> is a masterclass in architectural clarity under pressure. Let us examine it in detail.

The Message: A Complete Parameter Map

The assistant's reply begins with a clean, structured breakdown:

Frozen (requires_grad = False): - embed_tokens (line 96) — verifier's embedding table, 163,840 × 7168 = 1,174.4M params - verifier_lm_head (line 117) — frozen copy of verifier's lm_head (32,000 × 7168 = 229.4M params), used as a reference/teacher signal

>

Trainable (requires_grad = True, default): - lm_head — draft lm_head, 32,000 × 7168 = 229.4M params — initialized from verifier but trained - fc — the linear projection that maps concatenated hidden states to model dim (inside Eagle3DraftModel) - layers[0] — single Llama decoder layer (attention + MLP)

This is not merely a list. It is a carefully reasoned argument, presented with parameter counts, line numbers, and architectural rationale for each decision. The message then provides a four-point rationale section, each bullet explaining why a given component is frozen or trainable, and concludes with an empirical verification step — checking the upstream speculators library to confirm the convention matches.

Why This Message Was Written: The Reasoning and Motivation

The immediate trigger was the user's question, but the deeper motivation is more interesting. The user was not asking out of idle curiosity. They were asking because they understood a fundamental truth about speculative decoding training: if the wrong parameters are frozen, the entire training run is wasted. Training a draft model that cannot learn to predict the verifier's hidden states because its embedding table is being updated (and thus drifting from the verifier's embeddings) would produce a drafter that fails at inference time. Conversely, freezing the wrong parameters — like the draft LM head — would prevent the model from adapting its output distribution to the draft role.

The assistant's response reveals an implicit understanding that the user needed not just a yes/no answer, but a pedagogical explanation. The assistant could have simply said "Yes, the freezing is correct." Instead, it chose to enumerate every parameter group with its status, size, and rationale. This choice reflects a recognition that in a research engineering context, the reasoning behind a configuration is as important as the configuration itself. If the user disagreed with any single rationale, they could catch a design flaw before the 10-hour training run began.

How Decisions Were Made: Tracing the Code

The decision process visible in this message is a two-step verification pipeline. First, the assistant consulted the training script (04_train.py) that had been SCP'd to the remote machine in the preceding message. The assistant had already read the file in <msg id=4232>, locating the _patched_setup_embeddings_and_lm_heads function. This function is a replacement for the standard Eagle3DraftModel._setup_embeddings_and_lm_heads, modified to use pre-loaded verifier weights instead of calling load_model_layers. Within that function, the assistant identified the specific requires_grad assignments:

• Line 96: self.embed_tokens.weight.requires_grad = False
• Line 117: self.verifier_lm_head.weight.requires_grad = False The remaining components (lm_head, fc, layers[0]) were left at their default requires_grad = True state, which is the PyTorch default for newly created parameters. But the assistant did not stop at reading its own training script. It then performed a cross-validation step, SSHing into the remote machine to check the upstream speculators library:

grep -n "requires_grad" ~/ml-env/lib/python3.12/site-packages/speculators/models/eagle3/core.py

This returned lines 258 and 280, confirming that the official speculators implementation also freezes embed_tokens and verifier_lm_head. This dual verification — checking both the custom training script and the upstream library — demonstrates a rigorous engineering mindset. The assistant was not merely asserting correctness; it was providing evidence.

Assumptions Made by the Assistant

Several assumptions underpin this message, and examining them reveals the conceptual framework within which the assistant operates.

Assumption 1: The EAGLE-3 paper's conventions are authoritative. The assistant repeatedly invokes "the EAGLE-3 paper and the speculators library convention" as the ultimate justification for the freezing decisions. This assumes that the paper's architecture is optimal for this specific use case (Kimi-K2.5 with a reduced 32K vocabulary). In practice, this is a reasonable assumption — the paper underwent peer review and the speculators library is a maintained implementation — but it is worth noting that the assistant does not independently verify whether, say, fine-tuning the embedding table with a small learning rate might improve performance. The assumption is that the paper's design decisions are correct by default.

Assumption 2: The verifier's embedding table and LM head are perfect teachers. The rationale for freezing embed_tokens is that "the drafter receives token IDs and needs to produce the same embeddings as the verifier. Training these would cause a mismatch since the verifier's embeddings are fixed." This assumes that the verifier's embeddings are indeed the ground truth that the drafter should match. In the EAGLE-3 framework, this is architecturally correct — the drafter's embeddings are fed into the verifier's frozen trunk during inference, so they must be identical. However, the assumption could break if the verifier itself is later fine-tuned, requiring a corresponding update to the drafter's embeddings.

Assumption 3: The verifier_lm_head is used as a teacher signal via KL divergence or similar. The assistant states this parenthetically, indicating some uncertainty about the exact loss function. This is a reasonable inference — in speculative decoding training, the draft model's output distribution is typically trained to match the verifier's distribution via KL divergence — but the assistant does not verify the actual loss function in the training script. If the training script used a different loss formulation (e.g., cross-entropy against hard targets), the frozen verifier_lm_head might serve a different purpose or be unnecessary.

Assumption 4: The single decoder layer (layers[0]) is the core of what EAGLE-3 learns. This is architecturally accurate — EAGLE-3's innovation is a lightweight transformer layer that predicts future hidden states from a concatenation of current hidden states and token embeddings — but the assistant does not consider whether additional layers or a different architecture might be beneficial for the Kimi-K2.5 model specifically.

Potential Mistakes or Incorrect Assumptions

The most notable potential issue in this message is the implicit assumption that the upstream speculators library's conventions are identical to what the training script implements. The assistant checks that speculators also freezes embed_tokens and verifier_lm_head, but it does not verify that the training script's other components (lm_head, fc, layers[0]) match the upstream implementation's trainable parameter set. A discrepancy here — for example, if speculators additionally froze the lm_head or made fc non-trainable — would indicate a bug in the training script that the assistant missed.

Additionally, the assistant's parameter counts contain a subtle inconsistency. The embed_tokens is listed as 163,840 × 7168 = 1,174.4M parameters, but the lm_head is listed as 32,000 × 7168 = 229.4M parameters. The vocabulary sizes differ (163,840 vs 32,000) because the training script uses a vocabulary mapping (t2d and d2t tensors) that reduces the full verifier vocabulary to a smaller draft vocabulary. The assistant does not explain this mapping in the message, which could confuse a reader who notices the discrepancy. The 163,840 vocabulary corresponds to the full Kimi-K2.5 vocabulary (likely including multi-byte tokenizations), while the 32,000 is the reduced vocabulary used by the drafter after applying the mapping.

Input Knowledge Required to Understand This Message

To fully grasp this message, a reader needs:

1. Understanding of speculative decoding (EAGLE-3): Knowledge that a draft model predicts hidden states several steps ahead, which are then verified by the full model. The drafter must produce embeddings identical to the verifier's, hence the frozen embedding table.
2. Familiarity with PyTorch's requires_grad mechanism: Understanding that freezing parameters prevents gradient computation and weight updates during training.
3. Knowledge of transformer architecture: Understanding what an embedding table, LM head, decoder layer (attention + MLP), and linear projection (fc) are, and how they interact.
4. Context about the Kimi-K2.5 model: Understanding that this is a Mixture-of-Experts model with a large vocabulary (163,840 tokens), and that the training pipeline uses a vocabulary mapping to reduce the output space to 32,000 tokens for the drafter.
5. Awareness of the training data pipeline: The hidden states extracted in the preceding steps (4.6 TB, 37,312 samples) contain the verifier's internal representations that the drafter will learn to predict.

Output Knowledge Created by This Message

This message produces several forms of knowledge:

Explicit knowledge: A complete, enumerated map of every parameter group in the EAGLE-3 training configuration, with its frozen/trainable status, parameter count, and line number in the source code.

Rationale knowledge: A four-point explanation of why each parameter group is configured as it is, connecting architectural decisions to the underlying theory of speculative decoding.

Verification knowledge: Empirical confirmation that the training script's configuration matches the upstream speculators library, provided via a live SSH command and its output.

Trust knowledge: The user can now proceed with the 10-hour training run with confidence that the configuration is architecturally sound. This is perhaps the most important output — the message converts uncertainty into certainty, enabling forward progress.

The Thinking Process Visible in the Reasoning

The assistant's thinking process, while not explicitly shown in a "reasoning" block, is clearly visible in the structure of the response. It follows a systematic pattern:

1. Identify the question: The user asks about frozen parameters. The assistant recognizes this as a correctness check, not a factual query.
2. Read the source code: The assistant reads 04_train.py to find the actual requires_grad assignments (visible in <msg id=4232>).
3. Categorize by status: The assistant separates parameters into "Frozen" and "Trainable" groups, a natural binary classification.
4. Quantify each group: For each frozen/trainable parameter, the assistant computes the parameter count by multiplying dimensions. This serves two purposes: it demonstrates the scale of what's being frozen (1.4 billion parameters frozen vs. ~229M+ trainable), and it provides a sanity check — the numbers should be plausible for the architecture.
5. Provide architectural rationale: For each parameter, the assistant explains why it is configured that way, connecting to the EAGLE-3 paper's design principles.
6. Verify against upstream: The assistant does not trust its own training script alone. It SSHes into the remote machine to check the speculators library's implementation, confirming that the convention is standard.
7. Present results clearly: The final message uses markdown formatting (bold headers, bullet lists, code formatting for parameter names) to make the information scannable and digestible. This thinking process reveals an engineer who operates with a "trust but verify" mentality. The training script was written by the assistant itself (or by the user), yet the assistant still checks it against the upstream library. This is the hallmark of a mature engineering practice: never assume your own code is correct without verification.

Conclusion

Message <msg id=4233> is a deceptively simple response that accomplishes several things at once. It answers a direct question, provides pedagogical depth, verifies against upstream sources, and builds trust for the critical training run ahead. In the broader context of the session — which spans VM crashes, disk migrations, 4.6 TB of data extraction, and multi-GPU training — this message represents a moment of architectural reflection before a significant commitment of compute resources. The assistant's structured, evidence-based response transformed a potential point of failure into a point of confidence, enabling the project to move forward with clarity about what was being learned and what was being held fixed.