The 5.5B Parameter Revelation: Inspecting the DFlash Drafter Checkpoint

Introduction

In the middle of a complex debugging session spanning multiple days of DFlash drafter training, a single bash command reveals a critical architectural detail that would ultimately explain a 4× performance gap against a reference model. Message [msg 8877] is deceptively simple: an SSH command that loads a PyTorch checkpoint and prints its structure. But the output it produces—a 5.5-billion-parameter model with a specific fc.weight shape of [5120, 20480]—contains the seed of a discovery that would reshape the entire training strategy.

The Message

The assistant executes the following command via SSH into the remote training host:

ssh -o ConnectTimeout=10 root@10.1.2.6 'pct exec 200 -- bash -c "source /root/venv/bin/activate && python3 -c \"
import torch
ckpt = torch.load(\\\"/workspace/checkpoints/step_20000/checkpoint.pt\\\", map_location=\\\"cpu\\\", weights_only=False)
print(\\\"Keys:\\\", list(ckpt.keys()))
sd = ckpt[\\\"model_state_dict\\\"]
total_params = sum(v.numel() for v in sd.values())
print(\\\"Total params in checkpoint:\\\", total_params / 1e6, \\\"M\\\")
print()
print(\\\"State dict keys (first 30):\\\")
for i, k in enumerate(sorted(sd.keys())[:30]):
    print(\\\"  %s: %s\\\" % (k, sd[k].shape))
print(\\\"...total keys:\\\", len(sd))
print()
print(\\\"Step:\\\", ckpt.get(\\\"step\\\"), \\\"Epoch:\\\", ckpt.get(\\\"epoch\\\"))
\""'

The output reveals:

Keys: ['model_state_dict', 'optimizer_state_dict', 'epoch', 'global_step', 'args']
Total params in checkpoint: 5518.19904 M

State dict keys (first 30):
  embed_tokens.weight: torch.Size([248320, 5120])
  fc.weight: torch.Size([5120, 20480])
  hidden_norm.weight: torch.Size([5120])
  layers.0.input_layernorm.weight: torch.Size([5120])
  layers.0.mlp.down_proj.weight: torch.Size([5120, 17408])
  layers.0.mlp.gate_proj.weight: torch.Size([17408, 5120])
  layers.0.mlp.up_proj.weight: torch.Size([17...

Context and Motivation

This message arrives at a critical juncture. The assistant has been monitoring the DFlash drafter training for hours, analyzing loss curves, computing per-epoch improvement velocities, and comparing metrics against the DFlash paper's reported results (arXiv 2602.06036). The training logs show a troubling pattern: improvement is slowing down dramatically. Between steps 2k–6k, the DDTree-8 streak metric was improving at 0.309 per thousand steps; by steps 18k–22k, that velocity had collapsed to 0.035 per thousand steps—a nearly 9× deceleration.

The user had just asked about testing the latest checkpoint locally to evaluate the drafter's actual performance against the target Qwen3.6-27B model running on the SGLang server (CT129). Before writing inference code, the assistant needs to understand the model's architecture: what parameters it contains, how many layers, what the embedding and projection dimensions are. This checkpoint inspection is the first step toward building an evaluation harness.

But there's a deeper reason for this inspection. The assistant's reasoning in the preceding messages shows it's already suspicious about the architecture. The DFlash paper describes a specific design where the drafter conditions on hidden states from the target model's layers, using a projection layer (fc) to map those hidden states into the drafter's dimension. The number of target layers used as input is a critical architectural decision—and the assistant is about to discover that their implementation might differ from the official one.

The Critical Discovery

The most important detail in the output is the shape of fc.weight: [5120, 20480]. This is the projection layer that maps concatenated hidden states from the target model's layers into the drafter's hidden dimension of 5120. The input dimension of 20480 equals 4 × 5120, meaning the fc layer takes hidden states from exactly 4 target layers.

This is where the architecture diverges from the z-lab reference model. The Qwen3.6-27B target model has 64 layers total. The DFlash design selects a subset of these layers to condition on—typically the last N layers before the final output. With 4 layers feeding into fc, the drafter sees layers 57–60 (the last 4 of 64), but critically, it never sees layer 61—the very last layer, which carries the richest next-token prediction information.

The z-lab model, by contrast, uses all 5 target layers (25600 → 5120), concatenating layers 57–61 and injecting them into every drafter layer's KV cache. Layer 61, being closest to the output head, contains the most refined representation of what token should come next. By excluding it from the conditioning signal, the assistant's drafter is operating with fundamentally less informative inputs.

This architectural discrepancy would later be confirmed as one of three critical bugs (as documented in [chunk 52.1]): the fc layer was using only 4 layers when it should have been using 5, creating a shortcut where the target layer's information was simultaneously absent from conditioning and present in the loss computation—a structural flaw that fundamentally limited the model's ceiling.

Assumptions and Knowledge Required

To interpret this message, one needs substantial context. The reader must understand that DFlash is a block diffusion method for speculative decoding, where a lightweight "drafter" model predicts multiple tokens in parallel, conditioned on hidden states extracted from the target LLM. The fc (fully connected) layer is the bridge between these two models, projecting the target's high-dimensional hidden states into the drafter's embedding space.

The vocabulary size of 248,320 reveals that this drafter uses the same vocabulary as Qwen3.6-27B—a massive embedding table that accounts for a significant fraction of the 5.5B parameters. The hidden dimension of 5120 matches the drafter's internal representation size, and the MLP intermediate dimension of 17408 follows the standard SwiGLU architecture pattern (roughly 8/3 × hidden_dim).

One must also understand the training infrastructure: the command chains through SSH to a Proxmox host, then into an LXC container (ID 200), activates a Python virtual environment, and only then runs the PyTorch code. This layered access pattern reflects the complex multi-host setup where the training runs inside a containerized environment on dedicated GPU hardware.

The Thinking Process

The assistant's reasoning shows a methodical approach to model inspection. Rather than guessing at the architecture from code, it directly loads the checkpoint and examines the actual saved parameters. The choice to print only the first 30 keys is pragmatic—the full state dict likely contains hundreds of keys (each layer has multiple weight matrices), and the first 30 provide a representative sample covering embeddings, the fc projection, normalization, and the first layer's MLP weights.

The command also checks the total parameter count (5.5B), which serves as a sanity check against the expected architecture. A drafter for a 27B target model should be substantially smaller—5.5B is roughly 20% of the target's size, which is reasonable for a speculative decoding draft model that needs to be fast enough to provide meaningful speedup.

Notably, the assistant doesn't yet draw the conclusion about the 4-layer vs 5-layer discrepancy from this output alone. That insight would come later, after building the evaluation harness and comparing against the z-lab model's performance. At this moment, the assistant is simply gathering data—recording the architectural facts that will become crucial evidence in the subsequent investigation.

Output Knowledge Created

This message produces concrete, verifiable knowledge about the model architecture:

Conclusion

Message [msg 8877] appears, on its surface, to be a routine checkpoint inspection—the kind of diagnostic command that experienced ML practitioners run dozens of times per project. But within its output lies the key to understanding why the DFlash drafter was plateauing far below its potential. The fc.weight shape of [5120, 20480] silently encodes a design decision—using 4 target layers instead of 5—that would later be identified as one of three critical bugs requiring a complete training restart. It's a powerful reminder that in machine learning engineering, the most important discoveries often hide in plain sight, waiting in the shapes of weight tensors that we've been staring at all along.