Chunk 52.0

## Summary This chunk focused on building a comprehensive evaluation infrastructure to compare the DFlash drafter's training progress against both the DFlash paper's reported metrics and the z-lab/Qwen3.6-27B-DFlash model. The user set up a full eval harness on CT129 (the SGLang server) that loads the target Qwen3.6-27B model on GPU with the `fla` library for correct linear attention, extracts hidden states from 10 fresh coding prompts, and runs the drafter inference using a reimplemented standard attention mechanism (since `flex_attention` requires CUDA). A critical discovery was that CPU-based hidden state extraction (using PyTorch's fallback for linear attention) produced numerically different results from the `fla`-based extraction used during training—4 of 5 target layers use Qwen3.5's linear attention, and the bf16 numerical differences caused completely garbled drafter output. Switching to GPU extraction with `fla` fixed this, revealing the model's true performance. The side-by-side comparison was stark: at step 20k (epoch 1.7), our model achieved τ≈3.0 DDTree-8 on fresh coding prompts, while the z-lab model achieved τ≈12.4—a 4x gap. The root cause was traced to an architectural difference: our `fc` projection only uses 4 target layers (20480→5120), reserving layer 61 exclusively for verifier loss computation, while z-lab concatenates all 5 target layers (25600→5120) and injects them into every drafter layer's KV cache. Layer 61 (near the last of 64 layers) carries the richest next-token information, and our model never sees it at inference time. Additional training log analysis showed tiny gradient norms (mean 0.06 after warmup, no clipping), a bimodal loss distribution, and no accuracy difference between short and long sequences. The user decided to abandon the current run (epoch 1.93 of 6) and restart with a fixed architecture: expanding `fc` to use all 5 layers, removing the separate verifier head in favor of computing target logits directly from the model output, eliminating the noise schedule (which was hurting convergence), and potentially reducing gamma from 10 to the paper's default 7. The training scripts were git-initialized and committed before making changes, with the user explicitly rejecting sunk cost fallacy in favor of fixing the fundamental architecture gap.

The Evaluation That Changed Everything: Building Infrastructure to Uncover Three Critical Bugs in DFlash Drafter Training 2593 words

Message Articles

Subagent Sessions