Chunk 52.1

## Summary This chunk focused on a deep investigation into why the DFlash training was plateauing, leading to the discovery of three critical bugs by comparing against the official speculators repository. First, the user verified that hidden state differences between torch fallback and fla were negligible (cosine similarity 0.9999+), ruling out that as a cause. Then, a detailed code comparison revealed: (1) **noise corrupting target logits** — noise was applied to the combined 5-layer hidden state tensor before extracting the last layer for target logit computation, directly corrupting the training signal; (2) **fc including the target layer** — the official code uses (N-1) layers for context injection while keeping the last layer exclusively for target logits, but our implementation fed all N layers to fc, creating a shortcut where the same information appeared in both conditioning and loss target; (3) **loss function mismatch** — the official DFlash uses pure hard cross-entropy loss with gamma=4.0, while we used 70% soft KL divergence (T=2.0) + 30% CE + streak-aware weighting + gamma=10, which diluted the gradient by forcing the model to match the full 248K-dim distribution instead of just getting the top-1 token correct. Based on these findings, v4 was abandoned at step 5,400 and v5 was launched with all three fixes: hidden states are now split so noise only applies to the fc input (4 layers) while the verifier last layer stays clean, the fc was reverted to 4-layer input matching the official architecture, and the loss was switched to pure hard CE with gamma=7.0 (the paper's recommended value for block_size=16). Soft KL, streak weighting, and the high gamma=10 were all disabled by default but kept as optional CLI flags. The v5 run (`v5-hardCE-g7-splitfc-cleanverifier`) was deployed and is running with early metrics showing higher loss values (expected for hard CE) but comparable accuracy to previous runs at the same step count. The v3 and v4 training artifacts were archived for potential future reference.

The Three-Bug Diagnosis: How Systematic Code Comparison Uncovered Critical Flaws in DFlash Drafter Training 2707 words

Message Articles