Chunk 70.0

In this chunk, the assistant systematically diagnosed a coherence bug where the model lost context on longer multi-turn prompts, particularly failing to retrieve a specific "needle" fact from a large context. Through code analysis, mathematical microtests on real checkpoint weights, and empirical endpoint testing, every speed patch (MHC bf16, routed scaling, indexer bf16, MMA decode) was exonerated as the root cause. The bug was isolated to the DSA sparse attention's top‑512 selection: the model reliably found the needle within ~2K tokens but lost it beyond ~4K, independent of position, while local sliding‑window attention and short contexts worked fine. This pointed to a fundamental coverage/ranking limitation in the sparse indexer, likely aggravated by NVFP4/fp8 quantization. The assistant then pursued a config‑only fix by raising `index_topk` from 512 to 1024, an officially supported value in sglang’s kernel. This doubled the reliable recall range (~2.5K→~5K tokens) without regressing multi‑turn or tool‑calling performance. The fix was applied to both prefill and decode servers in the PD‑disaggregated deployment, with a slight memory‑fraction reduction to accommodate the larger sparse buffers. The needle‑in‑haystack test at 5.5K tokens passed, confirming the improvement. The overarching theme is a rigorous, layered diagnosis: ruling out speed patches with targeted tests, identifying the true bottleneck in the stock DSA sparse attention, and applying the best available configuration lever. The work balances performance optimization with correctness, acknowledges the model’s architectural limits (aggressive sparse attention + quantization), and delivers a practical, deployable improvement while documenting the residual limitation for longer contexts.

The Anatomy of a Sparse Attention Failure: Diagnosing and Fixing Context-Loss in DeepSeek-V4 on Blackwell 2259 words

Message Articles