Chunk 70.1

In this chunk, the assistant pivoted from analyzing PD-vs-single-server discrepancies to implementing the decisive fix: switching the DSA indexer's key storage from fp8 to bf16. After confirming that the DeepSeek reference implementation uses bf16 index keys while sglang's fused compressor kernel forces fp8 (head_dim=128), the assistant designed an environment-gated bf16 index-K path. Initial attempts routed through a non-fused store path, which validated the hypothesis—bf16 index-K recovered needles at 4509 and 10,498 tokens that reliably failed with fp8—but caused OOM at 22K due to transient memory in the non-fused path. The assistant then extended the fused CUDA kernel (`fused_norm_rope_v2.cuh`) to support bf16 storage for the indexer by adding a `kBf16Store` template parameter, relaxing the static assertion that restricted bf16 to head_dim=512, and implementing a paged bf16 store path (256 bytes/token, no fp8 quantization). The compressor_v2.py was updated to set `bf16_store=True` for the indexer when the environment flag is active, routing through the now-bf16-capable fused kernel. This preserves the memory-efficient, fast fused path while matching the reference's bf16 index-key precision. The overarching themes are a rigorous, layered diagnosis that traced the recall failure to a deliberate sglang design choice (fp8 index keys), validated the fix through empirical needle tests, and delivered a production-quality solution by modifying the fused CUDA kernel rather than relying on a slower fallback. The work balances correctness (bf16 keys restore recall) with performance (fused kernel avoids OOM and maintains speed), and aligns the deployment with the reference implementation's precision choices.

The Precision That Mattered: How bf16 Index Keys Fixed a Sparse Attention Recall Failure 2204 words

Message Articles