Chunk 65.1

In this chunk, the assistant enabled tool-call and reasoning parsers (`kimi_k2`) on the live SGLang DDTree service for Kimi K2.6, then diagnosed a severe throughput regression the user reported (~32 t/s at ~5k context vs the 138 t/s baseline). The assistant built a context-sweep diagnostic (`bench_context_decode.py`) and ran controlled measurements to isolate the cause. The parsers were ruled out as the cause (they're output post-processing). Two-point decode measurements showed expected context-scaling behavior (255→216→113→67 t/s from ctx 64→5k), but long continuous 1500-token generations collapsed to ~32 t/s even at moderate context, revealing two compounding issues: the undertrained drafter's acceptance rate drops from ~5 to ~2.9 on long free-form/reasoning output, and KV cache fragmentation under `page_size=1` with non-contiguous DDTree commits degrades step time over sustained generation. Short-prompt tests confirmed that output-length alone doesn't degrade throughput (~140-155 t/s for 200-1200 tokens), isolating the problem to context-length scaling and drafter generalization on reasoning text. The assistant identified the primary levers: improving the drafter (longer-term), expanding the draft-window-size beyond 2048 to recover acceptance at longer contexts, and addressing KV cache fragmentation. The 32 t/s was confirmed as the real C=1 long-context performance of the current stack on PRO 6000, not a regression from the parser change. The diagnostic tools and analysis were committed to the repo, and the assistant offered to test draft-window tuning if the user wanted to pay the ~10-minute restart cost.

The Throughput Lever: Quantifying the MoE GEMM and Mapping the Road to B300 2214 words

Message Articles