Segment 66

In this sub-session, the assistant first completed the 200k context-length deployment on CT200, diagnosing severe decode slowdown at long context due to Triton MLA verify attention with page_size=1 causing scattered KV access at low bandwidth. After discovering that optimized MLA kernels lack sm_120 support, the assistant built a custom sm_120 verify kernel (verify_attn_flash.cu) and iteratively optimized it: making it capture-safe for CUDA graphs, increasing NSPLIT to 64, and adding 128-bit vectorized BF16 KV loads. These optimizations delivered a 3-6× end-to-end decode speedup over the Triton baseline across all context lengths (4k-65k). The assistant also implemented Tier 0 KV defragmentation by monkeypatching the allocator to keep per-request KV contiguous. The live service now runs the owned capture-safe kernel with CUDA graphs enabled and Tier 0 defrag active, achieving 3-6× Triton's decode throughput. The bottleneck has shifted to MoE expert imbalance at batch size 1, a structural ceiling beyond the verify kernel's scope.

Deploy 200k context-length on CT200Diagnose long-context decode bottleneckBuild custom sm_120 verify attention kernelMake verify kernel CUDA graph capture-safeOptimize verify kernel for occupancy and bandwidthImplement Tier 0 KV defragmentationDeploy optimized kernel into live SGLang serviceIdentify MoE imbalance as remaining bottleneck

The Custom sm_120 Verify Kernel: From 0.7 tok/s to 3–6× Decode Speedup on Blackwell Consumer GPUs 3997 words

Chunks