Segment 67

The assistant deployed DeepSeek-V4-Flash (FP4/nvfp4) on SGLang main across 8× RTX PRO 6000 Blackwell GPUs, achieving prefill-decode disaggregation with NIXL KV transfer, but performance was far below target (~10–25 tok/s vs 1000 tok/s). A systematic optimization campaign followed: pulling upstream SGLang, generating SM120 FP8 GEMM autotune configs, enabling MTP/EAGLE speculative decoding (47% single-request boost, zero at concurrency due to verifier saturation), and switching to the NVIDIA NVFP4 quantization to route MoE execution through tensor cores. NVFP4 delivered ~28 tok/s at C=16 (24% improvement), but profiling revealed the decisive bottleneck: the sm_120 fallback kernels for sparse-MLA decode and MoE slot-GEMV run on CUDA cores rather than tensor cores, and the MTP verifier halved effective batch size. The assistant concluded that configuration tuning and checkpoint changes can only yield incremental gains against this structural ceiling, and the path forward requires either disabling MTP to restore batch capacity or building a custom split-K tensor-core sparse-attention kernel.

Deploy DeepSeek-V4-Flash FP4 with PD disaggregationOptimize with MTP speculative decoding and FP8 tuningSwitch to NVFP4 quantization for tensor-core MoEDiagnose sm_120 fallback kernel bottleneckIdentify MTP verifier memory overhead

The 24% Ceiling: When Configuration Tuning Meets the sm_120 Kernel Wall 3602 words

Chunks