Segment 17

In this sub-session, we pivoted from the GLM-5 GGUF deployment to deploying nvidia/Kimi-K2.5-NVFP4, a 1T-parameter MoE model. We removed the old 402GB GLM-5 weights, downloaded the 540GB model across 119 safetensor shards, and configured vLLM to serve it. A critical blocker emerged: the NVFP4 checkpoint ships with FP8 KV cache configuration, but no MLA attention backend on SM120 (RTX PRO 6000) supports FP8 KV cache. This was resolved by removing kv_cache_quant_algo and kv_cache_scheme from the config files, falling back to fp16 KV cache. We created a systemd service (vllm-kimi-k25.service) with NCCL_PROTO=LL, tool calling, reasoning parser, and 128k context, achieving ~60 tok/s single-request throughput. We then investigated whether previous GLM-5 optimizations apply and whether coherence issues stem from leftover patches, confirming that all GLM-5-specific patches are absent (clean stock vLLM) and that FP8 KV cache is architecturally impossible on SM120 without writing FP8 dequantization into the Triton MLA kernel. Coherence testing across multiple prompts showed fully coherent, correct output with proper reasoning traces, and we confirmed that the PCIe allreduce bottleneck remains the primary throughput limiter.

Pivot from GLM-5 to Kimi-K2.5-NVFP4 modelDownload 540GB model across 119 safetensor shardsResolve FP8 KV cache blocker on SM120 by removing kv_cache_quant_algoCreate systemd service vllm-kimi-k25.service with tuned parametersVerify coherence of model output across multiple promptsConfirm absence of leftover GLM-5 patches in vLLM installationBenchmark single-request throughput at ~60 tok/sIdentify PCIe allreduce bottleneck as primary throughput limiter

The Kimi-K2.5-NVFP4 Deployment: Pivoting from GLM-5 to a 1T-Parameter Model on Blackwell GPUs 3928 words

Chunks