Chunk 17.0

This chunk covers the pivot from GLM-5 to deploying **nvidia/Kimi-K2.5-NVFP4**, a 1T-parameter MoE model (DeepSeek V3 architecture) quantized by NVIDIA. The old GLM-5 GGUF weights (402GB) were removed, the 540GB model was downloaded across 119 safetensor shards, and vLLM was configured to serve it. A critical blocker emerged: the NVFP4 checkpoint ships with FP8 KV cache configuration, but **no MLA attention backend on SM120 (RTX PRO 6000) supports FP8 KV cache** — TRITON_MLA, the only viable backend, hardcodes `NotImplementedError` for FP8. This was resolved by removing `kv_cache_quant_algo` from `hf_quant_config.json` and `kv_cache_scheme` from `config.json`, falling back to fp16 KV cache. A systemd service (`vllm-kimi-k25.service`) was created with NCCL_PROTO=LL, tool calling, reasoning parser, and 128k context, achieving ~60 tok/s single-request throughput. The second half investigates whether previous GLM-5 optimizations apply and whether coherence issues stem from leftover patches. **All GLM-5-specific patches were verified absent** — the vLLM installation is clean stock code. The NCCL_PROTO=LL and P2P_LEVEL=SYS settings were already carried forward. FP8 KV cache was confirmed **architecturally impossible on SM120** without writing FP8 dequantization into the Triton MLA kernel — a major engineering effort. Coherence testing across four prompts (factual, multi-step, multi-turn, creative) showed **fully coherent, correct output** with proper reasoning traces, suggesting the earlier "coherence issues" were either transient or caused by the user's specific prompts rather than systemic model corruption. The fundamental PCIe allreduce bottleneck remains the primary throughput limiter, with ~65-70% of decode time spent in NCCL communication.

From GLM-5 to Kimi-K2.5-NVFP4: A 1T-Parameter Model Deployment Saga on Blackwell GPUs 2384 words

Message Articles