The Config Patch That Saved the Deployment: Resolving FP8 KV Cache Incompatibility on Blackwell SM120

In the sprawling, multi-day journey to deploy a 540-billion-parameter Kimi-K2.5-NVFP4 model across eight RTX PRO 6000 Blackwell GPUs, one message stands out as the quiet pivot point between failure and success. Message <msg id=2129> is deceptively simple — a single bash command that launches vLLM with a familiar set of flags. But behind this seemingly routine restart lies the resolution of a critical architectural incompatibility that threatened to derail the entire deployment. This message is the moment where a deep understanding of GPU hardware, attention backend selection, and model quantization formats converged into a clean, surgical fix.

The FP8 KV Cache Blocker

To understand why this message matters, we must first understand the problem it was designed to test. The nvidia/Kimi-K2.5-NVFP4 model is a 1-trillion-parameter Mixture-of-Experts model based on the DeepSeek V3 architecture, quantized by NVIDIA using their NVFP4 format. The model ships with a quantization configuration that specifies kv_cache_quant_algo: "FP8" in hf_quant_config.json and a corresponding kv_cache_scheme in config.json. This tells vLLM to use FP8 quantization for the key-value cache, a memory-saving technique that halves the KV cache footprint compared to fp16.

The problem, as discovered in the preceding messages, is that no MLA (Multi-head Latent Attention) backend on SM120 supports FP8 KV cache. The RTX PRO 6000 Blackwell GPUs have compute capability 12.0 (SM120), which is a different variant from the data-center Blackwell GPUs (B200/B100) that support FlashMLA with FP8. On SM120, the only viable attention backend is TRITON_MLA, a Triton-based implementation. But TRITON_MLA hardcodes NotImplementedError for FP8 KV cache — it simply cannot handle it. The other backends (FLASH_ATTN_MLA, FLASHMLA, FLASHINFER_MLA) either don't support SM120 at all or lack FP8 support on this architecture.

This created a deadlock: the model demanded FP8 KV cache, but the hardware could not provide it through any available backend.

The Surgical Fix: Patching Configuration, Not Code

The assistant faced a fork in the road. One path was to write FP8 dequantization support into the Triton MLA kernel — a major engineering effort requiring deep understanding of Triton, CUDA, and the Blackwell architecture. The other path was to override the KV cache dtype at the configuration level, falling back to fp16. The assistant chose the latter, and the reasoning was sound: the TRITON_MLA backend had already been proven to work on SM120 with fp16 KV cache during the earlier GLM-5 deployment (see <msg id=2116> through <msg id=2128>).

The fix was executed in message <msg id=2128>, immediately preceding our subject message. The assistant used a Python script to:

  1. Remove kv_cache_quant_algo from hf_quant_config.json
  2. Remove kv_cache_scheme from config.json This was a minimal, reversible change to the model's configuration files — no vLLM source code was modified, no custom kernels were written. The assumption was that without these configuration entries, vLLM would fall back to its default KV cache dtype (fp16/bf16), which the TRITON_MLA backend handles gracefully on SM120.

Anatomy of the Launch Command

Message <msg id=2129> issues the test of this fix. The bash command is carefully constructed:

pkill -9 -f "python3.*vllm" 2>/dev/null; sleep 3; rm -f /dev/shm/psm_* /dev/shm/sem.mp-*; NCCL_PROTO=LL NCCL_P2P_LEVEL=SYS nohup /root/ml-env/bin/python3 -m vllm.entrypoints.openai.api_server \
    --model /shared/kimi-k2.5-nvfp4 \
    --tensor-parallel-size 8 \
    --tool-call-parser kimi_k2 \
    --reasoning-parser kimi_k2 \
    --trust-remote-code \
    --port 8000 \
    --disable-log-requests \
    > /tmp/vllm_kimi3.log 2>&1 &

Every element of this command reflects lessons learned from previous failures:

Assumptions and Risks

The message rests on several critical assumptions:

  1. vLLM respects the patched config files — The assistant assumes that after removing the KV cache entries, vLLM will not attempt to use FP8 KV cache. This is reasonable but not guaranteed; vLLM might cache the original config or have hardcoded defaults for this model architecture.
  2. No other configuration issues will surface — The FP8 KV cache was the known blocker, but there could be other latent issues: missing tensor shards, incorrect model architecture detection, or dtype mismatches.
  3. The NCCL tuning flags are still optimal — The NCCL_PROTO=LL and NCCL_P2P_LEVEL=SYS settings were tuned for the GLM-5 model. The Kimi-K2.5 model has a different architecture (DeepSeek V3 with MLA), which may have different communication patterns.
  4. The Triton MLA backend will be selected automatically — With FP8 KV cache removed, the assistant assumes vLLM's attention backend selector will choose TRITON_MLA for SM120, which was proven to work during the GLM-5 deployment.

The Broader Context

This message is the culmination of a long debugging chain that began in segment 14, when the assistant first attempted to deploy the Kimi-K2.5 model. The FP8 KV cache issue was first identified in message <msg id=2115>, where the error log revealed "No valid MLA attention backend supports FP8 KV cache on SM120." The assistant then spent several messages investigating the problem, trying --kv-cache-dtype auto (which failed), checking for newer vLLM versions (none available), and finally arriving at the config patching solution.

The segment summary confirms that this approach succeeded: "Resolved FP8 KV cache blocker on SM120 by removing kv_cache_quant_algo from hf_quant_config.json and kv_cache_scheme from config.json, falling back to fp16 KV cache." The model loaded successfully, achieved ~60 tok/s single-request throughput, and was deployed as a production systemd service.

What This Message Represents

Message <msg id=2129> is a testament to the power of understanding the full stack — from GPU architecture (SM120 vs. Hopper/Blackwell data-center variants) to attention backend selection algorithms to model quantization format specifications. The fix was not a brute-force approach (rewriting kernels) but a surgical one (removing two JSON keys). This required the assistant to: