From GLM-5 to Kimi-K2.5-NVFP4: A 1T-Parameter Model Deployment Saga on Blackwell GPUs

Introduction

In the high-stakes world of deploying cutting-edge large language models on novel hardware, the path from a HuggingFace model card to a working inference server is rarely a straight line. This article chronicles a remarkable engineering journey: the pivot from a deeply troubled 744B-parameter GLM-5 GGUF deployment to the successful deployment of nvidia/Kimi-K2.5-NVFP4, a 1-trillion-parameter Mixture-of-Experts model based on the DeepSeek V3 architecture, running across eight RTX PRO 6000 Blackwell GPUs. The story spans model downloads measured in hundreds of gigabytes, architectural incompatibilities between GPU compute capabilities and attention backends, surgical configuration patches, shell escaping nightmares, memory budget calculations, and ultimately a validated inference server achieving ~60 tokens per second.

This chunk of the opencode session represents a complete arc: from strategic pivot, through reconnaissance, download, debugging, configuration, launch, benchmarking, and finally production service creation. Each message in this sequence contributed a critical piece to the puzzle, and understanding how they fit together reveals the deep systems engineering required to deploy large models outside the datacenter.

The Strategic Pivot: Abandoning GLM-5 GGUF

The session's turning point came in [msg 2088], where the user delivered a concise but consequential message: the GLM-5 GGUF experiment was being abandoned. After dozens of rounds spent patching vLLM's GGUF loader, debugging Triton MLA attention backends for Blackwell SM120 GPUs, resolving tensor parallelism sharding mismatches, and ultimately achieving ~57 tok/s throughput on a 744B-parameter model, the user's verdict was damning: the model was "pretty unusable in that quant" [1]. The VM was snapshotted, preserving the state for potential future reference, and the team was moving to an entirely different model and quantization format.

The new target was nvidia/Kimi-K2.5-NVFP4, a model that the user believed "should be much simpler" because "latest vllm should run it natively" [1]. The user also noted it "should be a bit faster because it's less active params and less layers" — a reference to the MoE architecture where only a subset of experts are active per token, reducing the computational load compared to the dense GLM-5 model.

The assistant's response in [msg 2089] was a model of methodical planning. Rather than immediately executing commands, the assistant paused to research and plan, producing a structured todo list: research the model requirements, install the latest vLLM, download the model, launch and verify inference, and update the systemd service [2]. This planning phase was critical — it established a clear sequence of dependencies and identified the key unknowns before any action was taken.

Reconnaissance and Cleanup

Message [msg 2090] executed the reconnaissance phase, issuing two parallel tool calls: a webfetch to the HuggingFace model card and a bash command to inspect the current server state [3]. The results revealed that all 8 GPUs were still occupied by the old GLM-5 model (93,783 MiB used out of 97,887 MiB total per GPU), the old vllm-glm5 systemd service was still active, and the vLLM version check failed due to the non-standard Python environment using uv instead of pip.

The cleanup was methodical. The old 402GB GLM-5 GGUF weights were removed, freeing disk space for the 540GB Kimi-K2.5 model [4]. The old systemd service was stopped and disabled. The assistant then upgraded vLLM to the latest nightly build (v0.16.0rc2.dev313) using uv pip install --pre --upgrade [5].

A critical correction came in [msg 2096], where the user pointed out that the assistant's initial plan to use tensor-parallel size 4 was insufficient: "It's a 1T model, need TP8" [6]. This one-line correction saved hours of debugging — a 1T-parameter model cannot fit on 4 GPUs, and the assistant had underestimated the model's scale.

The 540GB Download

Downloading the Kimi-K2.5-NVFP4 model was an operation in itself. The model consists of 119 safetensor shards totaling 540GB, hosted on HuggingFace. The download was initiated using huggingface-cli download nvidia/Kimi-K2.5-NVFP4 --local-dir /shared/kimi-k2.5-nvfp4 [7].

The download process was monitored across multiple messages (<msgs id=2103-2108>), with the assistant periodically checking progress via du -sh and ls commands. At one point, a race condition nearly derailed the download: the assistant had started two parallel download processes (one from a subagent and one from the main session), which could have corrupted the files [8]. The assistant caught this and killed the redundant process.

After the download completed, the assistant performed a thorough verification in [msg 2109], checking the file count (119 safetensors), total size (540GB), and inspecting the model's configuration files to understand its architecture [9].

Reading the Blueprint: Architecture Inspection

Before launching the model, the assistant carefully inspected its configuration in [msg 2104]. The config.json revealed that Kimi-K2.5 uses the DeepSeek V3 architecture with Multi-Head Latent Attention (MLA), 61 layers, 32 expert heads (with 8 active per token), a hidden dimension of 7168, and a vocabulary of 256,000 tokens [10]. The quantization config showed &#34;kv_cache_quant_algo&#34;: &#34;FP8&#34; — a detail that would become critically important.

The assistant also discovered that the model uses the kimi_k2 tool parser and reasoning parser, which vLLM's nightly build supported natively. This was a significant advantage over the GLM-5 GGUF deployment, which had required extensive custom patches.

The FP8 KV Cache Wall

The first launch attempt in [msg 2110] failed almost immediately. The assistant launched vLLM with --tensor-parallel-size 8, --tool-call-parser kimi_k2, --reasoning-parser kimi_k2, and the NCCL tuning parameters carried forward from the GLM-5 deployment (NCCL_PROTO=LL, NCCL_P2P_LEVEL=SYS) [11]. The server crashed during initialization.

The debugging that followed (<msgs id=2111-2115>) is a textbook example of systematic log analysis. The assistant progressively refined its grep patterns to filter out noise from Triton warnings, NCCL messages, and custom_all_reduce errors, eventually isolating the critical error trace [12]. The breakthrough came in [msg 2115], where a carefully crafted grep command revealed the error originated in vllm/v1/attention/selector.py, line 98, in the _cached_get_attn_backend function [13].

Message [msg 2116] articulated the diagnosis: No valid MLA attention backend supports FP8 KV cache on SM120 (Blackwell RTX PRO 6000) [14]. The MLA backends available in vLLM are:

The Blackwell Divide

Message [msg 2127] contained the critical architectural insight that explained the incompatibility. The assistant realized that the model card's recommendation of vllm/vllm-openai:v0.15.0 docker image was written for data-center Blackwell GPUs (B200/B100), which have proper FlashMLA support for FP8 KV cache on Hopper/Blackwell data-center architectures [17]. The RTX PRO 6000, however, uses SM120 — the desktop/workstation variant of Blackwell — which is a different compute capability entirely. The distinction between "Blackwell" as a marketing term and "SM120" as a compute architecture was the key insight.

The assistant then proposed a surgical fix grounded in prior experience: remove the FP8 KV cache configuration from the model files, forcing vLLM to fall back to fp16 KV cache. This approach was validated by the earlier GLM-5 deployment, where TRITON_MLA had been proven to work correctly with fp16 KV cache on SM120 [17].

The Surgical Config Patch

Message [msg 2128] executed the fix. The assistant wrote a Python script that ran over SSH, editing two JSON files [18]:

  1. hf_quant_config.json: Removed kv_cache_quant_algo from the quantization object
  2. config.json: Removed kv_cache_scheme from the quantization_config object The script was defensive — it checked for the key's existence before deleting it — and handled the duplication of configuration across two files with different key names. The output confirmed both removals: "Removed kv_cache_quant_algo from hf_quant_config.json" and "Removed kv_cache_scheme from config.json" [18]. This fix required no code changes, no kernel development, no library upgrades — just two del operations on JSON dictionaries. It was the minimal intervention that unblocked the entire deployment, and it worked because the assistant understood that the FP8 KV cache setting was a performance optimization, not a correctness requirement. The model weights remain in NVFP4 quantization; only the KV cache format was changed.

Shell Escaping and Launch Troubles

Even after the config patch, launching the model proved surprisingly difficult. The assistant's initial attempt to launch via a one-liner nohup command over SSH failed — the log file was never created and no process appeared [19]. The assistant diagnosed this as a zsh escaping issue: the remote shell was zsh, which handles certain escape sequences differently than bash, causing the command to be silently corrupted [20].

The fix was to write a self-contained shell script to /tmp/run_kimi.sh and then launch that script with nohup [21]. This eliminated the escaping problem entirely. The script used exec to replace the shell process with the Python process, ensuring clean process management.

The KV Cache Memory Budget

When the model finally began loading, a new blocker emerged. The model loaded successfully — 70.81 GiB per GPU — but then crashed during KV cache initialization [22]. The error trace pointed to _check_enough_kv_cache_memory in kv_cache_utils.py [23]. The model's default max_model_len was 262,144 tokens, but with 75GB of weights per GPU (out of 96GB total), only ~11.7GB remained for KV cache.

Message [msg 2141] captured the assistant's real-time memory budget calculation [24]:

"KV cache too small for the full 262144 context. The model's default is 262k, but with 75GB weights per GPU and 96GB total, there's only ~11.7GB for KV cache. It says 178k context fits. Let me limit to 131072 (128k) to be safe and leave room for batching."

The decision to cap at 128k rather than the profiled maximum of 178k was a deliberate stability margin. The assistant reserved roughly 25% headroom for batching, memory fragmentation, encoder cache (the model has a vision encoder requiring 8,192 tokens of cache), and temporary allocations during CUDAGraph compilation.

The Final Launch

Message [msg 2142] was the culmination of all the debugging. The assistant created a launch script with all the learned parameters [25]:

export NCCL_PROTO=LL
export NCCL_P2P_LEVEL=SYS
exec /root/ml-env/bin/python3 -m vllm.entrypoints.openai.api_server \
    --model /shared/kimi-k2.5-nvfp4 \
    --tensor-parallel-size 8 \
    --tool-call-parser kimi_k2 \
    --reasoning-parser kimi_k2 \
    --trust-remote-code \
    --max-model-len 131072 \
    --gpu-memory-utilization 0.95 \
    --port 8000 \
    --disable-log-requests

The response was simply: PID: 212172. The model loaded in 523 seconds, consuming 70.8 GiB per GPU, and the application started successfully [25].

Verification and Benchmarking

The first inference request in [msg 2145] confirmed the model was working correctly. The prompt "What is the capital of France? Answer in one sentence." returned a coherent reasoning trace and the correct answer: "The capital of France is Paris." [26].

Message [msg 2146] then benchmarked steady-state performance. The first request produced 107 tokens in 6.7 seconds (~16 tok/s), but the assistant correctly noted this included CUDAGraph warmup overhead and reasoning tokens [27]. A second request with a simple counting task ("Count from 1 to 50") at temperature=0.0 produced 512 tokens in 8.5 seconds, yielding ~60 tok/s steady-state throughput [27].

Message [msg 2147] delivered the triumphant summary: "512 tokens in 8.5s = ~60 tok/s. That's quite good for a 1T MoE model with NVFP4 quantization." [28]. The assistant also updated its todo list, marking all high-priority tasks as completed.

Coherence Verification and Systemd Service

The assistant then conducted a thorough coherence verification across multiple prompts in [msg 2148] and subsequent messages. Four test prompts covered factual knowledge ("What is the capital of France?"), multi-step reasoning ("If a train leaves Station A at 2:15 PM traveling at 120 km/h..."), multi-turn conversation, and creative writing [29]. All responses were fully coherent, correct, and properly formatted with reasoning traces.

The assistant also investigated whether any GLM-5-specific patches remained in the vLLM installation, confirming that the installation was clean stock code with no leftover modifications [30]. The FP8 KV cache limitation was confirmed as an architectural constraint of SM120 — writing FP8 dequantization into the Triton MLA kernel would be "a major engineering effort" [30].

Finally, the deployment was formalized as a systemd service (vllm-kimi-k25.service) with proper NCCL tuning, tool calling, reasoning parser, and 128k context configuration [31]. The service was enabled for automatic startup, bringing the deployment to a production-ready state.

The PCIe Allreduce Bottleneck

Throughout the benchmarking, the assistant identified the fundamental throughput limiter: the PCIe allreduce bottleneck. With 65-70% of decode time spent in NCCL communication, the inter-GPU interconnect is the primary constraint on performance [30]. The RTX PRO 6000 GPUs lack NVLink bridges, so all GPU-to-GPU communication must traverse the PCIe bus. On a system with NVLink or NVSwitch, throughput would likely be significantly higher. This is a hardware limitation that no software optimization can fully overcome.

Conclusion

The deployment of nvidia/Kimi-K2.5-NVFP4 across eight RTX PRO 6000 Blackwell GPUs represents a remarkable engineering achievement. The journey encompassed a strategic pivot from a failed GLM-5 GGUF deployment, a 540GB model download across 119 shards, the diagnosis and resolution of an FP8 KV cache incompatibility unique to the SM120 architecture, surgical configuration patching, shell escaping workarounds, memory budget calculations, and ultimately a validated inference server achieving ~60 tok/s throughput.

The key lessons from this saga are: