The Kimi-K2.5-NVFP4 Deployment: Pivoting from GLM-5 to a 1T-Parameter Model on Blackwell GPUs

Introduction

In the high-stakes world of deploying frontier large language models on novel hardware, the gap between a HuggingFace model card and a working inference server can swallow weeks of engineering effort. This article chronicles a remarkable pivot: the abandonment of a deeply troubled 744B-parameter GLM-5 GGUF deployment and the successful deployment of nvidia/Kimi-K2.5-NVFP4, a 1-trillion-parameter Mixture-of-Experts model based on the DeepSeek V3 architecture, running across eight NVIDIA RTX PRO 6000 Blackwell GPUs. The story spans a 540GB download across 119 safetensor shards, the discovery that FP8 KV cache is architecturally unsupported on SM120 GPUs, a surgical two-line JSON fix that unblocked the entire deployment, shell escaping nightmares, memory budget arithmetic, and ultimately a validated inference server achieving ~60 tokens per second with coherent, reasoned output.

This segment of the opencode session represents a complete arc: from strategic pivot through reconnaissance, download, debugging, configuration, launch, benchmarking, production service creation, and finally a rigorous post-launch investigation that confirmed the deployment's correctness and identified the fundamental hardware bottleneck. Understanding how each piece fits together reveals the deep systems engineering required to deploy large models outside the datacenter.

The Strategic Pivot: Why GLM-5 Was Abandoned

The session's turning point arrived in [msg 2088], where the user delivered a concise but consequential verdict: the GLM-5 GGUF experiment was being abandoned. After dozens of rounds spent patching vLLM's GGUF loader, debugging Triton MLA attention backends for Blackwell SM120 GPUs, resolving tensor parallelism sharding mismatches, and ultimately achieving ~57 tok/s throughput on a 744B-parameter model, the user's assessment was damning: the model was "pretty unusable in that quant" ([msg 2088]). The VM was snapshotted to preserve the state for potential future reference, and the team was moving to an entirely different model and quantization format.

The new target was nvidia/Kimi-K2.5-NVFP4, a model that the user believed "should be much simpler" because "latest vllm should run it natively" ([msg 2088]). The user also noted it "should be a bit faster because it's less active params and less layers" — a reference to the MoE architecture where only a subset of experts are active per token, reducing the computational load compared to the dense GLM-5 model. This pivot was not merely a change of model — it was a strategic bet on a different quantization paradigm (NVFP4 instead of GGUF), a different architecture (DeepSeek V3 MoE instead of GLM dense), and a different deployment approach (native vLLM support instead of custom patches).

The assistant's response in [msg 2089] was a model of methodical planning. Rather than immediately executing commands, the assistant paused to research and plan, producing a structured todo list with five high-priority tasks: research the model requirements, install the latest vLLM, download the model, launch and verify inference, and update the systemd service. This planning phase established a clear sequence of dependencies and identified the key unknowns before any action was taken — a discipline that would prove critical in the hours ahead.

Reconnaissance and Cleanup

Message [msg 2090] executed the reconnaissance phase, issuing two parallel tool calls: a webfetch to the HuggingFace model card and a bash command to inspect the current server state. The results revealed the full scope of the cleanup required. All 8 GPUs were still occupied by the old GLM-5 model, each consuming 93,783 MiB out of 97,887 MiB total. The old vllm-glm5 systemd service was still active. The vLLM version check failed because the environment used uv instead of pip for package management — a non-standard Python setup that would complicate future operations.

The cleanup was methodical. The old 402GB GLM-5 GGUF weights were removed, freeing disk space for the 540GB Kimi-K2.5 model. The old systemd service was stopped and disabled. The assistant then upgraded vLLM to the latest nightly build (v0.16.0rc2.dev313) using uv pip install --pre --upgrade. This was a critical step — the model card recommended vLLM v0.15.0, but the user specifically warned that the suggested Docker image "might have pro6000 gpu issues" ([msg 2088]), so bleeding-edge was the safer bet.

A critical correction came in [msg 2096], where the user pointed out that the assistant's initial plan to use tensor-parallel size 4 was insufficient: "It's a 1T model, need TP8" ([msg 2096]). This one-line correction saved hours of debugging — a 1T-parameter model cannot fit on 4 GPUs, and the assistant had underestimated the model's scale. The assistant updated its plan accordingly, setting --tensor-parallel-size 8 for all subsequent launch attempts.

The 540GB Download: 119 Shards and a Near-Race Condition

Downloading the Kimi-K2.5-NVFP4 model was an operation in itself. The model consists of 119 safetensor shards totaling 540GB, hosted on HuggingFace. The download was initiated using huggingface-cli download nvidia/Kimi-K2.5-NVFP4 --local-dir /shared/kimi-k2.5-nvfp4. The download process was monitored across multiple messages (<msgs id=2103-2108>), with the assistant periodically checking progress via du -sh and ls commands.

At one point, a race condition nearly derailed the download. The assistant had started two parallel download processes — one from a subagent and one from the main session — which could have corrupted the files by writing to the same directory simultaneously. The assistant caught this and killed the redundant process, a moment of operational awareness that prevented what could have been hours of re-downloading.

After the download completed, the assistant performed a thorough verification in [msg 2109], checking the file count (119 safetensors), total size (540GB), and inspecting the model's configuration files to understand its architecture. The disk usage showed 540GB consumed with 1.2TB still free — enough headroom for the model and future experiments.

Reading the Blueprint: Architecture Inspection

Before launching the model, the assistant carefully inspected its configuration in [msg 2104]. The config.json revealed that Kimi-K2.5 uses the DeepSeek V3 architecture with Multi-Head Latent Attention (MLA), 61 layers, 32 expert heads (with 8 active per token), a hidden dimension of 7168, and a vocabulary of 256,000 tokens. The model type was kimi_k25 with architecture KimiK25ForConditionalGeneration.

The quantization configuration was particularly revealing. The hf_quant_config.json showed &#34;quant_algo&#34;: &#34;NVFP4&#34; with &#34;kv_cache_quant_algo&#34;: &#34;FP8&#34; and a group size of 16. The config.json contained a kv_cache_scheme with num_bits: 8 and type: float. These settings specified that the KV cache should use FP8 quantization — a performance optimization designed for data-center Blackwell GPUs (B200/B100) that would become the central blocker of the deployment.

The assistant also discovered that the model uses the kimi_k2 tool parser and reasoning parser, which vLLM's nightly build supported natively. This was a significant advantage over the GLM-5 GGUF deployment, which had required extensive custom patches to vLLM's GGUF loader, attention backends, and weight utilities.

The FP8 KV Cache Wall

The first launch attempt in [msg 2110] failed almost immediately. The assistant launched vLLM with --tensor-parallel-size 8, --tool-call-parser kimi_k2, --reasoning-parser kimi_k2, and the NCCL tuning parameters carried forward from the GLM-5 deployment (NCCL_PROTO=LL, NCCL_P2P_LEVEL=SYS). The server crashed during initialization, producing a log file that was dense with Triton warnings, NCCL messages, and custom_all_reduce errors.

The debugging that followed (<msgs id=2111-2115>) is a textbook example of systematic log analysis. The assistant progressively refined its grep patterns to filter out noise, isolating the critical error trace. The breakthrough came in [msg 2115], where a carefully crafted grep command revealed the error originated in vllm/v1/attention/selector.py, line 98, in the _cached_get_attn_backend function. The error message was unambiguous: no valid MLA attention backend could be found.

Message [msg 2116] articulated the diagnosis with crystalline clarity: No valid MLA attention backend supports FP8 KV cache on SM120 (Blackwell RTX PRO 6000). The MLA backends available in vLLM are:

The Blackwell Divide: SM120 vs. B200/B100

Message [msg 2127] contained the critical architectural insight that explained the incompatibility. The assistant realized that the model card's recommendation of the vllm/vllm-openai:v0.15.0 Docker image was written for data-center Blackwell GPUs (B200/B100), which have proper FlashMLA support for FP8 KV cache on Hopper/Blackwell data-center architectures. The RTX PRO 6000, however, uses SM120 — the desktop/workstation variant of Blackwell — which is a different compute capability entirely.

The distinction between "Blackwell" as a marketing term and "SM120" as a compute architecture was the key insight. NVIDIA's Blackwell architecture spans multiple product lines: the B200 and B100 data-center GPUs (which use SM90 or a variant with full FlashMLA support), and the RTX PRO 6000 workstation GPU (which uses SM120 with different software compatibility). The model card's configuration was validated on the former, not the latter.

The assistant then proposed a surgical fix grounded in prior experience: remove the FP8 KV cache configuration from the model files, forcing vLLM to fall back to fp16 KV cache. This approach was validated by the earlier GLM-5 deployment, where TRITON_MLA had been proven to work correctly with fp16 KV cache on SM120 ([msg 2127]). The fix required no code changes, no kernel development, no library upgrades — just a configuration edit.

The Surgical Config Patch

Message [msg 2128] executed the fix with a Python script that ran over SSH, editing two JSON files:

  1. hf_quant_config.json: Removed kv_cache_quant_algo from the quantization object
  2. config.json: Removed kv_cache_scheme from the quantization_config object The script was defensive — it checked for each key's existence before deleting it — and handled the duplication of configuration across two files with different key names. The output confirmed both removals: "Removed kv_cache_quant_algo from hf_quant_config.json" and "Removed kv_cache_scheme from config.json" ([msg 2128]). This fix was the minimal intervention that unblocked the entire deployment. It worked because the assistant understood that the FP8 KV cache setting was a performance optimization, not a correctness requirement. The model weights remain in NVFP4 quantization; only the KV cache format was changed. The TRITON_MLA backend, which had been proven to work with fp16 KV cache on SM120 during the GLM-5 deployment, could now proceed without raising NotImplementedError. This moment illustrates a profound truth about deploying large models on novel hardware: the difference between success and failure can be as small as two del operations on JSON dictionaries. The hours of debugging that preceded this fix were not wasted — they were necessary to identify exactly which two lines to delete.

Shell Escaping and Launch Troubles

Even after the config patch, launching the model proved surprisingly difficult. The assistant's initial attempt to launch via a one-liner nohup command over SSH failed — the log file was never created and no process appeared. The assistant diagnosed this as a zsh escaping issue: the remote shell was zsh, which handles certain escape sequences differently than bash, causing the command to be silently corrupted.

The fix was to write a self-contained shell script to /tmp/run_kimi.sh and then launch that script with nohup ([msg 2142]). This eliminated the escaping problem entirely. The script used exec to replace the shell process with the Python process, ensuring clean process management:

#!/bin/bash
export NCCL_PROTO=LL
export NCCL_P2P_LEVEL=SYS
exec /root/ml-env/bin/python3 -m vllm.entrypoints.openai.api_server \
    --model /shared/kimi-k2.5-nvfp4 \
    --tensor-parallel-size 8 \
    --tool-call-parser kimi_k2 \
    --reasoning-parser kimi_k2 \
    --trust-remote-code \
    --max-model-len 131072 \
    --gpu-memory-utilization 0.95 \
    --port 8000 \
    --disable-log-requests

The response was simply: PID: 212172.

The KV Cache Memory Budget

When the model finally began loading, a new blocker emerged. The model loaded successfully — 70.81 GiB per GPU — but then crashed during KV cache initialization. The error trace pointed to _check_enough_kv_cache_memory in kv_cache_utils.py. The model's default max_model_len was 262,144 tokens, but with 75GB of weights per GPU (out of 96GB total), only ~11.7GB remained for KV cache.

Message [msg 2141] captured the assistant's real-time memory budget calculation:

"KV cache too small for the full 262144 context. The model's default is 262k, but with 75GB weights per GPU and 96GB total, there's only ~11.7GB for KV cache. It says 178k context fits. Let me limit to 131072 (128k) to be safe and leave room for batching."

The decision to cap at 128k rather than the profiled maximum of 178k was a deliberate stability margin. The assistant reserved roughly 25% headroom for batching, memory fragmentation, encoder cache (the model has a vision encoder requiring 8,192 tokens of cache), and temporary allocations during CUDAGraph compilation. This conservative approach prioritized reliability over maximum context length — a wise trade-off for a production deployment.

Verification and Benchmarking

The first inference request in [msg 2145] confirmed the model was working correctly. The prompt "What is the capital of France? Answer in one sentence." returned a coherent reasoning trace and the correct answer: "The capital of France is Paris." The model generated 107 tokens in 6.7 seconds, though the assistant correctly noted this included CUDAGraph warmup overhead and reasoning tokens.

Message [msg 2146] then benchmarked steady-state performance. A second request with a simple counting task ("Count from 1 to 50") at temperature=0.0 produced 512 tokens in 8.5 seconds, yielding ~60 tok/s steady-state throughput. The assistant's summary in [msg 2147] was appropriately measured: "512 tokens in 8.5s = ~60 tok/s. That's quite good for a 1T MoE model with NVFP4 quantization."

This throughput represented a marginal improvement over the GLM-5 deployment's ~57 tok/s, despite Kimi-K2.5 being a larger model (1T vs 744B parameters). The improvement was attributable to the MoE architecture's lower active parameter count per token and the reduced number of layers (61 vs 78), which translated to fewer allreduce calls per decode step.

Systemd Service Creation

The deployment was formalized as a systemd service (vllm-kimi-k25.service) in [msg 2164]. The service included the NCCL tuning parameters (NCCL_PROTO=LL, NCCL_P2P_LEVEL=SYS), the tool-call parser, reasoning parser, and 128k context configuration. The service started successfully, with both ExecStartPre steps passing and the main process running. The journal confirmed route registration at 22:24:20 — approximately 9 minutes after launch, consistent with the model loading and CUDAGraph compilation time.

The assistant's verification in [msg 2165] was deceptively simple: a single curl request asking "What is the capital of France? Answer briefly." returned "Paris" in 1.366 seconds with 43 tokens. This single word carried enormous weight — it confirmed that the 540GB of NVFP4 weights were correctly distributed across 8 GPUs with tensor parallelism, that the FP8 KV cache workaround produced coherent output, that the systemd service was functional end-to-end, and that the reasoning parser correctly separated chain-of-thought traces from the final answer.

The Post-Launch Investigation: Do Optimizations Transfer?

With the deployment confirmed operational, the user asked a question in [msg 2168] that drove the next phase of investigation: "Do previous optimizations, esp around allreduce apply? Would setting k/v cache to fp8 seemingly as the model recommends(?) also improve perf?"

This question demonstrated sophisticated understanding that the PCIe allreduce bottleneck was a hardware property, not a model-specific problem. The eight RTX PRO 6000 GPUs are connected via PCIe Gen5 lanes through the CPU, not via NVLink or NVSwitch. Any tensor-parallelism scheme that requires allreducing activations across all eight GPUs will be bottlenecked by PCIe bandwidth, regardless of which model is being served.

The assistant's investigation in [msg 2169] confirmed that the NCCL optimizations (NCCL_PROTO=LL, NCCL_P2P_LEVEL=SYS) were already applied to the Kimi service, carried forward from the GLM-5 deployment under a comment reading "# NCCL tuning for PCIe-only 8-GPU topology." The optimizations were already in effect.

The FP8 KV cache question received a definitive, architecture-grounded answer in [msg 2170]: no MLA attention backend in this vLLM version supports FP8 KV cache on SM120. The TRITON_MLA backend, the only viable option on Blackwell, has three explicit NotImplementedError guards for FP8. The Triton decode kernel contains no FP8 dequantization logic. Adding support would require writing FP8 dequantization into the Triton kernel — a major engineering effort involving kernel-level programming in Triton, a deep understanding of FP8 quantization formats (E4M3 and E5M2), and careful handling of the MLA-specific KV cache layout.

The assistant estimated that ~65-70% of decode time is spent in NCCL communication, a staggering proportion that highlights the severity of the PCIe bottleneck. The conclusion was honest and grounded: "The current ~60 tok/s is likely near the hardware ceiling for single-request throughput on this PCIe topology" ([msg 2170]).

The Coherence Crisis

Just as the optimization analysis concluded, the user raised an alarming concern in [msg 2171]: "Seems the model has heavy coherence issues in just 2-3 prompts, can this be k/v cache or are we running some dubious patches too?"

This message arrived at a moment of apparent success. The model was serving, the throughput was respectable, and the optimization analysis had been delivered. But the user's experience told a different story: after just two or three prompts, the output degraded into incoherence. The user proposed two specific, testable hypotheses targeting different system layers: KV cache corruption (perhaps from the FP8-to-fp16 workaround) and leftover patches from the GLM-5 deployment.

The assistant's response in [msg 2172] demonstrated the value of parallel evidence gathering. Two subagent tasks ran simultaneously: one audited every file that was patched during the GLM-5 work (gguf.py, gguf_loader.py, weight_utils.py, deepseek_v2.py, config.py, mla_attention.py), and the other ran four coherence tests against the live model covering factual knowledge, multi-step reasoning, multi-turn conversation, and creative writing.

The results were nuanced and instructive. The patch audit revealed that GLM-5 modifications were indeed still present in the vLLM installation. Most critically, deepseek_v2.py contained two torch.save debug blocks that triggered when exactly 5 tokens were being processed — since Kimi-K2.5 uses DeepseekV2Attention and DeepseekV2Model, these blocks would execute during live inference, causing CPU tensor copies and disk writes to /tmp/ mid-generation. This was a high-severity finding that could cause latency spikes, race conditions, or state corruption under specific conditions.

Yet the coherence tests told a different story. Across all four prompt types, the model produced fully coherent, correct output with proper reasoning traces. The perceived "heavy coherence issues" were either transient, triggered by a specific prompt condition that the tests didn't hit, or caused by something outside the model itself — perhaps a client-side issue or insufficient max_tokens budget where the reasoning trace consumed the entire allocation.

This was the most valuable kind of investigation result: it revealed that the system had both a real problem (dangerous debug code that could cause corruption under specific conditions) and was currently functioning correctly. The tension between these findings drove the next steps: the debug code must be reverted, but there was no evidence of systemic coherence failure.

The PCIe Allreduce Bottleneck: A Hardware Ceiling

Throughout the benchmarking and optimization analysis, one finding remained constant: the PCIe allreduce bottleneck is the fundamental throughput limiter. With ~65-70% of decode time spent in NCCL communication, no amount of model-level optimization can dramatically improve single-request throughput on this hardware topology. The ~60 tok/s figure is near the ceiling for PCIe-connected GPUs.

The architectural comparison between GLM-5 and Kimi-K2.5 confirmed this analysis. Kimi has 61 layers (vs 78 for GLM-5), 384 experts with top-8 routing (vs 256), the same hidden size of 7168, and the same kv_lora_rank of 512. Critically, Kimi has only 1 dense layer compared to GLM-5's 3, and 0 MTP layers versus GLM-5's 1. Fewer layers mean fewer allreduce calls per decode step — approximately 22% less allreduce overhead — which directly explains the throughput improvement from ~57 tok/s to ~60 tok/s.

On a system with NVLink or NVSwitch interconnects, throughput would likely be significantly higher. The RTX PRO 6000 GPUs lack NVLink bridges, so all GPU-to-GPU communication must traverse the PCIe bus. This is a hardware limitation that no software optimization can fully overcome — a fundamental constraint of the workstation GPU topology.

Key Lessons and Enduring Findings

Several findings from this segment have enduring value beyond the specific deployment:

FP8 KV cache is architecturally blocked on SM120. This is not a configuration issue or a bug — it is a fundamental limitation of the current software stack. The TRITON_MLA backend, which is the only viable MLA attention backend on Blackwell GPUs, has no FP8 dequantization logic. Adding it would require kernel-level engineering. This finding saves future investigation time: anyone asking "can we enable FP8 KV cache on Blackwell?" can be directed to this analysis.

The PCIe allreduce bottleneck is the primary throughput limiter. With ~65-70% of decode time spent in NCCL communication, no amount of model-level optimization can dramatically improve single-request throughput on this hardware topology. The ~60 tok/s figure is near the ceiling for PCIe-connected GPUs.

Hardware taxonomy matters. "Blackwell" is not a monolith — SM120 (workstation) differs fundamentally from B200/B100 (datacenter) in software compatibility. Model card configurations validated on data-center hardware may not work on workstation variants, and understanding the specific compute capability is essential for debugging.

Configuration is code. The FP8 KV cache blocker was resolved by editing two JSON files, not by patching vLLM source code. The ability to identify which configuration keys control which behaviors, and to understand the implications of removing them, is a critical engineering skill.

Prior experience transfers. The GLM-5 deployment proved that TRITON_MLA + fp16 works on SM120, directly enabling the Kimi-K2.5 fix. Every failed experiment generates knowledge that accelerates future work.

Leftover patches from previous deployments are a real risk. The GLM-5 debug instrumentation in deepseek_v2.py could have caused intermittent corruption during Kimi-K2.5 inference. The fact that it didn't manifest in the coherence tests does not mean it is safe — it means the triggering condition was not hit during testing. Clean installations are essential when pivoting between models.

Conclusion

The deployment of nvidia/Kimi-K2.5-NVFP4 across eight RTX PRO 6000 Blackwell GPUs represents a remarkable engineering achievement. The journey encompassed a strategic pivot from a failed GLM-5 GGUF deployment, a 540GB model download across 119 shards, the diagnosis and resolution of an FP8 KV cache incompatibility unique to the SM120 architecture, surgical configuration patching, shell escaping workarounds, memory budget calculations, and ultimately a validated inference server achieving ~60 tok/s throughput.

The post-launch investigation was equally valuable. By systematically testing whether previous optimizations transferred, whether FP8 KV cache could be re-enabled, and whether coherence issues stemmed from leftover patches, the team built a comprehensive understanding of the deployment's capabilities and limitations. The FP8 KV cache finding — that it is architecturally blocked on SM120 without kernel changes — is the kind of definitive answer that saves teams from chasing impossible optimizations. The coherence investigation — which found dangerous debug code but no evidence of systemic failure — demonstrated the value of parallel evidence gathering when diagnosing ambiguous symptoms.

In the end, the deployment is not merely functional — it is understood. The team knows what works, what doesn't, why the throughput is what it is, and what risks remain. That understanding, captured across this segment of the conversation, is the true deliverable of this phase of work. The model now serves as a production service, providing coherent, reasoned responses at interactive speeds — a testament to the power of systematic debugging and principled engineering in the face of novel hardware-software combinations.