The Launch: Deploying Kimi-K2.5-NVFP4 on Blackwell GPUs

A Pivotal Bash Command at the Crossroads of Setup and Deployment

In the sprawling narrative of an opencode coding session spanning dozens of segments and thousands of messages, there are moments that serve as fulcrums—points where the accumulated weight of prior work pivots toward a new phase of execution. Message 2110 is one such moment. It is a single bash command, issued by the AI assistant, that attempts to launch the nvidia/Kimi-K2.5-NVFP4 model on a vLLM server running across eight RTX PRO 6000 Blackwell GPUs. On its surface, the message is unremarkable: a nohup-wrapped Python invocation with a handful of flags and environment variables. But beneath that surface lies a dense web of reasoning, assumptions, and context that makes this message a rich subject for analysis.

The Road to Launch

To understand why this message was written, one must trace the path that led to it. The session had begun with the deployment of GLM-5, first in NVFP4 format via SGLang, then in GGUF format via a heavily patched vLLM. The GGUF effort was heroic—spanning custom patches for gguf_loader.py, weight_utils.py, a Triton MLA sparse attention backend, and CUDAGraph tuning—but the end result was disappointing. As the user put it in [msg 2088], "the model was pretty unusable in that quant." The GLM-5 GGUF deployment, despite achieving ~57 tok/s throughput, produced incoherent output that the team ultimately traced to a tensor parallelism sharding mismatch in the kv_b_proj weights.

The pivot to Kimi-K2.5-NVFP4 was a strategic retreat to safer ground. NVIDIA had published this model as a reference implementation: a 1-trillion-parameter MoE model based on the DeepSeek V3 architecture, quantized to NVFP4 (NVIDIA's 4-bit floating-point format), with explicit vLLM support. The model card recommended a straightforward launch command. The promise was clear: this should "run natively" on the latest vLLM, with no patching required.

The assistant's preparatory work was thorough. It stopped and disabled the old GLM-5 systemd service, cleaned GPU memory, removed the 402GB GGUF weights from disk, upgraded vLLM to the nightly build (0.16.0rc2), and downloaded the 540GB model across 119 safetensor shards—a download that took over 15 minutes. By the time message 2110 was composed, the stage was set. The old weights were gone, the new weights were in place, and the software stack was ready. All that remained was the launch.

The Message Itself

The assistant's message opens with a brief summary of the state: "Good, 540GB model, 1.2T still free. Now let me launch vLLM." It then reproduces the model card's recommended command as a reference point:

--model nvidia/Kimi-K2.5-NVFP4 --tensor-parallel-size 4 --tool-call-parser kimi_k2 --reasoning-parser kimi_k2 --trust-remote-code

But the assistant immediately notes two necessary adaptations: "we need TP=8, and we point to the local path." The actual command that follows is:

NCCL_PROTO=LL NCCL_P2P_LEVEL=SYS nohup /root/ml-env/bin/python3 -m vllm.entrypoints.openai.api_server \
    --model /shared/kimi-k2.5-nvfp4 \
    --tensor-parallel-size 8 \
    --tool-call-parser kimi_k2 \
    --reasoning-parser kimi_k2 \
    --trust-remote-code \
    --port 8000 \
    --disable-log-requests \
    > /tmp/vllm_kimi.log 2>&1 &
echo "Started PID: $!"

The command is dispatched via SSH to the remote machine (10.1.230.174), and the response confirms it started with PID 206293.

Anatomy of a Decision: Every Flag Explained

Every parameter in this command carries the weight of prior reasoning. The --tensor-parallel-size 8 flag, for instance, was not the assistant's default choice. The model card recommended TP=4, but the user had explicitly corrected this in [msg 2096]: "It's a 1T model, need TP8." The assistant accepted this correction without debate—a 1-trillion-parameter model, even compressed to 540GB via NVFP4 quantization, would strain four GPUs with 96GB each. Spreading across all eight GPUs reduced per-GPU memory pressure from ~135GB to ~68GB, leaving room for KV cache and intermediate buffers.

The NCCL environment variables (NCCL_PROTO=LL and NCCL_P2P_LEVEL=SYS) were carried forward from the GLM-5 deployment, where they had been discovered through extensive benchmarking in [msg 2073]. These settings configure NVIDIA's Collective Communications Library to use the "Low Latency" protocol and system-level P2P transport, bypassing the default shared-memory paths that had proven problematic on this multi-GPU setup. The assistant assumed—reasonably—that these optimizations would benefit the new model as well, since the underlying hardware topology had not changed.

The --tool-call-parser kimi_k2 and --reasoning-parser kimi_k2 flags came directly from the model card. These enable the model's native tool-calling and reasoning capabilities, which are part of the Kimi K2.5 architecture's value proposition. The --trust-remote-code flag was necessary because the model uses custom modeling code (a KimiK25ForConditionalGeneration class) that vLLM must load from the Hugging Face repository. The --disable-log-requests flag was a practical choice to keep the log files manageable during development. The --port 8000 flag maintained consistency with the previous GLM-5 deployment, allowing any client configurations to remain unchanged.

The Assumptions Beneath the Command

Every command encodes assumptions, and this one encodes several that deserve scrutiny. The most significant assumption was that vLLM 0.16.0rc2 (a nightly build) would support Kimi-K2.5-NVFP4 without modification. The model card recommended vLLM 0.15.0, but the user had explicitly asked for "latest vllm" to ensure Blackwell SM120 compatibility. The assistant was operating in a zone of uncertainty: the nightly build might have regressions or missing features compared to the tested release.

A deeper assumption concerned the FP8 KV cache configuration baked into the model's quantization config. The NVFP4 checkpoint ships with kv_cache_quant_algo set to FP8, but the assistant did not inspect this detail before launching. The model card's config inspection in [msg 2104] had revealed quantization_config containing FP8 settings, but the assistant's attention was focused on the model architecture parameters (hidden size, layers, experts) rather than the KV cache quantization scheme. This oversight would prove critical: as the chunk summary reveals, "no MLA attention backend on SM120 (RTX PRO 6000) supports FP8 KV cache" — the Triton MLA backend hardcodes NotImplementedError for FP8. The launch would fail, requiring a subsequent fix that removed kv_cache_quant_algo from the config to fall back to FP16 KV cache.

The assistant also assumed that the NCCL tuning from GLM-5 would transfer cleanly to the new model. This was a reasonable heuristic—the hardware topology was identical—but it ignored the possibility that different model architectures might have different communication patterns. DeepSeek V3 (the base architecture of Kimi-K2.5) uses MLA (Multi-head Latent Attention) with a compressed KV cache, which has a different communication profile than the GLM-5 architecture. The NCCL settings might have been suboptimal for this model, though they would later prove functional.

The Thinking Process Revealed

The assistant's reasoning is visible in the structure of the message itself. It begins by stating the current state ("540GB model, 1.2T still free") as a sanity check—confirming that the download completed correctly and that sufficient disk space remains. It then reproduces the model card's recommended command as a reference, showing that it is consciously adapting rather than blindly copying. The two modifications (TP=8 and local path) are explicitly noted, demonstrating awareness of the divergence from the canonical configuration.

The choice to use nohup and background the process reflects an understanding that model loading is a long-running operation that would block the SSH session. The log redirection to /tmp/vllm_kimi.log shows forethought about debugging: when the launch inevitably encounters issues (and it would), the logs would be available for inspection. The echo "Started PID: $!" at the end provides a handle for monitoring or killing the process if needed.

The Broader Significance

Message 2110 is more than just a launch command. It represents the transition from the setup phase to the deployment phase of the Kimi-K2.5 experiment. All the preparatory work—the download, the cleanup, the software installation—was in service of this moment. The command is the bridge between "we have the model files on disk" and "the model is serving requests."

It also represents a shift in strategy from the previous GLM-5 effort. With GLM-5, the assistant had to write custom patches for GGUF loading, implement a new Triton attention backend, and debug tensor parallelism sharding issues. With Kimi-K2.5-NVFP4, the hope was that everything would "just work" with the official vLLM codebase. The launch command embodies that hope: it uses standard flags, standard parsers, and standard paths. There are no custom patches, no workarounds, no hacks.

Yet the FP8 KV cache issue lurking beneath the surface would prove that even "native" support has its limits on cutting-edge hardware. The RTX PRO 6000 Blackwell GPUs (compute capability SM120) were so new that vLLM's Triton MLA backend had not yet implemented FP8 KV cache dequantization. The assistant's assumption that "latest vLLM" would handle everything was partially correct—the model loaded, the architecture was recognized, the parsers worked—but the hardware-software gap in the attention backend required a manual workaround.

Conclusion

Message 2110 is a deceptively simple artifact: a single bash command that launches a model server. But when read in context, it reveals the full arc of reasoning, decision-making, and assumption that characterizes complex ML infrastructure work. Every flag encodes a prior discovery or user instruction. Every environment variable carries the weight of earlier debugging sessions. The command is simultaneously a culmination and a beginning—the payoff for hours of setup work, and the first step into a new round of debugging and optimization.

In the end, the launch would succeed after resolving the FP8 KV cache issue, and the Kimi-K2.5-NVFP4 model would achieve ~60 tok/s single-request throughput with coherent output—a significant improvement over the unusable GLM-5 GGUF deployment. But none of that success was guaranteed at the moment this command was issued. Message 2110 captures the moment of transition, the leap of faith that every deployment requires: the belief that the accumulated preparation is sufficient, and that the model will run.