The Launch Command: Deploying Kimi-K2.5 INT4 on Eight Blackwell GPUs

Introduction

In the sprawling narrative of a multi-day machine learning deployment session, most messages are dense with debugging output, benchmark numbers, and iterative fixes. But occasionally a message arrives that is deceptively simple — a single command that encapsulates hours of accumulated knowledge, hardware discovery, and strategic pivots. Message [msg 2355] is such a moment. It is the launch command for the native INT4 variant of the Kimi-K2.5 model, a 547-billion-parameter MoE architecture, onto eight NVIDIA RTX PRO 6000 Blackwell GPUs. The message reads:

All clear. Now launch Kimi-K2.5 INT4. This uses MLA so needs TRITON_MLA on SM120, same as before. TP=8 required (547GB model). `` ssh root@10.1.230.174 "rm -f /dev/shm/psm_* /dev/shm/sem.mp-* /dev/shm/*vllm* /dev/shm/*nccl* 2>/dev/null; nohup env NCCL_PROTO=LL NCCL_P2P_LEVEL=SYS HF_HOME=/shared/huggingface TRANSFORMERS_CACHE=/shared/huggingface/hub /root/ml-env/bin/python3 -m vllm.entrypoints.openai.api_server --model /shared/kimi-k2.5-int4 --tensor-parallel-size 8 --tool-call-parser kimi_k2 --reasoning-parser kimi_k2 --trust-remote-code --max-model-len 131072 --gpu-memory-utilization 0.95 --port 8000 --disable-log-requests --enable-auto-tool-choice > /tmp/vllm_kimi_int4.log 2>&1 & echo PID=\$!" PID=235311 ``

On its surface, this is a routine deployment command. But beneath the shell syntax lies a dense concentration of engineering judgment, hardware constraints, and lessons learned across dozens of earlier failures and successes. This article unpacks that single message — the reasoning behind every flag, the assumptions baked into each parameter, and the context that made this particular invocation a turning point in the session.

The Strategic Pivot: Why This Model, Why Now

To understand why this message was written, one must understand the journey that preceded it. The session had been a whirlwind tour of 1-trillion-parameter model deployment on Blackwell hardware. Earlier, the NVFP4 variant of Kimi-K2.5 had been deployed and benchmarked, but it suffered from a fundamental bottleneck: PCIe allreduce across eight GPUs for the 61-layer MLA (Multi-head Latent Attention) architecture. The NVFP4 variant topped out at roughly 60 tok/s single-stream — respectable but far from the hardware's potential.

The team then pivoted to MiniMax-M2.5, a 230-billion-parameter FP8 model using Grouped Query Attention (GQA) instead of MLA. This model was a revelation: it loaded in 75 seconds, used only 4 GPUs with TP=4, and achieved over 2,500 tok/s at high concurrency. With TP=8 and Expert Parallelism (EP), it pushed nearly 4,000 tok/s. The lesson was clear: on PCIe-bound Blackwell hardware, GQA with smaller active parameters dramatically outperformed MLA with full parameter activation.

Yet the user's instruction at [msg 2337] was unambiguous: "Deploy and benchmark native kimi k2.5 - https://huggingface.co/moonshotai/Kimi-K2.5 that is int4." This was not a technical recommendation but a business requirement — the Kimi-K2.5 INT4 model was the target deployment, and the engineering task was to make it work as well as possible. The message at [msg 2355] represents the culmination of that directive: the moment when all preparation was complete and the model was finally launched.

The Preparation: What Had to Happen First

Before this launch command could be written, an extensive chain of prerequisites had to be satisfied. First, the NVFP4 Kimi-K2.5 model (540GB) had to be deleted to free disk space. The shared storage filesystem had 1.7TB total, with 755GB used before cleanup. After removing the NVFP4 model, 1.3TB was free — barely enough for the 547GB INT4 download plus the 215GB MiniMax model still present.

Second, the download itself had to complete. The INT4 variant from HuggingFace (moonshotai/Kimi-K2.5) was 595GB on the hub, spread across 64 safetensor shards. The download took approximately 28 minutes at roughly 20GB/minute, monitored carefully through a polling loop that tracked shard count and total size. The final size on disk was 547GB — slightly less than the hub listing, likely due to compression or metadata differences.

Third, the MiniMax service had to be cleanly shut down and all GPU memory released. This proved non-trivial: zombie worker processes continued holding GPU memory even after the systemd service was stopped and disabled. The assistant had to use fuser to find processes holding /dev/nvidia* devices and kill them aggressively. Only after this cleanup did all eight GPUs show 0 MiB used.

Fourth, the assistant had to reason about which vLLM configuration would work for this model. The INT4 Kimi-K2.5 uses the same DeepSeek V3 / MLA architecture as the NVFP4 variant, but with a critical difference: the quantization scheme. The NVFP4 version had used FP8 KV cache (kv_cache_scheme: fp8), which caused compatibility issues on SM120 (Blackwell) GPUs. The INT4 variant explicitly sets kv_cache_scheme: null — no FP8 KV cache — which avoids that entire class of problems. This was confirmed by inspecting the model's config.json on HuggingFace ([msg 2339]).

The Command Anatomy: Every Flag Has a History

The launch command is a study in accumulated engineering knowledge. Each flag and environment variable carries the weight of previous failures.

NCCL_PROTO=LL and NCCL_P2P_LEVEL=SYS: These NCCL environment variables control the communication protocol and peer-to-peer level for GPU-to-GPU communication across the PCIe fabric. The LL (Low Latency) protocol was chosen after extensive NCCL tuning experiments earlier in the session. The SYS level forces P2P communication through system memory rather than NVLink, which is the correct choice for Blackwell GPUs connected via PCIe rather than NVSwitch. These settings were not arbitrary — they were the result of systematic benchmarking that ruled out alternatives like Ring, LL128, and various channel/thread configurations.

--tensor-parallel-size 8: The model is 547GB, which exceeds the 96GB memory of any single GPU. TP=8 distributes the model across all eight GPUs, with each holding roughly 68GB of model weights. This was not negotiable — the NVFP4 version had also required TP=8. However, the assistant's comment "TP=8 required (547GB model)" reveals an assumption that the INT4 variant cannot fit on fewer GPUs. This is correct given the 96GB per GPU limit and the need for KV cache overhead.

--tool-call-parser kimi_k2 and --reasoning-parser kimi_k2: These flags enable the model-specific parsers for tool calling and reasoning. The Kimi-K2.5 model has native support for function calling and chain-of-thought reasoning, and these parsers ensure the API server can properly format and parse these structured outputs. The assistant had used similar flags for MiniMax (--tool-call-parser minimax_m2 --reasoning-parser minimax_m2) and knew to look for the Kimi equivalents.

--trust-remote-code: This flag is required because the Kimi-K2.5 model uses custom modeling code hosted on HuggingFace (the auto_map in config.json points to modeling_kimi_k25.KimiK25ForCausalGeneration). Without this flag, vLLM would refuse to load the custom architecture. This is a security tradeoff — trusting remote code execution — that is standard practice for deploying models with custom architectures.

--max-model-len 131072: This sets the maximum sequence length to 131,072 tokens (128K). This matches the model's native context window and is consistent with the earlier NVFP4 deployment. It also constrains the KV cache size, which is critical for MLA models where the KV cache is a significant memory consumer.

--gpu-memory-utilization 0.95: This leaves 5% of GPU memory unallocated, providing headroom for temporary buffers, CUDA graphs, and memory fragmentation. The value 0.95 is aggressive but standard for production deployments where every megabyte counts. Earlier in the session, the team had experimented with different utilization values and found 0.95 to be stable.

--enable-auto-tool-choice: This enables the model to autonomously decide when to use tool calls, rather than requiring explicit tool choice from the client. This is part of the Kimi-K2.5's agentic capabilities.

The cleanup command: rm -f /dev/shm/psm_* /dev/shm/sem.mp-* /dev/shm/*vllm* /dev/shm/*nccl* removes shared memory artifacts left by previous vLLM and NCCL processes. These files can cause conflicts if a new process encounters stale semaphores or shared memory segments. This cleanup was learned the hard way — earlier launches had failed due to leftover shared memory from crashed processes.

Assumptions Embedded in the Launch

Every launch command carries assumptions, and this one is no exception. Some are well-founded; others are bets that will be resolved only when the model starts serving.

Assumption: The INT4 quantization will work correctly on SM120. The NVFP4 variant had required custom dequantization kernels and had caused numerous issues. The INT4 variant uses compressed-tensors format with group_size=32 and symmetric quantization, but only on the MoE routed expert weights. Attention layers, shared experts, and the lm_head remain in bfloat16. The assistant assumes that vLLM's INT4 support will handle this correctly on Blackwell GPUs. This is not guaranteed — earlier in the session, FP8 block quantization had failed on SM120 due to alignment issues (192 % 128 ≠ 0).

Assumption: TRITON_MLA backend works for INT4. The comment "This uses MLA so needs TRITON_MLA on SM120, same as before" assumes that the Triton MLA sparse attention backend, which was developed and debugged earlier for the NVFP4 variant, will work identically for the INT4 variant. The MLA attention mechanism is the same regardless of quantization format, so this is a reasonable assumption — but the INT4 model might have different tensor shapes or memory layouts that could trigger edge cases in the Triton kernels.

Assumption: TP=8 without EP is sufficient. Unlike the MiniMax deployment, which used Expert Parallelism (EP) to distribute experts across GPUs, this launch uses only tensor parallelism. The assistant's reasoning is that TP=8 is "required" due to model size, but doesn't consider EP as an additional optimization. For the NVFP4 variant, EP had not been used either. The tradeoff is that without EP, all GPUs must hold all expert weights (sharded only by tensor parallelism), which increases memory pressure but avoids the allreduce overhead of EP for expert routing.

Assumption: The model will load successfully. At this point, no one has verified that the downloaded 64 safetensor shards are intact, that the config.json is compatible with the vLLM version installed, or that the custom modeling code will execute without errors. The download completed without errors and all 64 shards are present, but the true test is the load itself. The assistant is proceeding with confidence based on the NVFP4 experience, but the INT4 format introduces new variables.

The Thinking Process: What the Message Reveals

The assistant's reasoning is visible in the brief comment that precedes the command. "All clear" signals that the GPU memory cleanup is complete — a non-trivial achievement given the zombie process struggle. "Now launch Kimi-K2.5 INT4" states the immediate objective. "This uses MLA so needs TRITON_MLA on SM120, same as before" connects this deployment to the earlier NVFP4 work, asserting that the attention backend solution is reusable. "TP=8 required (547GB model)" provides the sizing justification.

What's not visible in the message but is evident from the surrounding context is the strategic calculus. The assistant knows that the NVFP4 variant was slow (~61 tok/s single-stream) and that the MiniMax model was much faster (~4,000 tok/s at high concurrency). Yet here they are deploying another MLA model that will likely be slower than MiniMax. The unspoken reasoning is: this is what the user asked for. The engineering task is not to choose the fastest model but to make this specific model work as well as possible.

There is also a subtle tension in the message. The assistant has just spent hours proving that GQA models (MiniMax) are vastly superior on this hardware. The launch of Kimi-K2.5 INT4 represents a return to MLA territory — a step backward in performance terms, but a necessary one for the deployment goal. The message does not express hesitation or disappointment; it simply executes.

Input Knowledge Required

To fully understand this message, one needs knowledge spanning multiple domains:

  1. Hardware architecture: Understanding that SM120 refers to the Blackwell GPU compute capability, that PCIe allreduce is a bottleneck for 8-GPU communication, that 96GB HBM3 memory per GPU constrains model sizing, and that NCCL protocols (LL, LL128, Ring) have different performance characteristics on PCIe versus NVLink topologies.
  2. Model architecture: Knowledge that Kimi-K2.5 uses DeepSeek V3-style MLA (Multi-head Latent Attention) with 61 layers, that it is a Mixture-of-Experts model with routed and shared experts, and that the INT4 quantization via compressed-tensors applies only to specific weight groups.
  3. vLLM internals: Understanding of tensor parallelism (TP) versus expert parallelism (EP), the role of TRITON_MLA backend for MLA attention on non-NVIDIA architectures, the significance of --trust-remote-code for custom model architectures, and the memory management implications of --gpu-memory-utilization.
  4. System administration: Knowledge of shared memory cleanup (/dev/shm), process management with fuser and kill, systemd service lifecycle, and GPU memory monitoring with nvidia-smi.
  5. Session history: The debugging journey that led to the NCCL protocol selection, the Triton MLA backend development, the FP8 KV cache incompatibility resolution, and the GGUF dequantization shard ordering fix — all of which inform the confidence behind this launch.

Output Knowledge Created

This message produces several forms of knowledge:

  1. A running vLLM API server serving the Kimi-K2.5 INT4 model on port 8000, which will be the subject of subsequent benchmarking and eventual production deployment.
  2. A log file at /tmp/vllm_kimi_int4.log that will capture any errors during model loading, weight initialization, and first inference — providing immediate feedback on the assumptions made.
  3. A process ID (235311) that can be used for monitoring, debugging, or termination.
  4. A benchmarkable configuration — the specific combination of TP=8, NCCL_PROTO=LL, NCCL_P2P_LEVEL=SYS, and gpu-memory-utilization=0.95 — that can be compared against the NVFP4 and MiniMax baselines.
  5. A validation point for the hypothesis that the INT4 variant will avoid the FP8 KV cache issues that plagued the NVFP4 deployment on SM120.

Mistakes and Potential Pitfalls

While the message is technically sound, several potential issues lurk beneath the surface.

The NCCL settings may be suboptimal for this specific model. The NCCL_PROTO=LL and NCCL_P2P_LEVEL=SYS were tuned for the MiniMax model, which uses GQA. The Kimi-K2.5 uses MLA, which has different communication patterns — MLA requires allreduce for the latent attention projections, which may benefit from different NCCL settings. The assistant is applying a one-size-fits-all solution that may not be optimal.

No warmup or verification step. The command launches the server but does not include any health check or test query. The assistant will need to poll for readiness separately (as was done for MiniMax). If the model fails to load, the error will be captured in the log file but the assistant won't know until the next round.

The --enable-auto-tool-choice flag may cause issues. This feature enables the model to autonomously decide when to use tool calls. If the model's tool-calling behavior is not well-calibrated, it could produce unexpected outputs or infinite reasoning loops. This is a feature flag that was not present in the NVFP4 deployment and represents new risk.

The model may be slower than expected. Based on the NVFP4 experience, single-stream throughput was ~61 tok/s. The INT4 variant should be faster due to reduced memory bandwidth pressure from the quantized weights, but the MLA attention remains the dominant cost. The assistant may be hoping for 80+ tok/s but has not set explicit expectations.

Conclusion

Message [msg 2355] is a masterclass in condensed engineering communication. In a few lines of shell command, it captures the culmination of a multi-day journey through driver installation, kernel debugging, model downloading, process management, and hardware discovery. Every flag tells a story of a failure overcome. Every environment variable encodes a benchmark comparison. The message is simultaneously a launch command, a hypothesis about what will work, and a bet on the accumulated knowledge of the session.

What makes this message particularly interesting is what it does not say. It does not express doubt about returning to MLA after proving GQA's superiority. It does not second-guess the user's model choice. It does not hedge with contingency plans. It simply executes — clean, confident, and forward-moving. This is the voice of engineering maturity: the ability to hold multiple competing considerations in mind while committing to a single course of action.

The Kimi-K2.5 INT4 launch would go on to achieve 82 tok/s single-stream and over 2,200 tok/s at high concurrency — significantly outperforming the NVFP4 variant and exceeding the 40-50 tok/s target. But at the moment this message was written, none of that was known. There was only the command, the accumulated knowledge, and the hope that it would work.