The Pivot: Deploying Native INT4 Kimi-K2.5 on Blackwell GPUs

In the sprawling, multi-day journey to deploy trillion-parameter language models on eight NVIDIA RTX PRO 6000 Blackwell GPUs, few moments capture the blend of technical deliberation and operational necessity as cleanly as message <msg id=2348>. At first glance, it is a brief, almost perfunctory instruction: stop the currently running MiniMax-M2.5 service and prepare to launch the native INT4 variant of Kimi-K2.5. But this message sits at a critical inflection point in the session, where weeks of debugging, kernel development, and benchmarking converge into a single deployment decision. It encapsulates the assistant's reasoning about model architecture compatibility, quantization format differences, and the accumulated knowledge of what makes these Blackwell GPUs tick.

The Context: A Whirlwind of Model Pivots

To understand why this message was written, one must trace the arc of the preceding hours. The session had been a relentless exploration of the hardware-software boundary for large language model inference. Earlier segments detailed the painful resolution of flash-attention build issues, the development of a custom Triton MLA sparse attention backend for Blackwell's SM120 architecture, and the debugging of incoherent model output caused by tensor parallelism sharding mismatches (see segments 14–16). The team had pivoted from GLM-5-NVFP4 to nvidia/Kimi-K2.5-NVFP4, then to MiniMax-M2.5 FP8, and finally benchmarked MiniMax at an impressive ~4,000 tok/s using TP=8 with Expert Parallelism (EP).

Then came the user's directive in <msg id=2337>: "Deploy and benchmark native kimi k2.5 - https://huggingface.co/moonshotai/Kimi-K2.5 that is int4; Free disk space first probably." This was a clear pivot away from the NVFP4 variant (which had been bottlenecked by PCIe allreduce for its 61-layer MLA architecture, achieving only ~61 tok/s single-stream) and toward the native INT4 quantization of the same model. The assistant dutifully checked disk space (1.7TB filesystem with 929GB free), deleted the 540GB NVFP4 model, and initiated a 28-minute download of the 547GB INT4 model across 64 safetensor shards.

By <msg id=2348>, the download was complete, and the assistant stood at the threshold: MiniMax was still running, consuming all eight GPUs, and the new model needed to be loaded. The message is the transition point.

The Reasoning: Why This Model, Why Now?

The assistant's opening sentence — "Now stop MiniMax and launch Kimi-K2.5 INT4" — is deceptively simple. It reflects a deliberate choice informed by the benchmarking data collected just minutes earlier. MiniMax-M2.5, a 230B FP8 GQA model, had achieved stellar throughput (nearly 4,000 tok/s at high concurrency), but it was a different architectural family. The user's explicit request was for Kimi-K2.5, the 1T-parameter DeepSeek V3-architecture model with Mixture-of-Experts and Multi-head Latent Attention (MLA).

The assistant then articulates the critical reasoning: "This model uses MLA (TRITON_MLA needed for SM120), same as the NVFP4 version." This single sentence carries enormous weight. It acknowledges that the INT4 variant shares the same fundamental architecture as the previously problematic NVFP4 Kimi-K2.5, meaning the hard-won Triton MLA backend — developed and debugged across multiple sessions specifically for Blackwell's SM120 compute capability — should be directly applicable. The assistant is implicitly betting that the weeks of work on the MLA attention kernel will pay off here, rather than requiring yet another custom kernel.

But the assistant also notes a crucial difference: "But the quantization is compressed-tensors INT4 with only the MoE routed expert weights quantized — attention layers are full precision." This observation is key. The NVFP4 variant used NVIDIA's NVFP4 quantization format across all weights, which had caused FP8 KV cache incompatibility issues on SM120 (requiring removal of kv_cache_quant_algo). The INT4 variant, by contrast, uses the open-source compressed-tensors format and only quantizes the MoE expert weights — the attention layers remain in full bfloat16 precision. This means fewer quantization-related pitfalls during loading, and potentially better output quality since the attention mechanism operates at full fidelity.

Assumptions Embedded in the Message

Every technical decision rests on assumptions, and this message is no exception. The most significant assumption is that the TRITON_MLA backend, which was developed and tested against the NVFP4 Kimi-K2.5, will work without modification for the INT4 variant. This is a reasonable assumption — both models share the same MLA architecture, and the Triton kernel operates on the attention mechanism, not the quantized expert weights. However, it is not guaranteed: if the INT4 model uses a different tensor layout, hidden dimension configuration, or attention head structure, the kernel could fail silently or produce incoherent output (as happened earlier with the kv_b_proj sharding mismatch in segment 15).

A second assumption is that vLLM's loading pipeline can handle the compressed-tensors INT4 format for this specific model architecture. The assistant had already patched vLLM extensively — modifying gguf_loader.py and weight_utils.py for the GLM-5 GGUF model, fixing a latent DeepSeek V2/V3 bug in kv_b_proj mapping, and implementing force-dequantization logic for quantized indexer weights. But those patches were for GGUF and NVFP4 formats, not for compressed-tensors. The assistant is implicitly trusting that vLLM's built-in compressed-tensors support (a relatively new feature) will handle the 547GB model correctly across eight GPUs.

A third, more subtle assumption is that the system has sufficient memory bandwidth and PCIe capacity to serve the INT4 model efficiently. The NVFP4 variant had been bottlenecked at ~61 tok/s single-stream due to PCIe allreduce across eight GPUs for the 61-layer MLA architecture. The INT4 variant, being smaller in bit-width (4-bit vs NVFP4's ~8-bit effective), might reduce memory pressure but does nothing to alleviate the allreduce bottleneck. The assistant seems to be hoping that the INT4 format's computational efficiency will compensate — a hope that would later be validated when the INT4 model achieved 82 tok/s single-stream, well exceeding the 40–50 tok/s target.

Input Knowledge Required

To fully grasp this message, one must understand several layers of prior knowledge. First, the distinction between MLA (Multi-head Latent Attention) and GQA (Grouped Query Attention) architectures: MLA is used by DeepSeek V3-derived models like Kimi-K2.5 and requires a specialized attention kernel, while GQA is used by MiniMax-M2.5 and is more straightforward to implement. The assistant had invested significant effort in developing a Triton-based MLA backend for Blackwell's SM120 architecture, which lacks native FP8 support for certain operations.

Second, the quantization landscape: NVFP4 is NVIDIA's proprietary 4-bit floating-point format, while compressed-tensors is an open-source library for weight quantization that supports INT4 with configurable group sizes. The message notes that only "MoE routed expert weights" are quantized — this refers to the gate, up, and down projection matrices within each Mixture-of-Experts layer, while the attention projections (q, k, v, o) and shared expert weights remain in full precision.

Third, the operational context: MiniMax was currently occupying all eight GPUs with ~96.6GB each, and the kill -9 command would free those resources. The nvidia-smi check afterward would confirm that GPU memory was released before launching the new model.

Output Knowledge Created

This message produces both explicit and implicit knowledge. Explicitly, it records the decision to terminate the MiniMax service and the reasoning behind the model choice. It documents the assistant's understanding of the INT4 model's quantization scheme — a detail that would be crucial for debugging any subsequent loading failures.

Implicitly, the message creates a hypothesis about model performance: that the INT4 Kimi-K2.5, despite being 547GB and using the MLA architecture that previously bottlenecked on PCIe allreduce, could outperform the NVFP4 variant. This hypothesis would be tested in the subsequent benchmarking, where the INT4 model achieved 82 tok/s single-stream (vs ~61 for NVFP4) and scaled to 2,276 tok/s at high concurrency — a significant improvement, though still below MiniMax's 4,000 tok/s peak.

The Thinking Process

The assistant's reasoning in this message reveals a methodical, pattern-matching approach to problem-solving. It immediately identifies the architectural commonality between the INT4 and NVFP4 variants ("same as the NVFP4 version"), leveraging prior debugging effort as a reusable asset. It then identifies the key difference (quantization format and scope), implicitly assessing risk: attention layers at full precision means fewer quantization artifacts, and compressed-tensors is a well-supported format in vLLM.

The message also demonstrates operational awareness. The assistant knows that MiniMax must be stopped before the new model can be loaded (GPUs are finite), and it verifies this with the nvidia-smi check. The sleep 3 between kill and check is a small but telling detail — it reflects experience with GPU memory release timing.

Conclusion

Message <msg id=2348> is a hinge point in a much larger narrative. It is the moment when weeks of kernel development, debugging, and benchmarking crystallize into a single deployment action. The assistant's brief analysis — noting architectural continuity, quantization differences, and operational prerequisites — encapsulates the technical judgment required to navigate the complex landscape of large model deployment on cutting-edge hardware. It is a reminder that in the world of AI infrastructure, the most important decisions are often the quietest ones: the choice of which model to run, the recognition of which prior work transfers, and the careful sequencing of stop-and-start operations that keep the system moving forward.