The Pivot That Defined a Production Stack: Deploying Native INT4 Kimi-K2.5

In the middle of an intense benchmarking session on an 8× RTX PRO 6000 Blackwell GPU machine, a single user message redirected the entire trajectory of the conversation. The message, sent at index 2337, reads:

Deploy and benchmark native kimi k2.5 - https://huggingface.co/moonshotai/Kimi-K2.5 that is int4; Free disk space first probably

On its surface, this is a straightforward instruction: download a new model and test it. But to understand why this message was written when it was, and why it carries the weight it does, one must appreciate the full arc of the session that preceded it — a multi-hour odyssey of model deployment, hardware bottleneck discovery, and strategic pivoting that had just reached a stunning climax.

The Context That Made This Message Inevitable

The session had been systematically working through a portfolio of 1-trillion-parameter models, searching for the optimal deployment on a unique hardware configuration: eight NVIDIA RTX PRO 6000 Blackwell GPUs connected over PCIe (not NVLink). This distinction — PCIe interconnects rather than a high-bandwidth fabric — would become the dominant theme of the entire investigation.

Earlier in [msg 2311] through [msg 2332], the team had just completed an exhaustive benchmark of MiniMax-M2.5, a 230B-parameter FP8 model using Grouped-Query Attention (GQA). The results were spectacular: with Tensor Parallelism 8 and Expert Parallelism enabled, MiniMax achieved nearly 4,000 tokens per second at high concurrency, with a flat throughput curve from C=256 through C=1024. This was a 3.2× improvement over the NVFP4 Kimi-K2.5 that had been tested earlier, which topped out around 1,200 tok/s due to its Multi-Head Latent Attention (MLA) architecture requiring expensive allreduce operations across all 8 GPUs on every token.

The MiniMax results had effectively proven a thesis: on PCIe-bound Blackwell hardware, GQA + smaller active parameters dramatically outperforms MLA + full 1T-parameter models for aggregate throughput. But the user's message at 2337 signals a deliberate return to the Kimi-K2.5 family — specifically, the native INT4 quantized version. Why?

The Reasoning Behind the Pivot

The NVFP4 variant of Kimi-K2.5 had already been tested and found wanting: ~61 tok/s single-stream, ~1,200 tok/s peak, with the bottleneck clearly identified as PCIe allreduce latency for the 61-layer MLA architecture. But the NVFP4 quantization was a relatively exotic format — NVIDIA's NVFP4, which required specific kernel support and had demonstrated compatibility issues on the SM120 architecture (the Blackwell compute capability).

The native INT4 version at moonshotai/Kimi-K2.5 represented a fundamentally different approach. It used the compressed-tensors library with symmetric INT4 quantization at group_size=32, and crucially, only the MoE routed expert weights were quantized — attention layers, shared experts, and the lm_head remained in full precision. This selective quantization strategy meant the model might preserve more quality while still achieving the dramatic memory reduction that INT4 provides.

The user's explicit mention — "that is int4" — reveals a precise understanding of the tradeoffs. The NVFP4 variant had failed not because Kimi-K2.5 was a bad model, but because the NVFP4 format interacted poorly with the hardware and the MLA architecture's communication pattern. The INT4 variant might sidestep those issues entirely: it didn't use FP8 KV cache (the config showed kv_cache_scheme: null), eliminating the SM120 FP8 kernel blocker that had plagued earlier attempts, and the INT4 quantization was a mature, well-supported path in vLLM.

The Assumption Embedded in the Instruction

The message contains a subtle but critical assumption: "Free disk space first probably." This parenthetical reveals the user's mental model of the system's state. They knew the NVFP4 model occupied 540GB on the shared storage, and they knew the INT4 model was listed at 595GB on Hugging Face. With only 929GB free on the 1.7TB volume (as later confirmed in [msg 2339]), the 540GB NVFP4 model needed to be deleted to make room.

This assumption was correct — and it reflects a deep understanding of the operational constraints. The user didn't need to ask "how much space do we have?" or "will it fit?" They already knew the disk layout, the model sizes, and the arithmetic required. This is the hallmark of an operator who has internalized the physical constraints of the system.

What the Message Achieved: A Cascade of Decisions

The message triggered an immediate and precisely orchestrated sequence of actions. The assistant responded with a structured todo list ([msg 2338]) containing five high-priority items: check disk space, delete the NVFP4 model, download the INT4 model, stop the running MiniMax service, deploy and benchmark Kimi-K2.5 INT4.

Each of these steps required non-trivial decision-making. The disk cleanup alone involved verifying that the NVFP4 model could be safely removed (it was no longer needed since the INT4 version was the target), confirming that no other data would be affected, and executing the deletion with proper error handling. The download, at 595GB across 64 safetensor shards, required monitoring over 28 minutes to completion.

But the most consequential decision was the deployment configuration. The assistant chose TP=8 (tensor parallelism across all 8 GPUs) for the 547GB model, with --tool-call-parser kimi_k2 and --reasoning-parser kimi_k2 to match the model's native capabilities. Notably, Expert Parallelism was not enabled — a deliberate choice, since the INT4 Kimi-K2.5, like its NVFP4 sibling, used MLA rather than GQA, and EP does not benefit MLA's communication pattern.

The Thinking Process Visible in the Response

The assistant's subsequent messages reveal a careful reasoning process. In [msg 2341], after fetching the model's config.json, the assistant noted:

"595GB needs ~540GB free after deleting NVFP4. Let me check: current used=755GB, NVFP4=540GB. After delete: 755-540=215GB used, so 1700-215=1485GB free. Plenty."

This arithmetic — performed before any action — demonstrates a systematic approach to resource management. The assistant also analyzed the model's quantization scheme from the config, observing that only MoE experts were quantized and that the KV cache scheme was null, which meant "no FP8 KV cache in this version (good, no SM120 issues)." This last observation connects directly to the painful debugging history from earlier in the session, where FP8 KV cache incompatibility on SM120 had blocked deployment entirely.

The Output Knowledge Created

This message, and the work it initiated, produced several forms of knowledge:

  1. Operational knowledge: The exact sequence of steps required to swap a 540GB model for a 595GB model on a shared storage volume with 1.7TB capacity, including cleanup of zombie GPU processes that held memory after the MiniMax service was stopped.
  2. Benchmark knowledge: The INT4 Kimi-K2.5 would ultimately achieve 82 tok/s single-stream and 2,276 tok/s at high concurrency — significantly outperforming the NVFP4 variant's 61 tok/s and ~1,200 tok/s, though still far below MiniMax's 4,000 tok/s peak.
  3. Architecture knowledge: The comparison between NVFP4 and INT4 quantization on the same model architecture provided a controlled experiment showing that quantization format alone could account for a 30%+ performance difference, independent of model architecture.

Mistakes and Incorrect Assumptions

The message itself contained no mistakes — it was a correct assessment of the situation and a well-motivated instruction. However, the assumption that the INT4 variant would solve the MLA allreduce bottleneck proved optimistic. While the INT4 version did improve throughput by ~30% over NVFP4, it still could not match MiniMax's GQA-based architecture on this PCIe-bound hardware. The fundamental bottleneck — 8 GPUs communicating over PCIe for every MLA attention step — was a hardware constraint that no quantization format could eliminate.

This is not a failure of the message's reasoning, but rather a testament to the value of empirical testing. The user's instruction to "deploy and benchmark" was precisely the right response to an open question: would INT4 quantization change the performance characteristics enough to make Kimi-K2.5 competitive? The answer, as the subsequent benchmarks would show, was "better, but not competitive with GQA models."

Conclusion

Message 2337 represents a pivotal moment in a larger narrative about hardware-aware model selection. It demonstrates how a single, well-informed instruction can cascade through a complex system — freeing disk space, downloading half a terabyte of model weights, cleaning up GPU memory, launching a new inference server, and ultimately producing the data that would inform the final production decision. The message's brevity belies its sophistication: it encodes an understanding of disk constraints, model quantization formats, hardware compatibility, and the relative performance of competing architectures. It is a masterclass in concise, high-leverage technical direction.