The Pivot to Native INT4: A Critical Decision Point in Model Deployment

In the rapidly evolving landscape of large language model deployment, few moments are as consequential as the decision to abandon one model variant for another. At message index 2341, the assistant makes exactly such a pivot, analyzing the native INT4 quantization of moonshotai/Kimi-K2.5 and determining that it is viable for deployment on an 8× Blackwell GPU system. This message represents a critical juncture where hardware constraints, quantization schemes, and deployment logistics converge into a single, decisive evaluation.

The Context: A Session of Rapid Pivots

To understand the significance of this message, one must appreciate the broader session context. The assistant had spent the preceding rounds deploying and benchmarking multiple 1-trillion-parameter models on a system with 8 NVIDIA RTX PRO 6000 Blackwell GPUs (each with 96GB of VRAM). The journey had been one of constant adaptation: from the NVFP4 Kimi-K2.5 (which hit PCIe allreduce bottlenecks for its MLA architecture), to the MiniMax-M2.5 FP8 model (which achieved an impressive ~4,000 tok/s with TP=8 and Expert Parallelism), and now back to Kimi-K2.5 in its native INT4 form.

The user's instruction at [msg 2337] was direct and pragmatic: "Deploy and benchmark native kimi k2.5 - https://huggingface.co/moonshotai/Kimi-K2.5 that is int4; Free disk space first probably." This instruction carried an implicit recognition that the NVFP4 variant had been a dead end — its FP8 KV cache had caused SM120 compatibility issues, and its PCIe allreduce bottleneck for the 61-layer MLA architecture had limited throughput to ~61 tok/s single-stream. The native INT4 variant promised to sidestep these problems.

What the Message Actually Says

The assistant's response at [msg 2341] is deceptively concise, but every line carries significant analytical weight. Let us quote it in full:

Key details: - 595 GB total, 64 safetensors shards (first is 995MB, rest ~9.81GB each) - INT4 quantization via compressed-tensors, group_size=32, symmetric - Only MoE experts are quantized — attention layers, shared experts, gate/up/down MLP, and lm_head are all excluded (see the ignore list) - Same DeepSeek V3 / MLA architecture as the NVFP4 version - kv_cache_scheme: null — no FP8 KV cache in this version (good, no SM120 issues)

>

595GB needs ~540GB free after deleting NVFP4. Let me check: current used=755GB, NVFP4=540GB. After delete: 755-540=215GB used, so 1700-215=1485GB free. Plenty.

This is followed by a todowrite update marking the disk space check as completed and the NVFP4 deletion as in progress.

The Reasoning Process: What the Assistant Evaluated

The message reveals a multi-faceted evaluation that draws on knowledge accumulated across the entire session. The assistant is reasoning about:

1. Storage feasibility. The model is 595 GB — a colossal download that requires careful disk management. The assistant had previously checked disk space at [msg 2339], finding 1.7TB total with 755GB used (540GB from NVFP4 Kimi, 215GB from MiniMax-M2.5). The arithmetic is straightforward: deleting the NVFP4 variant frees 540GB, leaving 1.3TB free — more than enough for the 595GB INT4 model. The calculation "1700-215=1485GB free" is slightly optimistic (it assumes the entire 1.7TB is available and ignores filesystem overhead), but the margin is comfortable.

2. Quantization scheme analysis. The assistant identifies that this model uses compressed-tensors INT4 quantization with group_size=32 and symmetric quantization. This is a fundamentally different approach from the NVFP4 variant (which used NVIDIA's FP4 format). The critical insight is the ignore list: only the MoE expert weights are quantized to INT4, while attention layers, shared experts, gate/up/down projections, and the language model head remain in higher precision (likely BF16). This selective quantization is a deliberate design choice by the model authors — MoE experts constitute the bulk of the model's parameters, so quantizing them to INT4 achieves most of the memory savings while keeping the attention mechanism (which is more sensitive to quantization error) at full precision.

3. Architectural continuity. The assistant notes that this is "Same DeepSeek V3 / MLA architecture as the NVFP4 version." This is important because it means the deployment infrastructure (vLLM's GGUF loader, the Triton MLA attention backend, the weight loading logic) should work without major modifications. The assistant had already invested significant effort in patching vLLM to support this architecture (see <msg id=2313-2324> for the MoE parallel config investigation), and architectural continuity means those patches remain relevant.

4. The FP8 KV cache elimination. This is perhaps the most consequential observation: kv_cache_scheme: null. The NVFP4 variant had attempted to use FP8 KV cache compression, which caused failures on the Blackwell SM120 architecture (as documented in segment 17). The INT4 variant sidesteps this entirely by not using any KV cache quantization. The assistant's reaction — "good, no SM120 issues" — reflects relief that a major source of deployment friction has been eliminated by design.</p>## Assumptions and Their Validity

The assistant makes several assumptions in this message, most of which are well-founded but worth examining:

Assumption 1: The NVFP4 model can be safely deleted. The assistant assumes that the NVFP4 variant is no longer needed. Given that the user explicitly asked to deploy the INT4 variant and the NVFP4 had demonstrated poor performance (PCIe bottleneck, FP8 KV cache issues), this is reasonable. However, it is worth noting that deleting a 540GB model is irreversible — if the INT4 variant failed to load, the team would have lost the fallback option. The assistant implicitly assumes the INT4 variant will work.

Assumption 2: The storage arithmetic is correct. The assistant calculates 1700 - 215 = 1485GB free after deletion. This assumes: (a) the 1.7TB figure is the usable capacity, not the raw capacity; (b) no other processes will consume disk space during the download; (c) the filesystem metadata overhead is negligible. These are reasonable assumptions for a ZFS pool (the mount point rpool/data/shared suggests ZFS), but the actual available space after deletion was 1.3TB (as confirmed at [msg 2342]), not 1.485TB — a discrepancy of ~185GB that may reflect ZFS snapshots, metadata reservations, or the difference between GiB and GB reporting.

Assumption 3: The INT4 model will load without FP8 KV cache issues. The assistant correctly identifies that kv_cache_scheme: null eliminates the SM120 FP8 compatibility problem. However, this assumption overlooks other potential issues: the INT4 quantization via compressed-tensors requires vLLM to have a compatible weight loader, and the model's KimiK25ForConditionalGeneration architecture class may require custom model code. The assistant does not explicitly verify vLLM compatibility in this message.

Assumption 4: The model is worth deploying despite its size. At 595GB, this model will consume most of the available disk and will require significant VRAM. The assistant implicitly assumes that the INT4 variant will outperform the NVFP4 variant (which achieved ~61 tok/s) and approach the MiniMax-M2.5's performance (~4,000 tok/s at high concurrency). This assumption is based on the reasoning that native INT4 quantization avoids the FP8 KV cache bottleneck and may have better GPU utilization characteristics.

Input Knowledge Required

To fully understand this message, the reader needs:

  1. Knowledge of the hardware configuration: 8× NVIDIA RTX PRO 6000 Blackwell GPUs, each with 96GB VRAM, connected via PCIe (not NVLink). This explains why PCIe allreduce is a bottleneck and why the assistant is sensitive to KV cache quantization schemes.
  2. Understanding of quantization formats: The distinction between NVFP4 (NVIDIA's FP4 format, used in the earlier model variant) and INT4 via compressed-tensors (an open-source quantization framework). The group_size=32, symmetric parameters indicate that weights are quantized in groups of 32 elements with a symmetric range (zero point = 0).
  3. Knowledge of the DeepSeek V3 / MLA architecture: Multi-head Latent Attention (MLA) is a memory-efficient attention mechanism that uses low-rank KV projections. The 61-layer MLA architecture was identified earlier as the source of PCIe allreduce bottlenecks because each layer requires allreduce operations for the KV projections.
  4. The SM120 architecture issue: Blackwell GPUs use compute capability SM120, which lacks native FP8 support in certain configurations. The NVFP4 variant's FP8 KV cache had caused failures, making the absence of FP8 KV cache in the INT4 variant a significant advantage.
  5. The session's history of patching vLLM: The assistant had previously modified vLLM's GGUF loader and weight utilities to support the Kimi-K2.5 architecture. The "Same DeepSeek V3 / MLA architecture" note confirms these patches should remain applicable.

Output Knowledge Created

This message produces several important outputs:

  1. A storage plan: The assistant establishes the order of operations: delete NVFP4 (540GB), then download INT4 (595GB). This is encoded in the todowrite structure, which tracks completion status.
  2. A compatibility verdict: The INT4 variant is deemed deployable. The assistant identifies no show-stopping issues — the architecture is known, the quantization scheme is supported (or at least not known to be unsupported), and the FP8 KV cache issue is absent.
  3. A size estimate: The 595GB figure (derived from the Hugging Face model repository inspection at [msg 2340]) informs the deployment planning. The assistant correctly notes that this is slightly larger than the NVFP4 variant (540GB), requiring the deletion to proceed first.
  4. A quantization architecture analysis: The observation that "only MoE experts are quantized" is a non-trivial insight derived from inspecting the model's configuration. This has implications for memory usage (attention layers remain in BF16, consuming significant VRAM) and performance (INT4 matrix multiplications for experts, BF16 for attention).

The Thinking Process: A Window into Deployment Reasoning

The message's structure reveals the assistant's cognitive process. It begins with raw data extraction (model size, shard count, quantization parameters), proceeds to compatibility analysis (architecture match, KV cache scheme), and concludes with logistical planning (disk space arithmetic, deletion schedule). This mirrors the workflow of an experienced ML engineer evaluating a new model variant for deployment.

The parenthetical observation — "(good, no SM120 issues)" — is particularly revealing. It shows the assistant connecting a current observation (null KV cache scheme) to a past failure mode (FP8 KV cache on SM120) and drawing a positive conclusion. This kind of cross-referencing between current data and historical context is a hallmark of effective troubleshooting.

The todowrite update is also significant. It shows the assistant structuring its work into a formal plan with completion tracking. The "in_progress" status on the NVFP4 deletion and "pending" on the download indicate that the assistant intends to execute these steps in sequence, not in parallel. This is sensible given the disk space constraint — the 540GB deletion must complete before the 595GB download can begin.

Broader Significance

This message sits at a pivot point in the session. The NVFP4 Kimi-K2.5 had been a disappointment — despite being a 1T-parameter model with NVIDIA's optimized FP4 format, it was bottlenecked by PCIe allreduce and FP8 KV cache incompatibility. The MiniMax-M2.5 had demonstrated that GQA architectures with smaller active parameters could achieve 4,000 tok/s on the same hardware. The native INT4 Kimi-K2.5 represents a third path: keep the full 1T-parameter MLA architecture but use aggressive INT4 quantization to reduce memory pressure and eliminate the FP8 KV cache dependency.

The assistant's analysis in this message sets the stage for what follows: a 36-minute download, a successful model load, and benchmark results showing 82 tok/s single-stream and 2,276 tok/s at high concurrency — significantly outperforming the NVFP4 variant and validating the pivot decision. The message thus represents not just a data-gathering exercise, but a strategic evaluation that determines the direction of the entire deployment effort.