The 36-Minute Threshold: Loading a 547GB INT4 Model Across 8 Blackwell GPUs

In any large-scale ML deployment, there comes a moment when all the preparation, debugging, and infrastructure work converges into a single point of truth: the model loads, the GPUs fill with weights, and the system declares itself ready. Message [msg 2360] captures exactly such a moment. It is deceptively brief — a single line of text followed by an nvidia-smi command and its output — but it represents the culmination of hours of careful engineering, the crossing of a critical threshold, and the prelude to a round of benchmarking that would ultimately determine the viability of an entire deployment strategy.

The message reads in full:

Ready after ~36 minutes total load time. Let me smoke test and benchmark: [bash] ssh root@10.1.230.174 "nvidia-smi --query-gpu=index,memory.used,memory.total --format=csv,noheader" 0, 96733 MiB, 97887 MiB 1, 96733 MiB, 97887 MiB 2, 96733 MiB, 97887 MiB 3, 96733 MiB, 97887 MiB 4, 96733 MiB, 97887 MiB 5, 96733 MiB, 97887 MiB 6, 96733 MiB, 97887 MiB 7, 96733 MiB, 97887 MiB

To understand why this message matters, one must appreciate the journey that led to it. The user had just asked the assistant to deploy and benchmark the native INT4 version of Kimi-K2.5, a 1-trillion-parameter Mixture-of-Experts model from Moonshot AI (see [msg 2337]). This was the third major model pivot in a single session. The team had already deployed and benchmarked the NVFP4 variant of Kimi-K2.5 (which topped out at ~61 tok/s single-stream, bottlenecked by PCIe allreduce across 8 GPUs), then pivoted to MiniMax-M2.5 (a 230B FP8 GQA model that achieved an astonishing ~4,000 tok/s at high concurrency with Expert Parallelism), and was now returning to the Kimi architecture with a different quantization scheme in hopes of better performance.

The Weight of 547 Gigabytes

The INT4 Kimi-K2.5 model, hosted at moonshotai/Kimi-K2.5 on Hugging Face, weighs 547GB spread across 64 safetensor shards. Downloading it took approximately 28 minutes over the network. Loading it into GPU memory — the process of reading those 64 shards from disk, dequantizing the INT4-compressed MoE expert weights, and distributing the tensors across 8 GPUs using tensor parallelism — took another 36 minutes. That is an extraordinarily long load time by any standard. Most production models load in seconds or a few minutes. A 36-minute load indicates either a severe bottleneck in the I/O path, an extremely large model, or both.

In this case, it is both. The model is 547GB of compressed weights, and vLLM must load each of the 64 safetensor shards sequentially (or with limited parallelism), dequantize the INT4 compressed-tensors format on the fly, and scatter the tensors to the correct GPU devices. The assistant's monitoring logs (see [msg 2357]) show that each shard took approximately 34 seconds to process, with the loading progressing from 22% to 100% over the course of roughly 26 minutes of active weight loading (plus additional time for initialization, KV cache allocation, and warmup).

GPU Memory Pressure: Living at 98.8% Capacity

The nvidia-smi output in the message reveals a striking detail: every one of the 8 RTX PRO 6000 Blackwell GPUs is using 96,733 MiB out of 97,887 MiB total. That is 98.8% memory utilization — a razor-thin margin of only 1,154 MiB (about 1.2GB) of headroom per GPU. This is not a comfortable operating point. In production serving, such tight memory pressure means that any spike in KV cache usage, any fragmentation in the memory allocator, or any increase in batch size could trigger an out-of-memory (OOM) error that would crash the serving process.

The assistant had configured --gpu-memory-utilization 0.95 (95%), but the actual usage climbed to 98.8%. This discrepancy suggests that vLLM's memory estimation was slightly off, or that the model's weight footprint plus the KV cache allocation for the configured --max-model-len 131072 exceeded the预留 budget. The 1.2GB of remaining headroom per GPU is barely enough for a few dozen concurrent request slots, let alone the hundreds or thousands that the team had been benchmarking with MiniMax.

This memory pressure also explains why the NVFP4 variant of Kimi-K2.5 had been abandoned earlier in the session. The NVFP4 format, while offering higher precision, likely had a larger memory footprint that simply could not fit within the 98GB per GPU budget. The INT4 quantization, by compressing only the MoE routed expert weights (while leaving attention layers, shared experts, and the LM head in full precision), struck a balance between model quality and memory fit.

The Significance of INT4 Quantization

The INT4 variant uses the compressed-tensors format with group_size=32 and symmetric quantization, applied exclusively to the MoE expert weights. This is a critical architectural detail. In a Mixture-of-Experts model like Kimi-K2.5, the routed experts constitute the bulk of the parameters — they are the "wide" part of the network that gives the model its massive scale. By quantizing only these weights to INT4, the model retains full precision for the attention mechanism (which is sensitive to quantization error) while dramatically reducing the memory footprint of the dominant parameter block.

The assistant had noted earlier (see [msg 2341]) that the model's configuration explicitly excludes attention layers, shared experts, gate/up/down MLP projections, and the language model head from quantization. This selective quantization strategy is a pragmatic compromise: it preserves the parts of the network most sensitive to numerical precision while aggressively compressing the parts where INT4 introduces acceptable error.

A Threshold Crossed

Message [msg 2360] functions as a threshold-crossing announcement. The "Ready" declaration signals that the 36-minute ordeal of loading is over, and the system is now in a state where it can serve requests. The assistant immediately follows with "Let me smoke test and benchmark," indicating that the real purpose of this deployment — performance measurement — is about to begin.

The subsequent messages (see [msg 2361] through [msg 2366]) reveal the payoff: the INT4 Kimi-K2.5 achieves 81.4 tok/s single-stream in the initial benchmark, and 80.1 tok/s on a 512-token generation task. This is dramatically better than the NVFP4 variant's 61 tok/s and well above the user's target of 40-50 tok/s. The INT4 quantization, by reducing the weight size of the MoE experts, has lowered the memory bandwidth required for each forward pass, allowing the PCIe-bound allreduce to keep up more effectively.

Assumptions and Hidden Decisions

The message rests on several implicit assumptions. First, the assistant assumes that the model has loaded correctly — that all tensors are in the right places, that the dequantization kernels produce correct results, and that the Triton MLA attention backend (required for Blackwell SM120 GPUs) is functioning properly. These are not trivial assumptions. Earlier in the session (see segments 14-16), the team had to debug incoherent model output caused by a tensor parallelism sharding mismatch in the kv_b_proj layer, and had to fix bugs in the Triton MLA attention backend. The fact that the model loaded without crashing is itself a testament to the accumulated debugging work.

Second, the assistant assumes that the 98.8% memory utilization is stable and sustainable. This is a risky assumption — any memory fragmentation or KV cache growth beyond expectations could cause an OOM. The assistant does not verify memory stability with a sustained load test before declaring readiness.

Third, the message implicitly assumes that the benchmark results will be favorable enough to justify the 36-minute load time and the 547GB disk footprint. This is a bet on INT4 quantization delivering the performance uplift needed to make Kimi-K2.5 competitive with MiniMax-M2.5 (which loaded in 75 seconds and achieved 4,000 tok/s). As the subsequent messages show, this bet pays off — but it was not guaranteed at the moment the message was written.

The Thinking Process

The assistant's reasoning in this message is minimal but purposeful. The phrase "Ready after ~36 minutes total load time" acknowledges the long wait and implicitly justifies why no progress was reported during that period. The "Let me smoke test and benchmark" signals a transition from the passive waiting phase to an active measurement phase. The choice of nvidia-smi as the first verification step is deliberate: it confirms that all 8 GPUs have memory allocated, that the allocation is uniform across devices (a sign of correct tensor parallelism), and that the memory pressure is within acceptable bounds. Only after this hardware-level sanity check does the assistant proceed to the actual smoke test (a curl request to the API endpoint).

The message also reveals a subtle but important aspect of the assistant's workflow: it treats the nvidia-smi output as a necessary precondition for further testing. If any GPU showed anomalous memory usage (e.g., one GPU with 0 MiB while others had 96GB), that would indicate a tensor parallelism failure and the assistant would need to debug before proceeding. The uniform memory distribution across all 8 GPUs is a silent confirmation that the model distribution logic in vLLM worked correctly.

Output Knowledge Created

This message creates several pieces of actionable knowledge:

  1. Load time: The Kimi-K2.5 INT4 model requires approximately 36 minutes to load on this hardware. This is a critical operational metric for anyone planning to deploy this model — it means that service restarts, scaling events, or failovers will incur a 36-minute downtime window.
  2. Memory footprint: The model consumes 96.7GB per GPU across 8 GPUs, totaling approximately 774GB of GPU memory. This confirms that the model requires at least 8 GPUs with 98GB each (or more GPUs with smaller memory) to deploy.
  3. Feasibility: The model loads successfully and the API responds to health checks. This is the first confirmation that the INT4 variant is a viable deployment option on Blackwell SM120 hardware with vLLM.
  4. Baseline for optimization: The 36-minute load time and 98.8% memory utilization establish a baseline against which future optimizations (e.g., faster weight loading, memory-efficient attention, or different quantization formats) can be measured.

Conclusion

Message [msg 2360] is a quiet milestone in a much larger narrative. It does not contain dramatic breakthroughs or complex technical analysis. It simply announces that a 547GB model has finished loading after 36 minutes, and that the GPUs are full. But in the context of the session — with its history of failed builds, incoherent output, tensor parallelism bugs, and hardware bottlenecks — this message represents a victory. The model is in memory. The system is ready. The benchmarks can begin. And as the subsequent messages show, the results would be well worth the wait.