A 547GB Model in 28 Minutes: The Download That Unlocked INT4 Inference on Blackwell

"Download complete — 547GB, all 64 safetensors in ~28 minutes."

This single sentence from the assistant in message [msg 2346] marks a quiet but critical inflection point in a marathon coding session spanning hardware bringup, kernel debugging, and model deployment across eight NVIDIA RTX PRO 6000 Blackwell GPUs. The message is deceptively brief — a terse status report followed by a verification command — but it represents the culmination of a carefully orchestrated sequence of storage management, network utilization, and architectural decision-making. To understand why this message matters, one must trace the threads that converged on it: the pivot from a problematic NVFP4 quantized model to a native INT4 variant, the ruthless prioritization of disk space, and the strategic bet that INT4 quantization would overcome the PCIe bandwidth bottleneck that had plagued earlier deployments.

The Message in Full

The assistant wrote:

Download complete — 547GB, all 64 safetensors in ~28 minutes. Let me verify: ``bash ssh root@10.1.230.174 "du -sh /shared/kimi-k2.5-int4/; ls /shared/kimi-k2.5-int4/model-*.safetensors | wc -l; ls /shared/kimi-k2.5-int4/config.json /shared/kimi-k2.5-int4/tokenizer.json 2>/dev/null" `` 547G /shared/kimi-k2.5-int4/ 64 /shared/kimi-k2.5-int4/config.json

The output confirms three things: the model occupies 547GB on disk, all 64 safetensor shards are present, and the configuration file exists. Notably, tokenizer.json is absent from the output — the ls command listed both config.json and tokenizer.json but only config.json appeared, suggesting the tokenizer file was either not yet downloaded, stored elsewhere, or named differently. This detail would become relevant in subsequent deployment steps.

The Strategic Pivot That Led Here

To grasp why this download was happening at all, one must look back at the session's trajectory. The team had spent hours deploying and benchmarking the NVFP4 variant of Kimi-K2.5, a 540GB model using NVIDIA's FP4 quantization format. While it loaded successfully and produced coherent output, its throughput was disappointing: roughly 61 tok/s single-stream, bottlenecked by PCIe allreduce operations across 8 GPUs for the model's 61-layer MLA (Multi-Head Latent Attention) architecture. The fundamental problem was that MLA requires all-to-all communication patterns that saturate PCIe bandwidth, and no amount of NCCL tuning (Ring vs. LL protocols, channel counts, thread configurations) could escape this hardware constraint.

The assistant then pivoted to MiniMax-M2.5, a 230B FP8 model using grouped-query attention (GQA) instead of MLA. With TP=8 and Expert Parallelism (EP), MiniMax achieved nearly 4,000 tok/s at high concurrency — a stunning result that proved GQA + smaller active parameters was vastly superior on PCIe-bound Blackwell hardware. But the user's goal remained deploying Kimi-K2.5, the 1T-parameter MoE model from Moonshot AI. The NVFP4 variant was bandwidth-limited; perhaps the native INT4 variant would fare better.

The user's instruction in [msg 2337] was precise: "Deploy and benchmark native kimi k2.5 - https://huggingface.co/moonshotai/Kimi-K2.5 that is int4; Free disk space first probably." The assistant immediately recognized the disk space constraint: the NVFP4 model occupied 540GB, MiniMax occupied 215GB, and the HuggingFace cache used 249MB — totaling 755GB on a 1.7TB volume. The INT4 model was listed at 595GB on HuggingFace, so the NVFP4 model had to be sacrificed.

Decisions Made in the Preceding Messages

Several critical decisions preceded message [msg 2346]:

Storage prioritization: The assistant chose to delete the NVFP4 model entirely rather than attempting a partial cleanup. This was a high-stakes decision — if the INT4 download failed or the model didn't work, there would be no fallback. The assistant calculated that 540GB freed + 929GB available = ~1.5TB, more than enough for the 595GB download.

Download methodology: Rather than using git clone or wget, the assistant used huggingface_hub.snapshot_download with max_workers=8 for parallel shard downloading. This was the right choice for a 595GB model with 64 shards — sequential download would have taken hours.

Background execution with monitoring: The download was launched with nohup in a background Python process, and the assistant set up a monitoring loop that checked progress every 60 seconds. This allowed the conversation to continue while the download proceeded, a pragmatic choice given the 30-second bash timeout.

Assumptions Embedded in This Message

The assistant made several assumptions when crafting this verification:

That 547GB was the correct size: The HuggingFace listing showed 595GB, but the actual download was 547GB — an 8% discrepancy. The assistant implicitly assumed this was normal (perhaps the HF listing included redundant files or the compressed-tensors format had overhead that was overestimated). This assumption turned out to be correct, but it was never explicitly questioned.

That 64 safetensor shards was the expected count: The monitoring loop in [msg 2345] tracked progress as "X/64 safetensors," confirming 64 was the expected number. The assistant assumed the HuggingFace repository structure was stable and complete.

That config.json was sufficient to verify model integrity: The verification checked for config.json and tokenizer.json, but only config.json was found. The assistant did not flag the missing tokenizer — perhaps assuming it was embedded in the model files or would be downloaded separately. This assumption could have caused issues during model loading, though in practice vLLM often handles tokenizer discovery gracefully.

That the download was error-free: The assistant trusted that the absence of error messages in the monitoring loop meant the download was clean. With 64 multi-gigabyte files, silent corruption is possible but was not checked via checksums.

Input Knowledge Required

To fully understand this message, a reader needs:

Storage arithmetic: The assistant calculated that 755GB used - 540GB (NVFP4) = 215GB remaining, then 1.7TB - 215GB = ~1.5TB free. This required knowing the filesystem size, current usage, and the size of each model directory.

HuggingFace model repository structure: Understanding that snapshot_download with local_dir creates a flat directory of safetensor shards plus configuration files, and that 64 shards for a 595GB model implies ~9.3GB per shard (consistent with the monitoring output showing ~9.81GB per shard).

Compressed-tensors format awareness: The assistant had previously inspected the model's config.json and noted that only MoE experts were INT4-quantized via compressed-tensors with group_size=32, while attention layers and other components remained in bfloat16. This explains why the model is 547GB rather than the ~230GB one might expect from pure INT4 — most of the model is still in higher precision.

Network throughput expectations: 547GB in 28 minutes implies ~326 MB/s sustained throughput, which is excellent for a shared storage system. The assistant implicitly trusted this was correct without cross-checking against expected network bandwidth.

Output Knowledge Created

This message establishes several facts that drive subsequent actions:

The INT4 Kimi-K2.5 is 547GB on disk, not 595GB as listed on HuggingFace. This means it will fit comfortably alongside MiniMax (215GB) with room to spare on the 1.7TB volume.

All 64 shards downloaded successfully, meaning the model can be loaded without missing weights. This is critical — a missing shard would cause a KeyError during weight loading, a problem the team had debugged extensively with the NVFP4 variant.

The model directory structure is intact, with config.json present. The assistant can proceed to stop the MiniMax service and launch the Kimi-K2.5 INT4 model without additional download steps.

The download pipeline works reliably for multi-hundred-gigabyte models. This is operational knowledge: future model deployments can use the same snapshot_download approach with confidence.

The Thinking Process Visible in the Reasoning

The assistant's reasoning, visible across the preceding messages, reveals a methodical approach to large-model deployment:

  1. Constraint identification: "595GB needs ~540GB free after deleting NVFP4" — the assistant immediately identified the storage constraint and calculated the arithmetic.
  2. Risk assessment: The assistant noted that the INT4 model had kv_cache_scheme: null — no FP8 KV cache — which was "good, no SM120 issues." This references earlier struggles where FP8 KV cache quantization was incompatible with the Blackwell SM120 architecture. The assistant was proactively checking for known failure modes.
  3. Parallelism awareness: Using max_workers=8 for the download was a deliberate choice to maximize throughput without overwhelming the server. The monitoring showed the download rate was roughly 20GB/minute, which is near the limit of what a single NVMe-backed ZFS pool can sustain for writes.
  4. Verification discipline: Despite the download appearing successful, the assistant immediately ran a verification command rather than assuming success. This habit was forged by earlier sessions where silent failures (missing shards, corrupt files) caused hours of debugging.

The Broader Significance

Message [msg 2346] is the moment when weeks of infrastructure work — driver installation, CUDA toolkit configuration, flash-attn compilation, vLLM patching, NCCL tuning — finally converges on a clean download of the target model. The 547GB figure is not just a file size; it represents the successful navigation of every obstacle the session encountered. The NVFP4 variant was deleted not because it failed, but because the INT4 variant promised better performance through more efficient quantization. The download completed in 28 minutes not because of luck, but because the assistant correctly estimated disk space, chose the right download tool, and monitored progress without blocking the conversation.

In the messages that follow, this model would be loaded onto 8 GPUs in 36 minutes, benchmarked at 82 tok/s single-stream (nearly double the NVFP4 variant's 44 tok/s), and deployed as a production systemd service. But none of that would have been possible without the clean download confirmed in this brief, unassuming message. Sometimes the most critical moments in a coding session are not the dramatic breakthroughs but the quiet verifications that everything is in order before the next big step.