The Capstone: Deploying a 1T-Parameter MoE Model on Blackwell GPUs

Introduction

In the course of a marathon coding session spanning dozens of rounds, the assistant reached a pivotal moment: message [msg 2167], a concise summary declaring that the nvidia/Kimi-K2.5-NVFP4 model—a 1-trillion-parameter Mixture-of-Experts (MoE) model based on the DeepSeek V3 architecture—was successfully deployed on a machine with 8 NVIDIA RTX PRO 6000 Blackwell GPUs. This message is not merely a status update; it is the capstone of an intense debugging and configuration effort that spanned model downloads, hardware compatibility investigations, KV cache architecture constraints, and systemd service integration. Understanding this message requires unpacking the journey that led to it, the technical decisions crystallized in its bullet points, and the assumptions—both correct and incorrect—that shaped the deployment.

The Subject Message

The assistant wrote:

All done. Here's the summary:

>

## Kimi-K2.5-NVFP4 is live

>

Model: nvidia/Kimi-K2.5-NVFP4 — 1T parameter MoE (DeepSeek V3 architecture), NVFP4 quantized by NVIDIA Service: vllm-kimi-k25.service on 10.1.230.174:8000 Config: TP=8, 128k context, 0.95 GPU memory utilization, tool calling + reasoning parser enabled

>

What was needed to make it work on SM120 (RTX PRO 6000): - Removed kv_cache_quant_algo: FP8 from hf_quant_config.json and kv_cache_scheme from config.json — the NVFP4 checkpoint ships with FP8 KV cache config, but no MLA attention backend on SM120 supports FP8 KV cache. Falls back to fp16 KV cache instead (TRITON_MLA backend) - Set --max-model-len 131072 (model defaults to 262k but only ~11.7GB KV cache available per GPU after 70.8GB model weights at 0.95 util) - Some CUTLASS SM120 grouped gemm tactics fail in flashinfer autotuner — harmless, it selects working alternatives

>

Performance: ~60 tok/s single request (including reasoning tokens), model loads in ~9 minutes

Why This Message Was Written: Context and Motivation

To appreciate why this message exists, one must understand the broader narrative. The session began with a completely different objective: deploying the GLM-5-NVFP4 model using GGUF quantization on vLLM. That effort consumed dozens of rounds—patching vLLM's gguf_loader.py, building custom tools like llama-gguf-split to merge 402 GB of split GGUF files, debugging Triton MLA attention backends, and fighting with tensor parallelism sharding mismatches. Despite heroic engineering, the GLM-5 deployment produced incoherent output, and the root cause was never fully resolved.

The pivot to Kimi-K2.5-NVFP4 represented a strategic reset. Rather than continuing to fight with a custom GGUF pipeline for an architecture (glm-dsa) that had minimal community support, the assistant switched to a model that was already packaged as a standard Hugging Face safetensor checkpoint—the NVFP4 format from NVIDIA. This decision immediately eliminated the GGUF patching burden, but introduced a new class of problems: the NVFP4 checkpoint came with configuration baked in for FP8 KV cache, which was incompatible with the Blackwell GPU architecture.

The message at [msg 2167] is the moment of closure. After hours of downloading 540 GB across 119 shards, debugging KV cache allocation errors, fighting with stuck GPU memory in an LXC container, and verifying model coherence across multiple prompts, the assistant could finally declare success. The summary format—concise, bullet-pointed, with clear "what was needed" sections—reflects the assistant's role as a technical documenter, distilling hard-won knowledge into actionable guidance.

The FP8 KV Cache Blocker: A Hardware-Software Boundary

The most significant technical finding in this message concerns the FP8 KV cache. The NVFP4 checkpoint from NVIDIA ships with kv_cache_quant_algo: FP8 in its hf_quant_config.json and kv_cache_scheme in config.json. This is a reasonable default: FP8 KV cache halves memory usage compared to fp16, which is critical for large-context deployments. However, the RTX PRO 6000 Blackwell GPUs (compute capability SM120) present a unique constraint.

The assistant's investigation (detailed in the surrounding messages) revealed that the TRITON_MLA attention backend—the only viable backend for Multi-head Latent Attention (MLA) on SM120—hardcodes a NotImplementedError for FP8 KV cache. This is not a bug or oversight; it reflects the reality that Blackwell's FP8 support in the MLA attention kernel path has not been implemented in the Triton compiler or the vLLM codebase. The assistant correctly identified that writing FP8 dequantization into the Triton MLA kernel would be a "major engineering effort" and instead chose the pragmatic workaround: stripping the FP8 configuration from the checkpoint's JSON files, forcing vLLM to fall back to fp16 KV cache.

This decision involved a critical assumption: that the model weights themselves (quantized to NVFP4) would still load and run correctly even if the KV cache operated at fp16 precision. The NVFP4 quantization applies to the model parameters (weights), not the KV cache activations. The KV cache quantization is a separate, orthogonal optimization for the attention layer's key-value states. Removing it does not corrupt the weights; it merely increases KV cache memory consumption. This assumption proved correct—the model loaded and produced coherent output.

The 128k Context Limit: Memory Arithmetic Under Constraints

The second workaround listed—setting --max-model-len 131072—reveals the tight memory budget on 96 GB GPUs. The model defaults to 262,144 tokens of context, but the assistant calculated that with 70.8 GB consumed by model weights per GPU (at 0.95 GPU memory utilization), only approximately 11.7 GB remained for KV cache. The fp16 KV cache for a 1T MoE model with MLA attention consumes roughly 65 bytes per token per GPU (accounting for the complex KV cache structure in MLA), meaning 262k context would require over 16 GB—an impossibility. The 128k limit brought the requirement down to ~8 GB, leaving headroom for batching.

This calculation reflects deep understanding of vLLM's memory model. The assistant had previously seen the error message from _check_enough_kv_cache_memory in [msg 2140] and knew exactly how to compute the constraint. The decision to use 128k rather than the 178k that vLLM reported as the maximum was conservative: "to be safe and leave room for batching" ([msg 2141]). This is a production-oriented mindset—not just getting the model to load, but ensuring it can serve multiple concurrent requests.

The CUTLASS SM120 Tactic Failures: An Expected Incompatibility

The third bullet point about CUTLASS grouped GEMM tactics failing in the flashinfer autotuner is almost an afterthought, but it represents a class of problems that could have derailed the deployment. The RTX PRO 6000 Blackwell GPUs use the SM120 compute architecture, which introduces new Tensor Core features (4th-gen Tensor Cores with TMA support). Many CUDA kernels and fused operations in the vLLM ecosystem were written for SM90 (Hopper) or earlier and have not been updated. The CUTLASS TMA (Tensor Memory Accelerator) grouped GEMM kernel, used for fused MoE expert routing, fails on SM120 with an assertion error.

The assistant correctly assessed this as harmless because the flashinfer autotuner tries multiple tactic implementations and falls back to working alternatives. This judgment required knowledge of the autotuner's architecture: it profiles each tactic on the actual hardware and selects the fastest working one. A tactic failure is logged as a warning, not an error. The model continues to function, albeit potentially with slightly suboptimal MoE kernel selection.

Assumptions, Correct and Incorrect

Several assumptions underpin this message. The most critical correct assumption was that removing FP8 KV cache configuration would not affect weight loading or inference correctness. This was validated by the coherence testing performed in the broader session—the assistant ran factual, multi-step, multi-turn, and creative prompts and verified that the model produced coherent, correct output with proper reasoning traces.

A subtler assumption was that the vLLM installation was "clean"—free of leftover patches from the GLM-5 GGUF effort. The assistant explicitly verified this ([msg 2167]'s surrounding context confirms all GLM-5-specific patches were absent). This mattered because the earlier GLM-5 deployment had produced incoherent output, and the assistant needed to rule out the possibility that stale patches were corrupting the Kimi model's behavior.

An assumption that proved incorrect was the expectation that the model would load with its default 262k context length. The assistant initially launched vLLM without --max-model-len, and the KV cache allocation failed ([msg 2140]). This was not a design flaw—it was a predictable consequence of the fp16 KV cache fallback consuming more memory than the FP8 configuration anticipated. The assistant corrected this by constraining the context length.

Input Knowledge Required

To fully understand this message, a reader needs knowledge of several domains:

Multi-head Latent Attention (MLA): The attention mechanism used by DeepSeek V3 and Kimi-K2.5. MLA compresses the KV cache by projecting keys and values into a low-dimensional latent space, then decompressing on-the-fly during attention computation. This makes KV cache memory behavior non-trivial—the per-token memory footprint depends on the latent dimension rather than the full head dimension.

NVFP4 Quantization: NVIDIA's 4-bit floating-point format for model weights. Unlike integer quantization (INT4/INT8), NVFP4 preserves the floating-point exponent range, which is important for outlier features in large models. The format is specific to NVIDIA's TensorRT-LLM and vLLM integration.

SM120 Architecture: The streaming multiprocessor design in Blackwell GPUs. SM120 introduces new Tensor Core capabilities (4th-gen with TMA) but also breaks compatibility with some CUDA kernels written for SM90 (Hopper). The RTX PRO 6000 Blackwell uses SM120.

Tensor Parallelism (TP=8): Distributing the model across 8 GPUs by splitting tensors along specific dimensions. For MoE models, this involves careful sharding of expert weights and attention projections. The assistant used TP=8, meaning each GPU holds 1/8 of the model.

Systemd Service Management: The deployment was packaged as a systemd unit file with ExecStartPre hooks for GPU memory cleanup, environment variables for NCCL tuning, and proper process lifecycle management.

Output Knowledge Created

This message creates several forms of output knowledge:

Deployable Artifact: The vllm-kimi-k25.service systemd unit, which can be started, stopped, and monitored like any system service. The model is accessible at 10.1.230.174:8000 via the OpenAI-compatible API.

Documented Workarounds: The three bullet points under "What was needed to make it work on SM120" constitute a troubleshooting guide for anyone attempting to deploy NVFP4 models on Blackwell GPUs. The FP8 KV cache removal, the context length constraint, and the CUTLASS tactic tolerance are all reusable findings.

Performance Baseline: The ~60 tok/s single-request throughput and ~9-minute load time establish a benchmark for this hardware configuration. The assistant later identified the PCIe allreduce bottleneck as the primary throughput limiter (~65-70% of decode time in NCCL communication), which is a crucial insight for anyone trying to optimize further.

Validation Evidence: The message implicitly communicates that the model produces coherent output. In the broader session, the assistant tested factual questions, multi-step reasoning, multi-turn conversation, and creative writing, all of which produced correct, sensible responses with proper reasoning traces. This validation was essential given the earlier GLM-5 coherence failures.

The Thinking Process Visible in the Message

While this message is a summary rather than a reasoning trace, the thinking process is embedded in its structure. The assistant chose to organize the information into three categories: what was deployed, what was needed to make it work, and what performance resulted. This triage reflects a mental model of deployment reporting: the "what" establishes identity, the "what was needed" captures the delta from a naive deployment, and the "performance" validates that the effort was worthwhile.

The ordering of the workarounds is also significant. The FP8 KV cache issue is listed first because it was the hardest problem—it required understanding the intersection of NVIDIA's checkpoint format, vLLM's attention backend architecture, and Blackwell's hardware capabilities. The context length constraint is second because it was a straightforward memory calculation once the fp16 KV cache decision was made. The CUTLASS tactic failures are last because they were harmless warnings that required no action.

This prioritization reveals the assistant's engineering judgment: solve the hard, architecture-level problems first, then tune the parameters, then document the noise. The message also omits many details that were critical during the journey—the GPU memory cleanup struggles, the systemd $$ escaping bug, the CUDAGraph compilation wait times—because they were transient operational issues, not enduring architectural findings. The summary distills signal from noise, which is the essence of technical communication.

Conclusion

Message [msg 2167] is a deceptively simple summary that encodes hours of complex engineering work. It marks the successful deployment of a 1-trillion-parameter MoE model on cutting-edge Blackwell GPUs, documents critical hardware-software incompatibilities (FP8 KV cache on SM120), establishes performance baselines, and provides reusable knowledge for future deployments. The message's concise format—model identity, required workarounds, performance numbers—is itself a thinking artifact, reflecting the assistant's prioritization of enduring architectural findings over transient debugging details. For anyone attempting to deploy large NVFP4 models on Blackwell hardware, this message is a compact but invaluable reference.