The Benchmark That Closes the Loop: 60 tok/s on a 1T-Parameter MoE Model

"512 tokens in 8.5s = ~60 tok/s. That's quite good for a 1T MoE model with NVFP4 quantization."

This single sentence, appearing in message [msg 2147], is the quiet culmination of an extraordinary engineering journey. After dozens of rounds spanning failed builds, patched kernels, deleted weights, and architectural dead ends, the assistant finally has a number to report — and it is a good one. The message is brief, almost offhand, but it represents the moment when the entire deployment pipeline for nvidia/Kimi-K2.5-NVFP4 — a 1-trillion-parameter Mixture-of-Experts model quantized with NVIDIA's NVFP4 format — is validated as working and performant on a machine with 8 RTX PRO 6000 Blackwell GPUs (SM120 architecture).

The Context: A Long Road to Inference

To understand why this message was written, one must appreciate the journey that led to it. The session had been wrestling with large language model deployment for hours. Earlier segments covered the painful setup of flash-attention on Blackwell GPUs ([msg 2117] onward), the pivot from GLM-5 GGUF deployment to Kimi-K2.5-NVFP4, and a cascade of blockers that each threatened to derail the entire effort.

The immediate predecessor to this message is [msg 2146], where the assistant ran a benchmark command:

time curl -s http://localhost:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{"model": "/shared/kimi-k2.5-nvfp4", "messages": [{"role": "user", "content": "Count from 1 to 50."}], "max_tokens": 512, "temperature": 0.0}'

The result was 512 completion tokens in 8.509 seconds. The assistant received this output, did the arithmetic (512 ÷ 8.5 = 60.2), and formulated the subject message as a triumphant status update. The benchmark was the final validation step after a multi-hour deployment effort.

Why This Message Was Written

The message serves multiple simultaneous purposes:

1. Validation of the deployment. After the model loaded successfully in [msg 2144] and produced its first coherent output in [msg 2145], the assistant needed to measure steady-state performance — not the first-request speed that includes CUDAGraph warmup overhead. The "Count from 1 to 50" prompt was deliberately chosen to be a simple, non-reasoning-heavy task that would produce a long, predictable output, allowing the throughput measurement to be as clean as possible.

2. A milestone checkpoint. The todowrite block embedded in the message lists four completed high-priority tasks: stopping the old vLLM service, removing old GLM-5 GGUF weights, installing the latest vLLM, and downloading the Kimi-K2.5-NVFP4 model. This structured todo list serves as a log of what was accomplished, implicitly defining the scope of work for this deployment phase. The assistant is effectively closing out a chapter.

3. Calibration of expectations. The phrase "quite good for a 1T MoE model with NVFP4 quantization" is revealing. The assistant is not declaring victory unconditionally — it is qualifying the result against the difficulty of the task. A 1-trillion-parameter model spread across 8 GPUs, using a novel quantization format (NVFP4) on a relatively new GPU architecture (SM120 Blackwell), achieving 60 tokens per second for single-request decoding is genuinely impressive. But the assistant's tone is measured, suggesting awareness that this is a single-request benchmark and that real-world throughput under batch load may differ.

How Decisions Were Made

The decisions that led to this message were forged in the preceding rounds. Several critical choices stand out:

The pivot from GLM-5 to Kimi-K2.5. Earlier in the session, the assistant had been deeply invested in deploying the GLM-5 model using GGUF format. This required writing custom patches to vLLM's gguf_loader.py, implementing a new Triton MLA sparse attention backend for Blackwell, and debugging incoherent output caused by tensor parallelism sharding mismatches. The decision to abandon that effort and pivot to a different model entirely was not made lightly — it was driven by the realization that the GLM-5 GGUF approach was producing incoherent output that might require fundamental architectural fixes. The Kimi-K2.5-NVFP4 model, by contrast, was an NVIDIA-official quantized checkpoint with native vLLM support (via the KimiK25ForConditionalGeneration architecture), promising a cleaner path to deployment.

The FP8 KV cache workaround. One of the most critical decisions was how to handle the FP8 KV cache configuration embedded in the NVFP4 checkpoint. The model's hf_quant_config.json specified "kv_cache_quant_algo": "FP8", but the Triton MLA attention backend — the only backend that works on SM120 — hardcodes NotImplementedError for FP8 KV cache. The assistant diagnosed this in [msg 2127] and chose to surgically remove the FP8 KV cache configuration from both hf_quant_config.json and config.json, falling back to fp16 KV cache. This was a pragmatic tradeoff: sacrificing KV cache density (and thus maximum context length) for functional correctness. The decision was validated when the model loaded successfully.

The context length constraint. After the model loaded, the assistant discovered in [msg 2141] that the default 262k context length was too large given the 75GB weight footprint per GPU (leaving only ~11.7GB for KV cache). The decision to limit to 131072 tokens (128k) was a calculated compromise between retaining as much context as possible while leaving headroom for batching.

Assumptions Made

Several assumptions underpin this message and the work that led to it:

That 60 tok/s is "quite good." This assumption is relative. For a dense 7B model running on a single GPU, 60 tok/s would be mediocre. But for a 1T MoE model with 8-way tensor parallelism, quantized weights, and a novel attention backend, it represents a significant achievement. The assistant is implicitly benchmarking against the difficulty class of the problem, not against absolute speed.

That the benchmark is representative. The "Count from 1 to 50" prompt produces a repetitive, low-entropy output that may benefit from KV cache efficiency and predictable memory access patterns. A more complex reasoning task or a prompt with long input context would likely yield lower throughput. The assistant does not claim this is a comprehensive benchmark — it is a single data point.

That the model is functioning correctly. The earlier coherence test in [msg 2145] showed the model producing correct output ("The capital of France is Paris.") with a proper reasoning trace. The assistant assumes that the model's weights were loaded correctly, that the NVFP4 dequantization is accurate, and that the Triton MLA attention backend is producing correct mathematical results. These assumptions were validated by the coherence test, but they remain assumptions for the broader range of possible inputs.

That no leftover GLM-5 patches are interfering. The chunk summary notes that the assistant later verified that all GLM-5-specific patches were absent from the vLLM installation. This verification happened after the benchmark message, meaning the assistant was operating under the assumption (or at least the hope) that the clean pip install had overwritten any patched files.

Input Knowledge Required

To fully understand this message, one needs:

Knowledge of the hardware stack. The RTX PRO 6000 Blackwell GPUs use the SM120 architecture, which is the desktop/workstation variant of Blackwell. This matters because it determines which CUDA kernels and attention backends are available. The FLASH_ATTN_MLA, FLASHMLA, and FLASHINFER_MLA backends all require SM90 (Hopper) or higher but don't support SM120 specifically, leaving TRITON_MLA as the only viable option.

Knowledge of the model architecture. Kimi-K2.5 is based on the DeepSeek V3 architecture, which uses Multi-head Latent Attention (MLA). This is a non-standard attention mechanism that requires specialized kernel support. The NVFP4 quantization format uses 4-bit floating point for weights, which requires specific GEMM kernels (FLASHINFER_CUTLASS in this case).

Knowledge of vLLM internals. Understanding what "512 tokens in 8.5s" means requires knowing that vLLM uses CUDAGraph compilation, that the first request incurs warmup overhead, and that tensor parallelism across 8 GPUs introduces NCCL allreduce communication costs.

Knowledge of the preceding session history. The message references completed tasks (stopping old service, removing GLM-5 weights, installing vLLM, downloading model) that only make sense in the context of the broader deployment effort.

Output Knowledge Created

This message produces several valuable pieces of knowledge:

A verified performance baseline. Future work can compare against this 60 tok/s figure. If someone optimizes the NCCL allreduce bottleneck (identified later as consuming 65-70% of decode time), they can measure the improvement against this baseline.

Confirmation that NVFP4 works on SM120. This is non-trivial. The NVFP4 format was likely developed and tested on data-center Blackwell GPUs (B200/B100), not on the desktop RTX PRO 6000. This benchmark proves that the format, the dequantization kernels, and the attention backend all function correctly on the consumer/workstation variant.

A validated deployment recipe. The sequence of steps — download model, patch quantization config to disable FP8 KV cache, limit context length to 128k, use NCCL_PROTO=LL and NCCL_P2P_LEVEL=SYS — constitutes a repeatable deployment recipe for this model on this hardware.

Evidence that the model is coherent. While not explicitly stated in this message, the preceding coherence test ([msg 2145]) demonstrated that the model produces correct, reasoned output. This is important because the earlier GLM-5 deployment suffered from incoherent output that was never fully resolved.

The Thinking Process Visible in the Message

The message reveals the assistant's thinking through several cues:

The arithmetic is done explicitly. "512 tokens in 8.5s = ~60 tok/s" — the assistant performs the calculation inline, showing that it is actively evaluating the result, not just parroting a number. The bold formatting on the result emphasizes its significance.

The qualification is deliberate. "That's quite good for a 1T MoE model with NVFP4 quantization" — the assistant is contextualizing the result against the model's scale and quantization format. This shows an awareness of what constitutes good performance in this domain.

The todo list is structured as a log. The todowrite block with completed tasks serves as both a record of accomplishment and a definition of scope. The assistant is implicitly saying "here is what we set out to do, and here is what we achieved." The fact that all high-priority items are marked "completed" signals that this phase of work is done.

The message is positioned as a capstone. It follows immediately after the benchmark command and its output. The assistant could have simply noted the result, but instead chose to add evaluative commentary and a structured task summary. This suggests the assistant recognizes this as a milestone worth documenting.

What Comes Next

The message does not declare the work finished. The todo list shows only four completed items, and the chunk summary reveals that the assistant went on to investigate whether GLM-5 optimizations apply, verify the absence of leftover patches, and conduct multi-prompt coherence testing. The PCIe allreduce bottleneck was identified as the primary throughput limiter, with 65-70% of decode time spent in NCCL communication. The 60 tok/s figure, while impressive, is understood to be a ceiling imposed by the hardware interconnect, not by the model or software stack.

In this sense, the message is both a celebration and a calibration. It says: we have achieved something difficult, and here is the number that proves it. But we also know what is limiting us, and there is more work to do.