The First Benchmark: Measuring INT4 Kimi-K2.5 on 8x Blackwell GPUs

In the high-stakes world of large language model deployment, the moment of first measurement carries an almost ceremonial weight. After hours of downloading, configuring, debugging, and waiting—36 minutes for a 547GB model to load across 8 GPUs—the first benchmark numbers arrive like a verdict. Message [msg 2365] in this opencode session captures exactly that moment: the assistant has just finished loading the native INT4 quantized version of Kimi-K2.5 (a 1-trillion-parameter MoE model from Moonshot AI) and is running its initial performance evaluation. The message is deceptively brief—a single bash command and its output—but it represents the culmination of an enormous engineering effort spanning multiple segments of work.

The Context: A Long Road to Deployment

To understand why this message matters, we must trace the path that led here. The session had been through a remarkable journey. Earlier segments had dealt with setting up NVIDIA drivers and CUDA on Ubuntu 24.04, wrestling with flash-attn compilation issues, and deploying the GLM-5-NVFP4 model using SGLang. Segment 13 through 17 involved a major pivot: downloading a 540GB GGUF model, patching vLLM to support the glm_moe_dsa architecture, debugging incoherent output caused by tensor parallelism sharding mismatches, fixing Triton MLA attention backend bugs, and ultimately achieving ~60 tok/s throughput.

Then came another pivot in segment 17: abandoning GLM-5 entirely for nvidia/Kimi-K2.5-NVFP4, a 540GB NVFP4-quantized model. That version achieved ~60 tok/s single-stream but was bottlenecked by PCIe allreduce across 8 GPUs for the 61-layer MLA (Multi-head Latent Attention) architecture.

Segment 18, where our target message resides, began with yet another pivot. The user had benchmarked MiniMax-M2.5 (a 230B FP8 GQA model) and achieved remarkable results—up to 4,000 tok/s with Expert Parallelism—but then issued a new instruction at [msg 2337]: "Deploy and benchmark native kimi k2.5 - https://huggingface.co/moonshotai/Kimi-K2.5 that is int4; Free disk space first probably."

This kicked off a fresh cycle: deleting the NVFP4 model (540GB), downloading the INT4 variant (547GB, 64 safetensor shards), stopping the MiniMax service, cleaning up zombie GPU processes, and launching the new model. The download completed in ~28 minutes, but loading the model took another ~36 minutes—a testament to the sheer size of a 1T-parameter model even in INT4 quantization.

What the Message Actually Says

The message opens with the assistant's analysis:

Baseline: 81.4 tok/s single-stream — already faster than the NVFP4 version (61 tok/s). That's because INT4 has lower weight bandwidth requirements for the quantized MoE experts. But we want 40-50 tok/s per user, which at 512 max_tokens means the actual generation speed. Let me measure single-stream tok/s more precisely:

This is a remarkable result. The INT4 variant is already 33% faster than the NVFP4 version at single-stream throughput, despite being a larger model (547GB vs 540GB). The assistant correctly identifies the reason: INT4 quantization has lower weight bandwidth requirements for the quantized MoE expert weights. Since the model uses a "compressed-tensors" scheme where only the MoE routed expert weights are quantized to INT4 (while attention layers, shared experts, and the LM head remain in full precision), the effective memory bandwidth consumed per token is lower, allowing faster decoding.

The assistant then runs a more precise measurement using a direct curl call to the vLLM API server:

ssh root@10.1.230.174 'START=$(date +%s%N); curl -s http://localhost:8000/v1/chat/completions -H "Content-Type: application/json" -d "{\"model\": \"/shared/kimi-k2.5-int4\", \"messages\": [{\"role\": \"user\", \"content\": \"Write a detailed step-by-step guide on how to build a basic web application using Python and Flask.\"}], \"max_tokens\": 512, \"temperature\": 0.6}" 2>/dev/null | python3 -c "
import sys,json,time
r=json.load(sys.stdin)
u=r[\"usage\"]
c=u[\"completion_tokens\"]
p=u[\"prompt_tokens\"]
print(f\"Prompt: {p} tok, Completion: {c} tok\")
"
END=$(date +%s%N); ELAPSED=$(( (END - START) / 1000000 )); echo "Wall time: ${ELAPSED}ms"'
Prompt: 28 tok, Completion: 512 tok
Wall time: 6390ms

The result: 512 completion tokens in 6,390 milliseconds, yielding approximately 80 tokens per second. This confirms the benchmark script's measurement and establishes that the INT4 Kimi-K2.5 comfortably exceeds the user's target of 40-50 tok/s for single-stream performance.

The Reasoning and Decision-Making

The message reveals several layers of reasoning. First, the assistant interprets the benchmark result in context: 81.4 tok/s is "already well above" the target. But there's a subtlety here—the assistant notes that the user's target of "40-50 tok/s per user" was specified with the understanding that at 512 max_tokens, this represents the actual generation speed. The assistant is careful to validate this with a more precise measurement rather than relying solely on the benchmark script's aggregate numbers.

Second, the assistant makes an important architectural observation: the INT4 variant outperforms NVFP4 because of lower weight bandwidth requirements. This is a non-obvious insight. NVFP4 (NVIDIA's FP4 format) and INT4 both use 4-bit quantization, but the difference lies in how the quantized weights interact with the memory subsystem. INT4 weights can be loaded more efficiently from GPU memory, reducing the bandwidth bottleneck that was the primary limiter for the NVFP4 variant.

Third, the assistant implicitly decides that the current performance is acceptable but doesn't stop there. The message ends with the benchmark result, but the subsequent messages (which are outside our target but relevant context) show the assistant proceeding to NCCL tuning experiments anyway—trying Ring algorithm, LL protocol, channel counts, and buffer sizes—to see if further improvements are possible. This reflects a thorough engineering mindset: even when the target is met, explore the optimization space.

Assumptions and Their Validity

The message rests on several assumptions. The assistant assumes that the benchmark script's measurement of 81.4 tok/s is accurate and representative. This is validated by the subsequent curl-based measurement yielding ~80 tok/s, confirming consistency.

The assistant assumes that the user's target of "40-50 tok/s" refers to single-stream throughput for a single user. This is a reasonable interpretation given the context—the user had been benchmarking single-stream performance throughout the session.

The assistant assumes that the INT4 quantization scheme is the primary reason for the performance improvement over NVFP4. This is likely correct but worth examining: the NVFP4 version also used 4-bit quantization (FP4), but the INT4 version uses a different quantization scheme (compressed-tensors with group_size=32, symmetric) applied only to MoE expert weights. The NVFP4 version may have had different quantization granularity or applied quantization to different weight groups, affecting memory bandwidth.

One potential blind spot: the assistant doesn't explicitly verify that the model is producing coherent output before benchmarking. The smoke test at [msg 2361] confirmed basic functionality ("The capital of France is Paris"), but the benchmark uses a different prompt. If the model were producing garbage (as happened earlier with the GLM-5 GGUF deployment due to kv_b_proj sharding mismatches), the throughput numbers would be meaningless. The assistant implicitly trusts that the model is working correctly based on the earlier smoke test.

Input Knowledge Required

To fully understand this message, one needs knowledge of:

Output Knowledge Created

This message creates several pieces of valuable knowledge:

  1. The INT4 Kimi-K2.5 achieves ~80 tok/s single-stream on 8x Blackwell GPUs with TP=8. This is a concrete performance data point for anyone deploying this model on similar hardware.
  2. INT4 quantization outperforms NVFP4 for this model on this hardware. The 33% improvement (81 vs 61 tok/s) is significant and suggests that quantization format choice matters even at the same bit width.
  3. The model meets the user's throughput target comfortably. The 40-50 tok/s target is exceeded by a wide margin (~60-100% headroom).
  4. The benchmark methodology is validated. The assistant cross-checks the benchmark script result with a direct API call, ensuring the numbers are trustworthy.
  5. The bottleneck is not NCCL algorithm choice. Subsequent messages (outside our target) show that NCCL tuning (Ring, LL, channels, threads) produces nearly identical results (81.9 vs 81.4 tok/s), confirming that the bottleneck is fundamental hardware bandwidth rather than communication algorithm selection.

The Thinking Process

The message reveals a disciplined engineering thought process. The assistant:

  1. States the headline result (81.4 tok/s) and immediately contextualizes it (faster than NVFP4's 61 tok/s).
  2. Provides a causal explanation (INT4 has lower weight bandwidth requirements).
  3. Connects to the user's requirement (target is 40-50 tok/s, this exceeds it).
  4. Identifies a potential measurement concern (the benchmark uses a specific max_tokens setting, need to verify actual generation speed).
  5. Designs a more precise measurement (direct curl call with timing, specific prompt, 512 max_tokens).
  6. Executes and reports the result. This is a pattern of "trust but verify"—the assistant doesn't take the benchmark script's output at face value but independently validates it with a different methodology. This is particularly important in distributed systems where benchmark scripts can have subtle bugs (wrong warmup, incorrect timing, off-by-one errors in token counting).

Significance in the Broader Narrative

This message sits at a turning point in the session. The INT4 Kimi-K2.5 deployment is the final model the team will work with—subsequent messages show the assistant creating a systemd service (vllm-kimi-k25-int4.service) to make it a persistent production deployment. The benchmark results here establish that this model is not only viable but performs well, exceeding targets and providing a stable baseline.

The message also closes a loop that opened much earlier in the session. The team had been chasing performance across multiple models (GLM-5, NVFP4 Kimi-K2.5, MiniMax-M2.5, INT4 Kimi-K2.5), each with its own set of deployment challenges. The INT4 variant finally delivers the combination of scale (1T parameters), quality (native Kimi-K2.5), and performance (~80 tok/s) that makes it a suitable production candidate.

Conclusion

Message [msg 2365] is a deceptively simple benchmark report that encapsulates hours of prior engineering work and establishes the foundation for the session's final production deployment. It demonstrates the importance of cross-validation in performance measurement, the impact of quantization format choice on inference throughput, and the value of systematic benchmarking in large-model deployment. The 81.4 tok/s single-stream result—confirmed at ~80 tok/s via direct measurement—represents a successful outcome for a complex multi-GPU deployment of a 1-trillion-parameter model, achieved through careful hardware selection, software configuration, and iterative debugging across multiple model architectures.