The Benchmark That Validated a Pivot: 81.4 tok/s from a 547GB INT4 Model on 8 Blackwell GPUs

The Message

[assistant] [bash] python3 /home/theuser/glm-kimi-sm120-rtx6000bw/benchmark.py
Warming up...

Benchmark results:
  --------------------------------------------------------------------------------
  C=   1 |   4/  4 ok (0 err) | wall=  25.2s |    81.4 tok/s | lat avg=  6.29 p50=  6.29 p99=  6.30
  C=   2 |   8/  8 ok (0 err) | wall=  30.8s |   133.1 tok/s | lat avg=  7.69 p50=  7.68 p99=  7.77
  C=   4 |  12/ 12 ok (0 err) | wall=  26.0s |   236.3 tok/s | lat avg=  8.66 p50=  8.64 p99=  8.72
  C=   8 |  16/ 16 ok (0 err) | wall=  19.8s |   413.1 tok/s | lat avg=  9.91 p50= ...

This single message, message 2364 in the conversation, represents the first performance measurement of the native INT4 Kimi-K2.5 model (moonshotai/Kimi-K2.5) after a grueling 36-minute load time onto 8 NVIDIA RTX PRO 6000 Blackwell GPUs. The numbers it reports — 81.4 tokens per second at a concurrency of 1 — would immediately reshape the trajectory of the entire deployment effort.

Context: A Long Road to This Benchmark

To understand why this message matters, one must appreciate the journey that preceded it. The session had been running for hours across multiple segments, wrestling with model deployment on a machine equipped with 8 Blackwell GPUs connected via PCIe. Earlier attempts had deployed the GLM-5-NVFP4 model, then pivoted to the NVFP4 variant of Kimi-K2.5, and most recently deployed MiniMax-M2.5 FP8 — each with its own set of challenges.

The NVFP4 Kimi-K2.5 had been a disappointment. Despite its 540GB footprint and the promise of NVIDIA's FP4 format, it delivered only ~61 tok/s single-stream on the same hardware. The fundamental bottleneck was identified as PCIe allreduce: the 61-layer MLA (Multi-head Latent Attention) architecture required massive allreduce operations across all 8 GPUs for every single token, and the PCIe bus simply could not keep up.

The team then pivoted to MiniMax-M2.5, a 230B FP8 model using GQA (Grouped Query Attention) instead of MLA. This model used only 4 GPUs (TP=4) and achieved 84 tok/s single-stream, scaling to an impressive ~4,000 tok/s at high concurrency. But the user's goal was not MiniMax — it was Kimi-K2.5, a 1-trillion-parameter MoE model from Moonshot AI.

The critical insight came when the user requested: "Deploy and benchmark native kimi k2.5 — https://huggingface.co/moonshotai/Kimi-K2.5 that is int4." This was the native INT4 quantized version, using compressed-tensors with group_size=32 symmetric quantization applied only to the MoE expert weights. The attention layers, shared experts, and other components remained in full precision. At 547GB (slightly larger than the NVFP4 variant's 540GB), it was a beast to load.

What This Message Actually Shows

The benchmark output is deceptively simple. It reports five rows of data, each representing a different concurrency level (C), from a single concurrent request up to 8 concurrent requests. At each level, it shows the number of successful requests, the wall-clock time, the aggregate throughput in tokens per second, and latency statistics.

The headline number is 81.4 tok/s at C=1. This single-stream throughput is the most important metric for many real-world use cases — a coding assistant, for instance, where each user's requests are processed one at a time. The user had explicitly asked for "single stream >40~50" in the preceding message ([msg 2359]), making this result a clear success: 81.4 tok/s nearly doubled the target.

But the message also reveals the model's scaling behavior. At C=2, throughput reaches 133.1 tok/s (1.6x the single-stream rate). At C=4, it hits 236.3 tok/s (2.9x). At C=8, it reaches 413.1 tok/s (5.1x). This near-linear scaling at low concurrency is characteristic of a system that is not yet hitting memory bandwidth or compute saturation — each additional request fills idle capacity efficiently.

Why 81.4 tok/s Was a Surprise

The NVFP4 variant of the same model had achieved only ~61 tok/s on the identical hardware. The INT4 variant, despite being 7GB larger (547GB vs 540GB), was 33% faster. This counterintuitive result demands explanation.

The key lies in the quantization format and how it interacts with the hardware. NVFP4 is NVIDIA's proprietary 4-bit floating-point format, designed for the Blackwell architecture's native FP4 tensor core support. However, the NVFP4 variant of Kimi-K2.5 apparently required additional dequantization overhead or had less favorable memory access patterns. The INT4 variant, using symmetric integer quantization with group_size=32, likely benefited from more efficient memory bandwidth utilization — the quantized MoE expert weights could be loaded and dequantized with lower overhead per byte transferred.

Moreover, the INT4 variant had kv_cache_scheme: null — meaning no FP8 KV cache compression. This was actually a benefit on Blackwell SM120 GPUs, as the team had previously discovered that FP8 KV cache caused compatibility issues. By avoiding FP8 KV cache entirely, the INT4 model sidestepped a class of bugs that had plagued earlier deployments.

The Reasoning Behind the Benchmark

The assistant's decision to run this benchmark immediately after the model loaded reflects a disciplined experimental methodology. The sequence was:

Verify model loaded correctly ([msg 2360]): Check GPU memory (96,733 MiB used on each of 8 GPUs, out of 97,887 MiB total — 98.8% utilization).
Smoke test ([msg 2361]): Send a simple query ("What is the capital of France?") to confirm the model produces coherent output. It returned "The capital of France is Paris" with 364 reasoning characters and 80 completion tokens.
Update benchmark script (<msg id=2362-2363>): Modify the benchmark parameters to match the new model's configuration.
Run benchmark ([msg 2364]): Execute the actual performance measurement. This systematic approach — verify, smoke test, configure, measure — is characteristic of production deployment workflows. Each step validates a prerequisite before proceeding to the next, minimizing the risk of wasted time debugging at the wrong layer.

What the Assistant Did Not Yet Know

At the moment this message was produced, the assistant had only the raw benchmark numbers. It did not yet know:

Whether the 81.4 tok/s was sustainable for longer generations (the benchmark used relatively short sequences).
What the breakdown was between reasoning tokens and content tokens (later discovered to be roughly 50/50 at 512 max_tokens).
Whether NCCL tuning could push the number even higher.
How the model would behave under sustained high-concurrency load. These questions would be answered in subsequent messages. The assistant would go on to measure 512 tokens in 6.39 seconds (80.1 tok/s), confirming the benchmark's accuracy. It would then attempt NCCL tuning with Ring algorithm, LL protocol, and other parameters — though the baseline was already so good that tuning was arguably unnecessary.

Assumptions Embedded in This Message

Several assumptions are baked into this benchmark result:

Assumption 1: The benchmark script measures what matters. The script uses a fixed prompt length and generates a fixed number of tokens. Real-world workloads have variable prompt lengths, variable generation lengths, and different ratios of prefill to decode. The 81.4 tok/s number is a best-case measurement under controlled conditions.

Assumption 2: Warming up is sufficient. The script warms up before measuring, but a single warm-up may not fully stabilize GPU clock speeds, memory bandwidth, or NCCL communication patterns. Real production systems often require sustained load to reach thermal equilibrium.

Assumption 3: TP=8 is the right configuration. The model uses tensor parallelism across all 8 GPUs. This was necessary because the 547GB model could not fit on fewer GPUs (each has only 97,887 MiB). But TP=8 maximizes allreduce overhead. The benchmark implicitly accepts this trade-off.

Assumption 4: The benchmark tool itself is correct. The benchmark script measures wall-clock time and divides by the number of tokens generated. If there are any timing inaccuracies or if the script miscounts tokens, the reported throughput could be off. The subsequent manual measurement (512 tokens in 6.39s = 80.1 tok/s) validated the benchmark's accuracy to within ~1.6%.

The Knowledge Flow

Input knowledge required to understand this message:

The hardware configuration: 8x NVIDIA RTX PRO 6000 Blackwell GPUs (96GB each), connected via PCIe.
The model: Kimi-K2.5 INT4, a 1T-parameter MoE model with MLA architecture, 547GB on disk.
The software stack: vLLM serving the model via OpenAI-compatible API, with TRITON_MLA backend for Blackwell SM120 support.
The deployment history: NVFP4 variant achieved ~61 tok/s, MiniMax-M2.5 FP8 achieved 84 tok/s single-stream.
The user's target: "single stream >40~50" tok/s. Output knowledge created by this message:
The INT4 variant achieves 81.4 tok/s single-stream, nearly doubling the user's target.
The model scales well at low concurrency (5.1x throughput at C=8).
The INT4 variant outperforms the NVFP4 variant by ~33% on the same hardware.
The PCIe allreduce bottleneck, while still present, is less severe for INT4 than for NVFP4.
The deployment is viable for production use without further tuning.

The Broader Significance

This message marks the turning point in a long session that had oscillated between different models, quantization formats, and deployment strategies. The NVFP4 Kimi-K2.5 had been a dead end at 61 tok/s. MiniMax-M2.5 had been a detour — impressive throughput but not the model the user wanted. The INT4 Kimi-K2.5 was the first configuration that delivered both the desired model and acceptable performance.

The 81.4 tok/s result validated several strategic decisions: the pivot from NVFP4 to INT4, the investment in a clean vLLM reinstall to remove stale GLM-5 patches, the choice of TP=8 despite its allreduce overhead, and the patience to wait 36 minutes for the model to load. It also confirmed that the hardware — 8 Blackwell GPUs connected via PCIe — was capable of serving a 1T-parameter model at useful speeds, even if the ultimate bottleneck remained the PCIe bus.

In the subsequent messages, the assistant would note: "Baseline: 81.4 tok/s single-stream — already faster than the NVFP4 version (61 tok/s). That's because INT4 has lower weight bandwidth requirements for the quantized MoE experts." This analysis, while concise, captures the essence of why this benchmark mattered: it proved that quantization format choice can matter as much as model architecture for inference performance.

The message also set the stage for the next phase of optimization. Having exceeded the user's target by nearly 2x, the assistant could have declared victory and deployed the service. Instead, it chose to push further with NCCL tuning — a testament to the relentless optimization mindset that characterizes high-performance ML engineering.