The Benchmark That Revealed the Bottleneck: Measuring NVFP4 Kimi-K2.5 on 8x Blackwell GPUs

In the sprawling narrative of deploying a 1-trillion-parameter Mixture-of-Experts model across eight NVIDIA RTX PRO 6000 Blackwell GPUs, few moments are as revealing as the first real benchmark. Message <msg id=2226> captures precisely that moment: after hours of debugging, reinstalling, patching, and waiting, the assistant finally runs a throughput benchmark against the freshly deployed NVFP4 Kimi-K2.5 model. The output is deceptively simple — a table of numbers — but it represents the culmination of an enormous effort and the first concrete data point in a hardware-boundary exploration that would reshape the entire deployment strategy.

The Road to This Moment

To understand why this message matters, one must appreciate the journey that preceded it. The assistant had spent the better part of a session wrestling with the NVFP4 Kimi-K2.5 deployment. A clean vLLM reinstall had been forced to remove stale GLM-5 debug patches that had accumulated from earlier experiments. The first service start crashed with a RuntimeError caused by a version mismatch between flashinfer-python (0.6.4) and flashinfer-cubin (0.6.3) — a subtle dependency inconsistency that required manual intervention to resolve. After fixing that, the service started successfully but took a grueling 13 minutes to become ready, spending most of that time in torch.compile and CUDAGraph warmup phases.

Once the server was finally responding, the assistant ran smoke tests to verify inference quality. The model correctly identified Paris as the capital of France and wrote a concise prime-number-checking Python function, complete with 1,619 characters of reasoning tokens. These tests confirmed that the model was functioning correctly — but they said nothing about performance. The real question remained: how fast could this 1T-parameter beast actually run on eight Blackwell GPUs connected via PCIe?

The Benchmark Execution

The assistant's first step was to adapt the existing benchmark script, which had been written for the earlier GLM-5 GGUF model. The original script used the /v1/completions endpoint with a 128-token limit — inappropriate for a reasoning model like Kimi-K2.5, which needs room for both reasoning and content tokens. The assistant updated the model path to /shared/kimi-k2.5-nvfp4, switched to the /v1/chat/completions endpoint, increased max_tokens, and adjusted the concurrency levels to be more realistic for a 1T model. The benchmark was then executed from the local machine, communicating with the remote server over the network.

The message itself shows the output:

[assistant] Now let me run the benchmark from the local machine (it talks to the container over the network): [bash] python3 /home/theuser/glm-kimi-sm120-rtx6000bw/benchmark.py Warming up...

>

Benchmark results: -------------------------------------------------------------------------------- C= 1 | 4/ 4 ok (0 err) | wall= 33.4s | 61.3 tok/s | lat avg= 8.36 p50= 8.36 p99= 8.36 C= 2 | 8/ 8 ok (0 err) | wall= 70.8s | 57.8 tok/s | lat avg= 17.69 p50= 18.62 p99= 21.02 C= 4 | 12/ 12 ok (0 err) | wall= 65.5s | 93.7 tok/s | lat avg= 21.85 p50= 21.47 p99= 22.88 C= 8 | 16/ 16 ok (0 err) | wall= 29.6s | 276.6 tok/s | lat avg= 14.80 p50= ...

The output is truncated in the message — the subsequent message <msg id=2227> reveals the full results, including C=16 at 543.8 tok/s and C=32 at an impressive 799.9 tok/s. But even this partial view tells a compelling story.

Interpreting the Numbers

The benchmark measures throughput in tokens per second (tok/s) across concurrency levels (C), where each concurrent request generates 512 tokens. Several features stand out.

At single-stream (C=1), the model delivers 61.3 tok/s with a latency of 8.36 seconds. For a 1T-parameter MoE model running on consumer-grade GPU hardware, this is respectable — but it's far from the 40–50 tok/s target that had been set earlier, and it hints at deeper limitations. The latency is remarkably consistent: the p50 and p99 are identical at 8.36 seconds, indicating that single-request inference is stable and predictable with no tail-latency issues.

The C=2 result is puzzling: throughput actually drops to 57.8 tok/s, and latency nearly doubles to 17.69 seconds. This is counterintuitive — normally adding concurrency should increase aggregate throughput even if individual requests slow down. The drop suggests that at low concurrency levels, the model's batching isn't efficient, possibly because the reasoning model's variable-length reasoning tokens create scheduling inefficiencies, or because the GPU isn't fully utilized with only two concurrent requests.

From C=4 onward, the scaling becomes more sensible. Throughput jumps to 93.7 tok/s at C=4, then to 276.6 tok/s at C=8. The latency actually decreases from C=4 to C=8 (from 21.85s to 14.80s), which is unusual and may indicate that the model's internal batching or CUDAGraph execution becomes more efficient once enough requests are queued. The subsequent message shows this trend continuing: C=16 reaches 543.8 tok/s, and C=32 peaks at 799.9 tok/s — nearly 800 tokens per second from a single 1T-parameter model on eight GPUs.

What the Results Reveal

These numbers are not just performance metrics; they are diagnostic signals about the hardware-software stack. The relatively modest single-stream throughput (61 tok/s) for a model running on eight GPUs with 96 GB of HBM2e each (97887 MiB per GPU, as shown in earlier checks) immediately suggests a communication bottleneck. The assistant's subsequent analysis in <msg id=2227> would identify this as the PCIe allreduce bottleneck inherent to the MLA (Multi-head Latent Attention) architecture, which requires synchronizing attention projections across all 8 GPUs for every single token.

The scaling behavior confirms this diagnosis. If the bottleneck were compute-bound (e.g., matrix multiplication), throughput would scale almost linearly with concurrency until GPU compute capacity is saturated. Instead, we see sub-linear scaling: C=1 gives 61 tok/s, C=8 gives 276 tok/s (4.5x improvement for 8x concurrency), and C=32 gives 800 tok/s (13x improvement for 32x concurrency). The diminishing returns are exactly what one would expect from a communication-bound system where each additional request adds more allreduce traffic.

The excellent latency consistency (tight p50/p99 spread) is a positive signal: it means the system is not experiencing memory contention, thermal throttling, or scheduling jitter. The GPUs are stable and the software stack is well-behaved. The bottleneck is purely architectural.

Assumptions and Limitations

The benchmark makes several assumptions worth examining. First, it uses only 4 requests per concurrency level, which is a small sample size for statistical significance. The tight p50/p99 spread mitigates this concern — the results are clearly stable — but a production benchmark would typically run dozens or hundreds of requests per level. Second, the benchmark runs from the local machine over the network, introducing network latency that could affect the results. However, since the benchmark measures end-to-end throughput including network round-trips, this actually reflects the real-world performance a client would experience. Third, the prompt is fixed — a single guide-writing task — which may not represent the diversity of real workloads. A reasoning model like Kimi-K2.5 produces variable-length reasoning tokens depending on the prompt, and the benchmark's fixed prompt may not capture the full range of performance characteristics.

The Broader Significance

This benchmark message is a turning point in the session. It provides the first quantitative evidence that the NVFP4 Kimi-K2.5, while functional and coherent, is fundamentally limited by the PCIe interconnect for MLA-based architectures. This insight directly motivates the pivot to MiniMax-M2.5 (a 230B FP8 GQA model that achieves nearly 4,000 tok/s) and ultimately to the native INT4 Kimi-K2.5 (which delivers 82 tok/s single-stream and 2,276 tok/s at high concurrency). Without this benchmark, the team might have continued tuning the NVFP4 variant, chasing marginal gains against a hardware ceiling. The numbers forced a strategic reassessment: the model architecture matters as much as the quantization format, and on PCIe-bound Blackwell hardware, GQA (Grouped Query Attention) decisively outperforms MLA.

In this light, message <msg id=2226> is far more than a routine benchmark output. It is the diagnostic that reshaped the entire deployment strategy — a moment of clarity where raw numbers revealed the invisible hand of hardware constraints.