Benchmarking the NVFP4 Kimi-K2.5: A Pivotal Performance Assessment on 8x Blackwell GPUs

Introduction

In the sprawling narrative of deploying and optimizing large language models on high-end GPU hardware, certain messages serve as inflection points — moments where raw data crystallizes into actionable insight. Message [msg 2227] is precisely such a moment. In this message, the assistant presents the first comprehensive benchmark results for the NVFP4 Kimi-K2.5 model running on an 8× RTX PRO 6000 Blackwell GPU system, then immediately acts on those results by pushing the benchmark to higher concurrency levels. This single message encapsulates the transition from "does it work?" to "how well does it work?" — a transition that would ultimately drive the session toward hardware-aware model selection and the discovery of fundamental system bottlenecks.

The Message in Full

The assistant writes:

Excellent benchmark results! Here's a summary:

>

| Concurrency | Throughput (tok/s) | Avg Latency (s) | P50 (s) | P99 (s) | |---|---|---|---|---| | 1 | 61.3 | 8.36 | 8.36 | 8.36 | | 2 | 57.8 | 17.69 | 18.62 | 21.02 | | 4 | 93.7 | 21.85 | 21.47 | 22.88 | | 8 | 276.6 | 14.80 | 17.40 | 17.40 | | 16 | 543.8 | 15.05 | 15.06 | 15.09 | | 32 | 799.9 | 20.38 | 20.43 | 20.46 |

>

These throughput numbers include both reasoning and content tokens. The system scales well — nearly 800 tok/s at C=32 with sub-21s latency. Let me push higher to see where it plateaus:

The message concludes with an edit to the benchmark script, adding concurrency levels C=48, C=64, C=96, and C=128 to probe the upper limits of the system.

Context and Motivation: Why This Message Was Written

To understand why this message exists, we must trace the events that led to it. The preceding messages ([msg 2198] through [msg 2226]) document a grueling debugging session. The NVFP4 Kimi-K2.5 service had crashed repeatedly due to a flashinfer-cubin version mismatch — the Python package was at version 0.6.4 while the compiled CUDA binaries were still at 0.6.3, causing a RuntimeError on startup. The assistant diagnosed this, fixed the version mismatch by reinstalling flashinfer-cubin==0.6.4, and successfully restarted the service. After a 13-minute startup (including model weight loading and torch.compile warmup), the server came online and passed smoke tests with correct reasoning and code generation output.

The benchmark was the natural next step. The assistant had just completed a clean vLLM reinstall (removing stale GLM-5 debug patches), re-applied SM120-necessary configuration changes, and verified the model produced coherent output. Now it needed to quantify performance: what throughput could this 1T-parameter MoE model actually deliver on eight Blackwell GPUs? This message represents the first hard data point after an extensive cleanup and debugging effort.

The motivation is twofold. First, the assistant needs to establish a performance baseline for the NVFP4 Kimi-K2.5 to determine whether it meets the deployment target (the user had previously mentioned a goal of 40–50 tok/s single-stream). Second, the assistant is probing the system's scaling characteristics — understanding how throughput and latency behave under increasing concurrency is essential for production deployment decisions, such as setting appropriate rate limits and batch sizes.

The Thinking Process: What the Assistant's Reasoning Reveals

The assistant's thinking is visible in several dimensions within this message. First, there is the careful framing of the results. The assistant notes that "These throughput numbers include both reasoning and content tokens" — a crucial clarification for a reasoning model like Kimi-K2.5, which generates internal reasoning tokens before producing visible output. In many benchmarks, reasoning tokens are either excluded or not tracked, leading to inflated throughput numbers. By explicitly stating that both are included, the assistant signals awareness of this measurement nuance.

Second, the assistant's interpretation of the results reveals analytical depth. The C=2 result (57.8 tok/s) is actually lower than C=1 (61.3 tok/s), which might seem like a regression. But the assistant does not flag this as a problem — likely recognizing that at very low concurrency, the overhead of request scheduling and context switching can outweigh the benefits of batching. The scaling from C=4 onward is nearly linear: C=4 delivers 93.7 tok/s (1.5× C=1), C=8 delivers 276.6 tok/s (4.5× C=1), and C=16 delivers 543.8 tok/s (8.9× C=1). This near-linear scaling through C=16 is impressive and suggests that vLLM's continuous batching is working effectively.

The assistant's decision to "push higher to see where it plateaus" is the key action in this message. This reveals an assumption that the system has not yet reached its saturation point. The assistant is operating under the hypothesis that throughput will continue to scale with concurrency until some resource — GPU compute, memory bandwidth, PCIe interconnect, or NCCL allreduce — becomes the bottleneck. Finding that plateau is critical for understanding the system's fundamental limits.

Assumptions Embedded in This Message

Several assumptions underpin the assistant's analysis and actions. The most significant is the assumption that the NVFP4 Kimi-K2.5 is a viable candidate for production deployment. At this point in the session, the assistant has not yet identified the PCIe allreduce bottleneck that would later drive a pivot to MiniMax-M2.5 and ultimately to the INT4 variant of Kimi-K2.5. The 61.3 tok/s single-stream result is encouraging — it exceeds the 40–50 tok/s target — but the assistant does not yet know that this model's MLA (Multi-head Latent Attention) architecture would prove fundamentally limited on PCIe-bound Blackwell hardware.

Another assumption is that the benchmark methodology is sound. The assistant modified the benchmark script from the GLM-5 era to work with Kimi-K2.5, switching from the completions endpoint to the chat completions endpoint and increasing max_tokens from 128 to 1024. The 128-token limit was appropriate for GLM-5 but would have been severely limiting for a reasoning model like Kimi-K2.5, which generates substantial reasoning tokens before producing visible output. The assistant correctly recognized this and adjusted accordingly.

The assistant also assumes that the benchmark is representative of real-world usage patterns. The prompt used is a "write a detailed step-by-step guide" prompt, which generates substantial output (the benchmark measures completion tokens, not just prompt processing). This is a reasonable choice for a text generation workload, but it may not capture the performance characteristics of other use cases like chat, code generation, or retrieval-augmented generation.

Input Knowledge Required

To fully understand this message, the reader needs substantial context. One must know that the NVFP4 Kimi-K2.5 is a 1-trillion-parameter Mixture-of-Experts model using NVIDIA's NVFP4 quantization format, deployed on 8× RTX PRO 6000 Blackwell GPUs (each with 96 GiB of VRAM). One must understand that "tok/s" refers to tokens per second — the standard throughput metric for LLM inference — and that these numbers include both reasoning and content tokens, which is significant because reasoning models like Kimi-K2.5 generate internal chain-of-thought tokens that consume generation budget without appearing in the visible output.

The reader must also understand the concept of concurrency (C) in benchmarking: how many simultaneous requests are being processed. The benchmark uses a fixed concurrency level, sending C requests at once and measuring how long all C take to complete, then computing throughput as (total_tokens_generated / wall_time). The latency metrics (avg, P50, P99) represent how long individual requests wait for completion.

Additionally, the reader needs to know about the preceding debugging saga: the flashinfer version mismatch, the clean vLLM reinstall, and the smoke tests that confirmed the model was working correctly before benchmarking began. Without this context, the benchmark results might seem to appear from nowhere.

Output Knowledge Created

This message creates several valuable pieces of knowledge. First and foremost, it establishes the first quantitative performance baseline for NVFP4 Kimi-K2.5 on Blackwell GPUs. The 61.3 tok/s single-stream result is a critical data point — it tells the team that the model meets the minimum throughput target and can handle production workloads at moderate concurrency.

Second, the message reveals the scaling characteristics of the system. The near-linear scaling from C=1 through C=32 (61 → 800 tok/s) demonstrates that vLLM's continuous batching is highly effective on this hardware, and that the system has significant headroom before hitting resource limits. The sub-21-second latency at C=32 is also notable — it means that even under moderate load, individual requests complete in a reasonable timeframe.

Third, the message creates a decision point. The assistant explicitly states "Let me push higher to see where it plateaus," which signals that the current results are preliminary and that further investigation is needed. This sets the stage for the subsequent benchmarks at C=48 through C=128 ([msg 2228]), which would reveal continued scaling to 1,238 tok/s — and ultimately lead to the discovery that NVFP4 Kimi-K2.5's MLA architecture is bottlenecked by PCIe allreduce bandwidth, prompting the pivot to MiniMax-M2.5 and then to the INT4 variant.

Mistakes and Incorrect Assumptions

While the message is analytically sound, some limitations are worth noting. The assistant does not yet recognize that the NVFP4 Kimi-K2.5's MLA architecture is fundamentally suboptimal for this hardware configuration. The 61-layer MLA model requires massive allreduce operations across all 8 GPUs for every attention computation, and on PCIe-connected GPUs (as opposed to NVSwitch-connected), this creates a bandwidth bottleneck that limits throughput. The assistant would discover this shortly after this message, when comparing NVFP4 Kimi-K2.5 results to MiniMax-M2.5 (a 230B GQA model that achieved nearly 4,000 tok/s).

Additionally, the benchmark measures throughput in terms of tokens per second but does not separately report prompt processing time (prefill) versus generation time (decode). For reasoning models, the prefill phase can be significant, and understanding the breakdown would provide deeper insight into where optimization efforts should focus.

The C=2 result (57.8 tok/s) being lower than C=1 (61.3 tok/s) is also worth interrogating. This could indicate that the overhead of scheduling two concurrent requests exceeds the batching benefit at very low concurrency, or it could be a measurement artifact. The assistant does not comment on this anomaly, which is a missed opportunity for deeper analysis.

The Broader Significance

Message [msg 2227] is significant not just for what it says, but for what it sets in motion. The benchmark results are encouraging enough to justify further investigation, but they also contain the seeds of the pivot that would define the remainder of the session. The 61 tok/s single-stream throughput, while meeting the target, is modest compared to what the hardware is capable of — as the team would soon discover when MiniMax-M2.5 delivered 84 tok/s single-stream with only 4 GPUs and nearly 4,000 tok/s with Expert Parallelism.

In this sense, the message is a classic example of "good enough to deploy, but not good enough to stop optimizing." The assistant's decision to push higher concurrency levels reflects a mindset of continuous improvement — never accepting the first result as final, always probing for the system's true limits. This mindset would serve the session well, leading through multiple model pivots and ultimately to the deployment of the INT4 Kimi-K2.5 as a production service with 82 tok/s single-stream and 2,276 tok/s at high concurrency.

Conclusion

Message [msg 2227] is a benchmark report that doubles as a decision point. It presents the first hard performance data for NVFP4 Kimi-K2.5 on 8× Blackwell GPUs after an extensive cleanup and debugging effort, showing 61 tok/s single-stream and scaling to 800 tok/s at C=32. The assistant's analytical framing — noting the inclusion of reasoning tokens, interpreting the scaling behavior, and pushing for higher concurrency — reveals a methodical approach to performance characterization. While the message does not yet identify the fundamental PCIe allreduce bottleneck that would later drive a model pivot, it provides the empirical foundation upon which that discovery would be built. In the arc of the session, this message is the calm before the storm — the moment when the system is working, the numbers are good, and the team is about to discover just how much better things could be.