Pushing the Limits: Benchmarking Kimi-K2.5 NVFP4 at High Concurrency on 8× Blackwell GPUs

Introduction

In the sprawling, multi-day journey of deploying and benchmarking trillion-parameter language models on a cluster of 8× NVIDIA RTX PRO 6000 Blackwell GPUs, few moments crystallize the tension between ambition and hardware reality as neatly as message <msg id=2228>. At first glance, it is a deceptively simple artifact: a single bash command executing a Python benchmark script, followed by four lines of tabular output. But this message sits at a critical inflection point in the session — it represents the assistant's attempt to probe the upper limits of a freshly deployed NVFP4 Kimi-K2.5 model after a long chain of debugging, patching, and reinstallation.

The message itself reads:

[assistant] [bash] python3 /home/theuser/glm-kimi-sm120-rtx6000bw/benchmark.py Warming up...

Benchmark results: -------------------------------------------------------------------------------- C= 32 | 32/ 32 ok (0 err) | wall= 35.3s | 464.4 tok/s | lat avg= 35.23 p50= 35.25 p99= 35.26 C= 48 | 48/ 48 ok (0 err) | wall= 37.4s | 657.0 tok/s | lat avg= 37.36 p50= 37.37 p99= 37.39 C= 64 | 64/ 64 ok (0 err) | wall= 40.8s | 803.0 tok/s | lat avg= 40.75 p50= 40.76 p99= 40.77 C= 96 | 96/ 96 ok (0 err) | wall= 48.3s | 1017.0 tok/s | lat avg= 48.23 p50= ...

The output is truncated mid-line at C=96, a detail that itself tells a story about the assistant's workflow and the limits of the recording mechanism. But even in this partial form, the message is rich with meaning.

The Context: Why This Message Was Written

To understand why this message exists, one must trace back through the preceding hours of work. The assistant had just completed a clean reinstallation of vLLM nightly, removing all the GLM-5 debug patches that had accumulated over previous segments. The NVFP4 Kimi-K2.5 model — a 1-trillion-parameter Mixture-of-Experts model using NVIDIA's NVFP4 quantization format — had been loaded onto 8 GPUs, and a first round of benchmarks had been run at lower concurrency levels (C=1 through C=32) in message <msg id=2226>.

Those initial results were promising: the model achieved 61.3 tok/s single-stream and scaled to 799.9 tok/s at C=32, with excellent latency characteristics. But the assistant, displaying a methodical engineering mindset, was not satisfied. The comment at the end of <msg id=2227> — "Let me push higher to see where it plateaus" — reveals the core motivation. The assistant had edited the benchmark script to add higher concurrency levels (C=48, C=64, C=96, and presumably beyond) and was now executing that modified benchmark.

The message is thus an act of exploration. It is the assistant asking: How far can this system go? Where is the saturation point? This is not merely idle curiosity — understanding the throughput ceiling is essential for capacity planning, service-level objective setting, and determining whether the deployment meets the user's requirements.

The Results: What the Numbers Reveal

The benchmark output shows four data points, each representing a different concurrency level (C), which is the number of simultaneous requests being processed by the server:

| Concurrency | Requests | Duration | Throughput | Avg Latency | P50 Latency | P99 Latency | |-------------|----------|----------|------------|-------------|-------------|-------------| | 32 | 32/32 | 35.3s | 464.4 tok/s | 35.23s | 35.25s | 35.26s | | 48 | 48/48 | 37.4s | 657.0 tok/s | 37.36s | 37.37s | 37.39s | | 64 | 64/64 | 40.8s | 803.0 tok/s | 40.75s | 40.76s | 40.77s | | 96 | 96/96 | 48.3s | 1017.0 tok/s | 48.23s | — | — |

Several features stand out immediately. First, there are zero errors across all runs — every single request succeeded. This is a testament to the stability of the deployment after the extensive debugging that preceded it. Second, the latency distribution is remarkably tight: at every concurrency level, the average, median (P50), and 99th percentile (P99) latencies are nearly identical, differing by at most 0.03 seconds. This indicates a system with virtually no tail latency — no straggler requests, no queuing jitter, no memory contention causing sporadic slowdowns. The system is operating in a regime where every request experiences essentially the same service time.

Third, the throughput scales almost linearly with concurrency. Going from C=32 to C=48 (a 50% increase in concurrency) yields a 41% increase in throughput. From C=48 to C=64 (33% increase) yields a 22% increase. From C=64 to C=96 (50% increase) yields a 27% increase. The scaling is sublinear but not dramatically so — the system has not yet hit a hard ceiling. This is consistent with a system that is still within its capacity but showing early signs of diminishing returns.

The Discrepancy: Two Different C=32 Results

One of the most interesting features of this message is a discrepancy that requires cross-referencing with the earlier benchmark run. In <msg id=2226>, the C=32 result was 799.9 tok/s with an average latency of 20.38 seconds. In this message, the C=32 result is 464.4 tok/s with an average latency of 35.23 seconds. That is a 42% drop in throughput and a 73% increase in latency — a dramatic difference for the same concurrency level on the same model.

What explains this? The assistant does not comment on the discrepancy in the message itself, but several hypotheses can be formulated. The most likely explanation is that the benchmark was modified between the two runs. In <msg id=2224>, the assistant edited the benchmark script to switch from the /v1/completions endpoint to the /v1/chat/completions endpoint, and increased max_tokens from 128 to a higher value. For a reasoning model like Kimi-K2.5, the chat endpoint triggers a different inference path — the model generates reasoning tokens before producing its final answer. These reasoning tokens are internal to the model and consume the token budget, meaning that for a fixed max_tokens setting, the chat endpoint will generate fewer visible output tokens per request. The throughput measurement (tok/s) counts all generated tokens, including reasoning tokens, but the wall-clock time per request increases because the model must first produce reasoning before generating the response.

This is a crucial methodological insight: benchmarking a reasoning model requires different instrumentation than benchmarking a standard language model. The assistant's earlier run at C=32 used the completions endpoint (no reasoning), while this run uses the chat endpoint (with reasoning). The 42% throughput drop is not a regression — it is the cost of reasoning.

Another possible factor is that the two benchmark runs may have used different request payloads. The earlier run used a 128-token max_tokens setting on the completions endpoint, while the modified script likely increased this value. Longer generation sequences reduce the effective throughput because the model must process more tokens per request, and the KV cache grows correspondingly.

The Truncated Output: A Window into the Workflow

The message ends abruptly mid-line at the C=96 result: "lat avg= 48.23 p50= ..." The output is truncated because the conversation recording mechanism captured only the first portion of the benchmark's output. This truncation is not a bug — it is a feature of how the assistant's tool calls work. The bash tool captures stdout and stderr from the command execution, but if the output is very long, only the beginning may be retained.

This truncation is significant because it means the most interesting data point — the saturation point — is likely missing. The benchmark was designed to push until the system plateaus, and the C=96 result of 1017 tok/s shows continued scaling. The missing data would have included C=128, C=192, or even higher, revealing where throughput stops increasing. The assistant would need to run a separate benchmark or inspect the logs to find this information.

Assumptions and Their Validity

The message rests on several assumptions, some explicit and some implicit. The assistant assumes that the benchmark script has been correctly modified to target the Kimi-K2.5 model using the chat endpoint. This is a reasonable assumption given the edit operations in <msg id=2224> and <msg id=2225>, but it is not verified — the assistant does not re-read the modified file before executing it.

The assistant also assumes that the benchmark results are reproducible and representative of steady-state performance. The single-run nature of the benchmark means that cold-start effects, NCCL warmup, or CUDA graph compilation could influence the results. The "Warming up..." line at the start of the output suggests the script includes a warmup phase, which mitigates this concern.

A more subtle assumption is that the benchmark's concurrency model (sending a fixed number of requests simultaneously) accurately reflects real-world traffic patterns. In production, request arrivals are typically Poisson-distributed, not batched. The benchmark's synchronous concurrency model may overestimate or underestimate real-world throughput depending on the request arrival distribution.

The Thinking Process: What the Message Reveals About the Assistant's Reasoning

Although the message itself is a simple command execution, the surrounding context reveals the assistant's thinking. The progression from <msg id=2226> (low-concurrency benchmark) to <msg id=2227> (analysis and script modification) to this message (high-concurrency benchmark) shows a systematic, hypothesis-driven approach. The assistant observed that the system scaled well to C=32 without saturation, formulated the hypothesis that higher concurrency would continue to scale, and designed an experiment to test that hypothesis.

The choice of concurrency levels (32, 48, 64, 96) is itself revealing. These are not arbitrary numbers — they follow a pattern of increasing by roughly 50% each step, which is a common experimental design for finding the knee in a scaling curve. The assistant is looking for the point where throughput stops increasing linearly, which would indicate that the system has reached a resource bottleneck (compute, memory bandwidth, or PCIe allreduce).

The fact that the assistant does not comment on the discrepancy between the two C=32 results is also interesting. It may indicate that the assistant recognized the difference as expected (due to the endpoint change) and did not consider it noteworthy. Alternatively, it may indicate that the assistant was focused on the scaling trend rather than the absolute numbers.

Output Knowledge Created

This message creates several pieces of knowledge that were not available before:

Throughput scaling curve: The NVFP4 Kimi-K2.5 on 8× Blackwell GPUs achieves 1017 tok/s at C=96, with near-linear scaling from C=32 to C=96. This is a concrete performance data point for capacity planning.
Latency stability: The near-identity of P50 and P99 latencies across all concurrency levels demonstrates that the system has excellent tail latency characteristics, which is critical for production deployments where latency SLAs must be met.
Error-free operation: Zero errors across 240 requests confirms that the deployment is stable after the extensive debugging that preceded it.
Saturation point unknown: The truncation means we do not know where the system plateaus, creating a knowledge gap that would need to be filled by further benchmarking.

Conclusion

Message <msg id=2228> is a snapshot of a system under stress — a trillion-parameter model running on cutting-edge hardware, being pushed to its limits by a methodical engineer. The numbers are impressive: over 1000 tokens per second at C=96, with perfect reliability and vanishingly small tail latency. But the message is also a reminder that benchmarking is never complete: the truncated output, the unexplained discrepancy with the earlier run, and the unknown saturation point all point to further work that remains to be done. In the broader arc of the session, this message represents a moment of validation — the deployment works, and it works well — but also a moment of transition to the next question: What comes next?