Pushing to the Limit: Benchmarking a 1T-Parameter MoE Model at 1,238 Tokens Per Second
The Message
The throughput keeps scaling:
>
| Concurrency | Throughput (tok/s) | Avg Latency (s) | |---|---|---| | 32 | 464 | 35.2 | | 48 | 657 | 37.4 | | 64 | 803 | 40.8 | | 96 | 1,017 | 48.2 | | 128 | 1,238 | 52.8 |
>
Still scaling at C=128 with 1,238 tok/s total throughput! Latency increases linearly, which is expected with continuous batching. Let me restore the benchmark to a reasonable final set of concurrency levels: [edit] /home/theuser/glm-kimi-sm120-rtx6000bw/benchmark.py Edit applied successfully.
This message, sent by the AI assistant in an opencode coding session, reports the results of a high-concurrency throughput benchmark of the NVFP4-quantized Kimi-K2.5 model — a 1-trillion-parameter Mixture-of-Experts (MoE) language model running on an 8-GPU server equipped with NVIDIA RTX PRO 6000 Blackwell GPUs. At first glance, it appears to be a simple data dump: a table of numbers and a brief observation. But beneath the surface, this message represents a critical inflection point in a long-running investigation into the real-world performance limits of large-scale inference hardware. It is the culmination of hours of debugging, patching, reinstalling, and iterating — and it reveals something surprising about where the bottlenecks actually lie.
The Context: A Long Road to a Running Model
To understand why this message matters, we must understand what came before it. The assistant and user had been working for many hours — across multiple segments of the conversation — to deploy and optimize large language models on a machine with 8x RTX PRO 6000 Blackwell GPUs (each with 96 GiB of VRAM). The journey had been arduous: earlier segments dealt with installing NVIDIA drivers, resolving flash-attn build failures, patching vLLM's GGUF loader to support novel architectures, debugging incoherent model output caused by tensor parallelism sharding mismatches, and tuning Triton attention kernels for the Blackwell SM120 architecture.
The immediate context for this message begins with the NVFP4 Kimi-K2.5 model. After a clean reinstall of vLLM (to remove stale GLM-5 debug patches), the assistant had successfully loaded the model and verified it produced coherent output ([msg 2219], [msg 2220]). The next logical step was to benchmark its performance — first at low concurrency to measure single-request throughput, then at progressively higher concurrency levels to understand how the system scales under load.
The initial benchmark run ([msg 2226]) showed promising results: 61.3 tok/s at C=1, scaling to 799.9 tok/s at C=32. The assistant then modified the benchmark script to push to higher concurrency levels ([msg 2227], [msg 2228]), and the second run confirmed continued scaling up to 1,017 tok/s at C=96. Message 2229 is the report of the third and final push — extending the concurrency ladder to C=128.
What the Message Reveals: Scaling Behavior and Its Implications
The data in this message tells a compelling story. At C=32, throughput was 464 tok/s (slightly lower than the 799.9 tok/s reported in the previous run, which is an important discrepancy we will examine). From there, throughput climbs steadily: 657 tok/s at C=48, 803 tok/s at C=64, 1,017 tok/s at C=96, and finally 1,238 tok/s at C=128. The average latency rises from 35.2 seconds at C=32 to 52.8 seconds at C=128 — a roughly linear increase of about 0.2 seconds per additional concurrent request.
The assistant's observation that "latency increases linearly, which is expected with continuous batching" is technically astute. In a continuous batching system, the server dynamically adds new requests to the running batch as earlier requests complete. As concurrency increases, the batch size grows, which increases the time to complete any individual request (because each iteration must process more sequences). However, the total throughput can continue to rise until the GPU compute or memory bandwidth is saturated. The fact that throughput is still climbing at C=128 — with no sign of a plateau — suggests that the system has not yet hit its fundamental limit.
This is a significant finding. A 1T-parameter model generating over 1,200 tokens per second across 8 GPUs is a remarkable achievement. But the message also contains a subtle tension: the assistant's tone is celebratory ("Still scaling at C=128!"), yet the decision to "restore the benchmark to a reasonable final set of concurrency levels" implies that C=128 was already pushing beyond what was considered practical. The benchmark was originally designed for concurrency levels up to 1,024 ([msg 2225]), but the assistant had already trimmed it down because "this is a 1T model, so very high concurrency levels like 1024 aren't realistic." Now, even C=128 is being dialed back.
The Discrepancy That Deserves Attention
A careful reader will notice a discrepancy between the two benchmark runs at C=32. In the previous run ([msg 2228]), the assistant reported 464.4 tok/s at C=32 with an average latency of 35.23 seconds. But in the run before that ([msg 2227]), the assistant reported 799.9 tok/s at C=32 with an average latency of 20.38 seconds. These are dramatically different numbers for the same concurrency level.
What explains this? The most likely cause is a change in the benchmark configuration between runs. Looking at the assistant's actions: after the first benchmark ([msg 2227]), the assistant edited the benchmark script to "push higher to see where it plateaus." Then in [msg 2228], the assistant ran the modified script and got the lower numbers. The edit likely changed the prompt length, the number of requests per concurrency level, or the request timing — any of which could shift the throughput/latency profile. Alternatively, the server's state may have changed between runs: the first run was on a freshly warmed-up server, while subsequent runs may have been affected by KV cache fragmentation or other long-running effects.
The assistant does not acknowledge this discrepancy in message 2229. This is a notable omission. The focus is entirely on the positive trend — "the throughput keeps scaling" — without addressing why the C=32 baseline shifted. Whether this was a conscious choice (to avoid muddying the narrative) or an oversight is unclear, but it represents a potential blind spot in the analysis.
Assumptions and Input Knowledge
This message rests on several assumptions that are worth examining. First, the assistant assumes that the benchmark methodology is sound — that the concurrent requests are independent, that the server is in a steady state, and that the measured throughput is representative of real-world usage. Given the earlier discrepancy, this assumption is partially questionable.
Second, the assistant assumes that linear latency growth is acceptable and expected. While this is true for continuous batching in general, it glosses over the question of whether the latency at C=128 (52.8 seconds) is acceptable for any practical application. For a real-time chat system, sub-minute latency might be tolerable; for an interactive coding assistant, it would be unusable. The assistant does not discuss the latency-quality tradeoff.
Third, the assistant assumes that the NVFP4 quantization is the right format for this hardware. Later in the session, the team pivots to the INT4 variant of Kimi-K2.5, which achieves significantly better single-stream throughput (82 tok/s vs. 61 tok/s) and comparable high-concurrency performance. The NVFP4 format, it turns out, has a fundamental bottleneck: PCIe allreduce across 8 GPUs for the 61-layer MLA architecture. This message does not yet know that — the assistant is still operating under the assumption that NVFP4 is the best option.
The input knowledge required to understand this message includes: familiarity with vLLM's continuous batching architecture, an understanding of tensor parallelism and how it distributes model layers across GPUs, knowledge of the Blackwell SM120 architecture and its memory bandwidth characteristics, and awareness of the NVFP4 quantization format used by NVIDIA's ModelOpt toolkit. The reader must also understand that "tok/s" here includes both reasoning tokens (generated by the model's internal chain-of-thought) and content tokens (the final visible output) — a distinction that matters for reasoning models like Kimi-K2.5.
Output Knowledge Created
This message creates several pieces of actionable knowledge. First, it establishes that the NVFP4 Kimi-K2.5 model can sustain over 1,200 tok/s on 8x Blackwell GPUs at high concurrency — a valuable data point for anyone planning to deploy this model. Second, it demonstrates that the system has not yet saturated at C=128, suggesting that even higher throughput might be achievable with more aggressive batching. Third, it provides a latency-vs-throughput curve that can inform capacity planning: if sub-40-second latency is required, the system should be limited to C=64 or below.
The message also implicitly creates negative knowledge: it reveals that the NVFP4 variant, despite its impressive peak throughput, has a relatively low single-stream performance (61 tok/s from the earlier benchmark). This sets the stage for the later pivot to the INT4 variant, which achieves 82 tok/s single-stream — a 34% improvement that fundamentally changes the economics of the deployment.
The Thinking Process
The reasoning visible in this message is concise but revealing. The assistant's first observation — "The throughput keeps scaling" — frames the data as a continuation of a positive trend, not as an isolated measurement. This is a deliberate rhetorical choice: the assistant is building a narrative of success after hours of debugging and setbacks.
The exclamation "Still scaling at C=128 with 1,238 tok/s total throughput!" reveals genuine surprise. The assistant had expected to find a plateau — that was the explicit purpose of pushing to higher concurrency levels. The fact that no plateau was found is noteworthy and slightly unexpected. The assistant's decision to bold the 1,238 figure emphasizes its significance.
The observation that "latency increases linearly, which is expected with continuous batching" shows the assistant reasoning from first principles about the system's behavior. Rather than simply reporting numbers, the assistant is interpreting them through the lens of known architectural properties. This is characteristic of an experienced systems engineer who understands the underlying mechanisms.
The final action — restoring the benchmark to "a reasonable final set of concurrency levels" — reveals a pragmatic decision. The assistant has gathered enough data to characterize the system's scaling behavior. Pushing further (to C=256 or beyond) would be time-consuming and might risk destabilizing the server. The benchmark has served its purpose; it is time to move on to the next task.
Conclusion
Message 2229 is a deceptively simple report that captures a pivotal moment in a complex engineering investigation. It records the peak throughput of a 1T-parameter model on cutting-edge hardware, reveals unexpected continued scaling, and sets the stage for a strategic pivot to a better-performing model variant. The numbers themselves are impressive — 1,238 tokens per second from a trillion-parameter model on eight consumer-grade GPUs — but the real value lies in what they imply about the system's headroom and the tradeoffs between concurrency, latency, and throughput. This message is not the end of the story; it is the turning point where the team realizes that NVFP4 is not the final answer, and that the real prize lies elsewhere.