The Plateau at 4,000 Tok/s: Discovering Hardware Limits Through Exhaustive Benchmarking

Message Overview

The subject message ([msg 2335]) is a benchmark result produced by the assistant in response to a simple user request: "try larger batch" ([msg 2333]). At first glance, it appears to be nothing more than a table of numbers — four rows of throughput measurements at increasing concurrency levels. But this message represents the culmination of an intense optimization journey spanning multiple model architectures, parallelism strategies, and hardware configurations. It is the moment where the assistant discovers that no amount of software tuning can overcome a fundamental hardware bottleneck, and that the system has reached its absolute performance ceiling.

The message reads:

[assistant] [bash] python3 /home/theuser/glm-kimi-sm120-rtx6000bw/benchmark.py
Warming up...

Benchmark results:
  --------------------------------------------------------------------------------
  C= 256 | 256/256 ok (0 err) | wall=  33.1s |  3962.8 tok/s | lat avg= 32.88 p50= 32.88 p99= 32.97
  C= 512 | 512/512 ok (0 err) | wall=  65.7s |  3988.1 tok/s | lat avg= 49.13 p50= 65.23 p99= 65.48
  C= 768 | 768/768 ok (0 err) | wall=  98.5s |  3991.2 tok/s | lat avg= 65.47 p50= 65.50 p99= 98.09
  C=1024 | 1024/1024 ok (0 err) | wall= 131.2s |  3997.0 tok/s | lat avg= 81.66 p50...

The output is truncated — the final line cuts off mid-sentence — but the pattern is unmistakable: throughput barely budges from ~3,960 to ~3,997 tok/s as concurrency quadruples from 256 to 1024. The system has hit a wall.

The Context: A Multi-Model Optimization Odyssey

To understand why this message matters, one must appreciate the journey that led to it. The assistant had spent the preceding hours deploying and benchmarking multiple 1-trillion-parameter models on an 8× Blackwell GPU system (NVIDIA RTX PRO 6000 Blackwell, each with 96 GB of VRAM). The path had been anything but straightforward.

Earlier in the session ([msg 2310]–[msg 2332]), the assistant had:

  1. Deployed NVFP4 Kimi-K2.5, achieving only ~61 tok/s single-stream due to a fundamental bottleneck: PCIe allreduce across 8 GPUs for the 61-layer MLA (Multi-head Latent Attention) architecture. The MLA design requires massive allreduce operations for every attention layer, and with 8 GPUs connected via PCIe rather than NVLink, the interconnect bandwidth became the dominant constraint.
  2. Pivoted to MiniMax-M2.5, a 230B-parameter FP8 model using GQA (Grouped Query Attention) instead of MLA. This model loaded in just 75 seconds and used only 4 GPUs with TP=4, achieving 84 tok/s single-stream and over 2,500 tok/s at high concurrency.
  3. Attempted TP=8 without EP, which failed due to FP8 block quantization alignment issues — the intermediate_size of 192 was not divisible by the block size of 128 when TP-sharded across 8 GPUs.
  4. Discovered that TP=8 with EP (Expert Parallelism) works, because with EP enabled, the MoE layers use EP=8 (experts distributed across GPUs) and TP=1 for MoE (no tensor sharding of expert weights), bypassing the alignment check entirely.
  5. Benchmarked TP=8+EP against TP=4, producing a detailed comparison table ([msg 2332]) showing that TP=4 was faster at low concurrency (84 vs 71 tok/s at C=1) but TP=8+EP dominated at high concurrency, reaching 3,982 tok/s at C=256 — a 54% improvement over TP=4's 2,586 tok/s. It was at this point that the user issued the laconic instruction: "try larger batch" ([msg 2333]). The assistant edited the benchmark script ([msg 2334]) to extend the concurrency range upward, and then ran the test that produced the subject message.

What the Data Reveals: A Perfect Plateau

The benchmark results in [msg 2335] are remarkable not for their peak value but for their consistency. At C=256, the system delivers 3,962.8 tok/s. At C=512, it delivers 3,988.1 tok/s — a mere 0.6% increase despite doubling the number of concurrent requests. At C=768, it reaches 3,991.2 tok/s, and at C=1024, it approaches 3,997.0 tok/s. The throughput is asymptotically approaching ~4,000 tok/s, and no amount of additional concurrency can push it higher.

This is the unmistakable signature of a hardware-limited system. The throughput plateau means that some resource — likely PCIe bandwidth for allreduce operations, or memory bandwidth for weight reads — has been fully saturated. Adding more requests only increases latency (from 32.88 seconds at C=256 to 81.66 seconds at C=1024) without improving throughput. The system is operating at its absolute maximum capacity.

The latency numbers tell their own story. At C=256, the p50 latency is 32.88 seconds — already quite high for an interactive service. At C=1024, the p50 latency balloons to 81.66 seconds, with a p99 of... well, the output is truncated, but the trend is clear. The system is trading latency for throughput, and it has reached the point where the trade-off yields no further benefit.

The Assumptions and Decisions Embedded in This Message

This message, though it contains no explicit reasoning, embodies several implicit assumptions and decisions:

The user's assumption in saying "try larger batch" was that increasing concurrency would continue to yield throughput gains. This is a reasonable assumption — earlier benchmarks showed throughput scaling strongly from C=1 through C=256. The user likely expected to see throughput continue climbing, perhaps to 5,000 or 6,000 tok/s. The message proves this assumption wrong.

The assistant's decision to edit the benchmark script to extend the concurrency range was a straightforward response, but it reflected a deeper understanding: that the system's behavior at extreme concurrency was unknown and worth exploring. The assistant did not object or suggest diminishing returns — it simply ran the experiment.

The implicit model behind both the user's request and the assistant's compliance is that throughput scales with concurrency until some resource is exhausted. This message provides the empirical data point that defines where that exhaustion occurs.

Input Knowledge Required to Interpret This Message

A reader needs substantial context to understand the significance of these numbers:

  1. The hardware configuration: 8× NVIDIA RTX PRO 6000 Blackwell GPUs, each with 96 GB VRAM, connected via PCIe (not NVLink). This is critical because PCIe bandwidth is the primary bottleneck for allreduce-heavy architectures like MLA, and even for GQA models with expert parallelism.
  2. The model architecture: MiniMax-M2.5 is a 230B-parameter FP8 model using GQA with 256 experts in its MoE layers. The GQA design reduces allreduce overhead compared to MLA, which is why it can scale to higher throughputs.
  3. The parallelism strategy: TP=8 with EP enabled. This means tensor parallelism across all 8 GPUs for attention layers, and expert parallelism across all 8 GPUs for MoE layers. The EP configuration avoids the FP8 alignment issue that plagued the TP=8-without-EP attempt.
  4. The previous benchmark results: The comparison table from [msg 2332] showing TP=4 vs TP=8+EP across concurrency levels from 1 to 256. Those results established the trajectory that the user expected to continue.
  5. The PCIe bottleneck analysis: Earlier in the session, the assistant identified that the NVFP4 Kimi-K2.5 model was bottlenecked by PCIe allreduce for its 61 MLA layers. This same bottleneck, though less severe for GQA, ultimately constrains the MiniMax model as well.## The Thinking Process: What the Assistant Knew Although the subject message contains only the raw benchmark output, the assistant's thinking can be reconstructed from the surrounding context. In the preceding message ([msg 2332]), the assistant had just presented a detailed comparison of TP=4 vs TP=8+EP, showing throughput scaling from 71 tok/s at C=1 to 3,982 tok/s at C=256. The assistant's analysis identified a "crossover point" around C=12-16 where TP=8+EP overtakes TP=4, and noted that the choice between configurations depends on the use case — single-user coding agents benefit from TP=4's lower latency, while multi-user API serving benefits from TP=8+EP's higher aggregate throughput. When the user said "try larger batch," the assistant understood this as a request to explore the upper bounds of the throughput curve. The assistant edited the benchmark script to extend the concurrency range beyond 256, likely adding C=512, C=768, and C=1024 as new test points. The resulting output — the subject message — shows that the system has reached a throughput plateau at approximately 4,000 tok/s. The assistant did not need to add commentary to this message because the data speaks for itself. The plateau is so flat (3,962 → 3,988 → 3,991 → 3,997 tok/s across a 4× concurrency increase) that no interpretation is necessary. The system is saturated.

What This Message Creates: Output Knowledge

This message produces several forms of knowledge:

  1. Empirical proof of the hardware limit: The 4,000 tok/s ceiling is now a measured fact, not a speculation. Any future optimization attempts must be evaluated against this baseline.
  2. Validation of the PCIe bottleneck hypothesis: Earlier analysis suggested PCIe allreduce was the primary bottleneck for MLA models. This benchmark confirms that even the more efficient GQA architecture, with expert parallelism distributing the MoE computation, ultimately hits the same PCIe wall — just at a much higher throughput level.
  3. A decision point for deployment: With the knowledge that 4,000 tok/s is the maximum achievable throughput, the team can now make informed decisions about which model to deploy as a production service. The MiniMax-M2.5 with TP=8+EP offers the highest throughput of any configuration tested, but the INT4 Kimi-K2.5 (benchmarked later in the session at 2,276 tok/s) may be preferable for applications that require its specific capabilities.
  4. A reference point for future hardware comparisons: If the system were upgraded with NVLink-connected GPUs or faster interconnects, this 4,000 tok/s benchmark would serve as a baseline for measuring the improvement.

Mistakes, Incorrect Assumptions, and Subtleties

The truncated output is a limitation of the message format. The final line cuts off at "lat avg= 81.66 p50..." — we never see the p99 latency for C=1024. This is a minor technical artifact of how the terminal output was captured, but it means the full latency distribution at the highest concurrency level is unknown.

The assumption that "larger batch" would yield higher throughput was incorrect. The user's request implicitly assumed that the throughput curve had not yet plateaued. The data proved otherwise. This is not a mistake in the traditional sense — it was a reasonable hypothesis that needed to be tested — but it is an assumption that was falsified by the experiment.

The benchmark measures throughput but not quality. At 4,000 tok/s with 1,024 concurrent requests, the per-request latency exceeds 80 seconds. This is unacceptable for interactive use cases. The benchmark does not capture whether the model's output quality degrades under such extreme batching conditions (e.g., due to KV cache pressure or numerical precision effects). The throughput numbers are impressive, but they come with a latency cost that may make them irrelevant for real-world deployment.

The plateau may not be purely a hardware limit. While PCIe bandwidth is the most likely bottleneck, there could be software factors at play: the vLLM scheduler's ability to manage 1,024 concurrent sequences, the NCCL allreduce implementation's scaling efficiency, or the Triton kernel launch overhead. Distinguishing hardware limits from software limits would require profiling tools (e.g., nsys, ncu) that were not applied in this message.

The Broader Significance

This message is a case study in the scientific method applied to systems engineering. The user formed a hypothesis ("increasing batch size will increase throughput"), the assistant designed an experiment (extend the benchmark concurrency range), and the data provided a clear answer (throughput plateaus at ~4,000 tok/s). The result is negative — it disproves the hypothesis — but it is immensely valuable. The team now knows the absolute performance ceiling of their hardware for this model and parallelism configuration.

The plateau at 4,000 tok/s also tells us something profound about the economics of large-scale ML inference. With 8× Blackwell GPUs (each costing thousands of dollars) and a 230B-parameter model, the system can serve approximately 4,000 tokens per second. For a typical coding agent generating 500 tokens per response, that's 8 concurrent users at 1-second latency — or 80 users at 10-second latency. The hardware investment required to serve large models at scale remains substantial, even with aggressive optimization.

In the end, this message is a testament to the value of exhaustive benchmarking. It would have been easy to stop at C=256 and declare victory at 3,962 tok/s. Pushing to C=1024 revealed that no further gains were possible, saving future effort that might have been spent chasing imaginary optimizations. The plateau is not a failure — it is knowledge.