Hitting the Memory Bandwidth Wall: Benchmarking MiniMax-M2.5 at 4,000 tok/s on 8x Blackwell GPUs

Introduction

In the high-stakes world of large language model deployment, few moments are as satisfying as watching a system hit its fundamental hardware limit — cleanly, predictably, and without errors. Message [msg 2336] captures exactly such a moment. After a long journey through model pivots, PCIe bottleneck analysis, FP8 alignment issues, and multiple configuration experiments, the assistant reports the final benchmark results for the MiniMax-M2.5 FP8 model running on 8x NVIDIA RTX PRO 6000 Blackwell GPUs with tensor parallelism 8 and expert parallelism enabled (TP=8+EP). The results are striking: throughput plateaus at approximately 4,000 tokens per second from concurrency level 256 all the way to 1,024, with zero errors and perfectly linear latency scaling.

This message is deceptively short — just a table and a few sentences of analysis — but it represents the culmination of dozens of prior messages, hours of debugging, and a systematic exploration of the hardware-software boundary. To fully appreciate what is being communicated here, one must understand the context that led to this benchmark, the technical decisions that made it possible, and the physical constraints it reveals.

The Journey to This Point

The path to this benchmark was anything but straightforward. The session began with an ambitious goal: deploying the GLM-5-NVFP4 model, a 1-trillion-parameter MoE architecture, on a system with 8x Blackwell GPUs. This required resolving flash-attention build issues, patching vLLM for custom architectures, debugging incoherent model output caused by tensor parallelism sharding mismatches, and implementing a custom Triton MLA sparse attention backend for the Blackwell SM120 architecture.

After achieving ~57 tok/s with GLM-5, the team pivoted to the nvidia/Kimi-K2.5 NVFP4 model, which delivered ~60 tok/s single-stream but revealed a fundamental bottleneck: PCIe allreduce across 8 GPUs for the 61-layer MLA architecture. The MLA (Multi-head Latent Attention) architecture requires all-to-all communication on every decode step, and with 8 GPUs connected via PCIe rather than NVLink, this communication overhead dominated.

This led to a strategic pivot. The team abandoned the NVFP4 Kimi-K2.5 and turned to MiniMax-M2.5, a 230B-parameter FP8 model using GQA (Grouped Query Attention) rather than MLA. GQA requires significantly less allreduce communication, making it far more suitable for PCIe-bound multi-GPU setups. The initial deployment used TP=4 on 4 GPUs, achieving 84 tok/s single-stream and up to 2,586 tok/s at high concurrency. A TP=8 attempt without EP failed due to FP8 block quantization alignment issues, but TP=8 with EP succeeded and dramatically improved throughput at high concurrency.

The comparison table in [msg 2332] showed that while TP=4 was faster at low concurrency (84 vs 71 tok/s at C=1), TP=8+EP dominated at high concurrency, reaching 3,982 tok/s at C=256 versus 2,586 tok/s for TP=4. The user's response was simple: "try larger batch" ([msg 2333]).

The Benchmark Request and Execution

The user's request to "try larger batch" was a natural next step. The comparison had shown TP=8+EP winning at C=256, but was this the ceiling? Could the system handle even more concurrent requests? Would throughput continue to scale, or would it plateau? Would the system crash under the pressure of 512, 768, or 1,024 simultaneous requests?

The assistant modified the benchmark script ([msg 2334]) and executed the benchmark ([msg 2335]). The results came back clean, and message [msg 2336] is the analysis of those results.

The Results: A System at Its Limit

The data in the message tells a clear story. Here is the exact table as presented:

| Concurrency | Throughput (tok/s) | Avg Latency | P99 Latency | |---|---|---|---| | 256 | 3,963 | 32.9s | 33.0s | | 512 | 3,988 | 49.1s | 65.5s | | 768 | 3,991 | 65.5s | 98.1s | | 1024 | 3,997 | 81.7s | 130.6s |

The most striking feature is the near-perfect flatness of throughput. From 3,963 tok/s at C=256 to 3,997 tok/s at C=1,024 — a variation of less than 1% across a 4x increase in concurrency. This is the textbook signature of a system that has reached its fundamental hardware bottleneck. The assistant identifies this as the "memory bandwidth limit," and the evidence strongly supports this diagnosis.

Latency, meanwhile, grows linearly with concurrency. Average latency goes from 32.9 seconds at C=256 to 81.7 seconds at C=1,024 — roughly a 3.2x increase for a 4x increase in concurrency, which is consistent with continuous batching where the marginal cost of adding one more request to an already-saturated batch is primarily the additional memory bandwidth needed to process its tokens.

The P99 latency tells an interesting story too. At C=256, P99 is essentially equal to average (33.0s vs 32.9s), indicating near-perfect uniformity. At higher concurrencies, the gap widens: at C=1,024, P99 is 130.6s versus 81.7s average. This 60% tail latency penalty is typical for systems operating at saturation, where request scheduling and memory contention create occasional stragglers.

What "Zero Errors" Really Means

The assistant emphasizes that there were "zero errors even at C=1024." This is not a trivial achievement. Running 1,024 concurrent inference requests on an 8-GPU system means managing 1,024 separate KV caches, 1,024 sets of attention computations, and 1,024 token generation loops — all competing for GPU memory, compute, and PCIe bandwidth. An OOM (out of memory) error at high concurrency is the most common failure mode, and the fact that the system handled 1,024 requests without a single failure demonstrates that the memory budgeting and batching logic in vLLM is working correctly.

This is particularly noteworthy given the earlier OOM failure at TP=2+EP ([msg 2311]), where the system ran out of memory because 230GB of model weights split across only 2 GPUs left insufficient room for KV cache. The TP=8+EP configuration distributes the weights across 8 GPUs (approximately 28.75GB per GPU), leaving roughly 68GB per GPU for KV cache — enough to handle very large batch sizes.

The Thinking Behind the Analysis

The assistant's reasoning in this message reveals a sophisticated understanding of system behavior under load. The key insight is that throughput flatness combined with linear latency growth is the expected signature of a saturated system. The assistant correctly identifies the bottleneck as memory bandwidth — the rate at which GPU memory can feed weights and KV cache data to the compute units.

This diagnosis is consistent with the earlier analysis of the NVFP4 Kimi-K2.5, where PCIe allreduce was identified as the bottleneck for MLA models. The MiniMax-M2.5 uses GQA, which requires much less allreduce communication, so the bottleneck shifts from PCIe bandwidth to GPU memory bandwidth — a more fundamental and less solvable constraint.

The assistant's language is measured but carries a note of satisfaction: "perfectly flat scaling with zero errors even at C=1024" and "The system handles 1024 concurrent requests cleanly." After hours of debugging OOM errors, FP8 alignment crashes, and incoherent model output, seeing the system run flawlessly at its hardware limit is a significant achievement.

Practical Implications

What does 4,000 tok/s mean in practice? For a coding agent or chatbot, this translates to the ability to serve hundreds of simultaneous users with sub-minute response times. At C=256, each request completes in about 33 seconds — acceptable for many interactive use cases. At C=1,024, the 82-second average latency is more suited to batch processing or applications where latency is less critical than throughput.

The flat throughput scaling also means that capacity planning is straightforward: the system delivers approximately 4,000 tok/s regardless of concurrency (above C=256). Adding more concurrent requests increases latency linearly but does not increase throughput. This is a classic "system at capacity" signal.

Conclusion

Message [msg 2336] is a masterclass in concise, data-driven system analysis. In just a few lines, the assistant communicates the results of a complex benchmark, identifies the fundamental hardware bottleneck, and validates the system's robustness under extreme load. The message stands as the culmination of a long optimization journey — from GLM-5 to Kimi-K2.5 to MiniMax-M2.5, from TP=4 to TP=8+EP, from OOM errors to clean saturation at 4,000 tok/s.

The most important lesson embedded in this message is about hardware-aware model selection. The pivot from MLA-based Kimi-K2.5 (which hit the PCIe allreduce wall at ~60 tok/s) to GQA-based MiniMax-M2.5 (which pushes through to 4,000 tok/s) was the single most impactful decision in the entire session. It demonstrates that for PCIe-bound multi-GPU deployments, attention architecture choice matters as much as — if not more than — raw model quality. The right model on the right hardware configuration can deliver orders of magnitude better throughput, and this message captures the moment that truth became visible in the benchmark data.