The Benchmark That Validated a Pivot: MiniMax-M2.5 on 8× Blackwell GPUs

Introduction

In the course of a marathon session deploying and benchmarking trillion-parameter language models on eight NVIDIA RTX PRO 6000 Blackwell GPUs, a single message stands out as a quiet watershed moment. Message [msg 2290] is deceptively simple: it contains the output of a benchmark script running against the MiniMax-M2.5 model, showing throughput figures across concurrency levels 1 through 8. Yet this message represents far more than a table of numbers. It is the culmination of a deliberate strategic pivot, the validation of a hardware-aware model selection hypothesis, and the moment when the team confirmed they had found a dramatically better fit for their PCIe-bound Blackwell infrastructure than the model they had spent days optimizing.

The Message Itself

The assistant executed a benchmark script and received the following output:

Warming up...

Benchmark results:
  --------------------------------------------------------------------------------
  C=   1 |   4/  4 ok (0 err) | wall=  24.3s |    84.2 tok/s | lat avg=  6.08 p50=  6.08 p99=  6.08
  C=   2 |   8/  8 ok (0 err) | wall=  28.7s |   142.6 tok/s | lat avg=  7.18 p50=  7.19 p99=  7.19
  C=   4 |  12/ 12 ok (0 err) | wall=  24.3s |   253.1 tok/s | lat avg=  8.09 p50=  8.09 p99=  8.11
  C=   8 |  16/ 16 ok (0 err) | wall=  19.6s |   417.7 tok/s | lat avg=  9.80 p50= ...

The format is a multi-concurrency throughput benchmark. Each row corresponds to a concurrency level (C=1, 2, 4, 8), showing the number of completed requests, wall-clock time, achieved tokens per second, and latency statistics. All requests succeeded with zero errors. The throughput scales from 84 tok/s at single-user concurrency to 418 tok/s at C=8, with latency growing modestly from 6.08 to 9.80 seconds.

Why This Message Was Written: The Reasoning and Context

To understand why this benchmark was run, one must look at the trajectory that led to it. The session had begun with a focus on the GLM-5-NVFP4 model, then pivoted to the NVFP4 variant of Kimi-K2.5 (a 1T-parameter DeepSeek V3-architecture model). After extensive debugging of GGUF loading, Triton MLA attention backends, and dequantization shard ordering, the NVFP4 Kimi-K2.5 had been deployed and benchmarked at approximately 61 tok/s single-stream. But the team had identified a fundamental bottleneck: PCIe allreduce across 8 GPUs for the 61-layer Multi-head Latent Attention (MLA) architecture. The MLA architecture, which requires complex allreduce operations for its KV cache projections, was choking on the PCIe bandwidth between GPUs.

The user then made a strategic decision: pivot to MiniMax-M2.5, a 230B-parameter FP8 model using Grouped Query Attention (GQA) instead of MLA. This model had several theoretical advantages: it was smaller (230B total, ~10B active parameters per token due to Mixture-of-Experts), used standard FlashAttention that didn't require custom SM120 backends, and could potentially run on just 4 GPUs instead of 8. The download completed, the service was deployed, and initial smoke tests showed correct output. The first benchmark run (see [msg 2288]) had already produced impressive results: 84.4 tok/s at C=1, scaling to 353.2 tok/s at C=8.

The assistant then said "These results are spectacular. Let me push even higher" ([msg 2289]) and edited the benchmark script to test additional concurrency levels. The message [msg 2290] is the second benchmark run — a re-validation with slightly modified parameters. The C=8 result improved from 353.2 to 417.7 tok/s, likely due to better warmup or reduced system noise. This message was written to capture and present these results as evidence that the pivot was working.## How Decisions Were Made

The decision to benchmark MiniMax-M2.5 was not arbitrary — it emerged from a systematic evaluation of hardware constraints. The team had spent days wrestling with NVFP4 Kimi-K2.5, only to discover that its MLA architecture was fundamentally ill-suited to the PCIe topology of their 8-GPU Blackwell machine. Each inference step required allreducing 61 layers of KV projection tensors across all 8 GPUs, and the PCIe Gen5 x16 links between GPUs (or through the CPU) became the bottleneck. The throughput plateaued at ~61 tok/s regardless of optimization attempts.

The pivot to MiniMax-M2.5 was motivated by its architectural differences: GQA instead of MLA, a much smaller total parameter count (230B vs 1T), and native FP8 quantization that required no custom dequantization kernels. The team's hypothesis was that a smaller, more efficiently quantized model with simpler attention mechanics would outperform the larger model on PCIe-bound hardware, even if the larger model had more raw capability.

The benchmark in [msg 2290] validated this hypothesis decisively. At single-user concurrency, MiniMax-M2.5 achieved 84.2 tok/s — a 38% improvement over Kimi-K2.5's 61 tok/s. At C=8, it reached 417.7 tok/s versus Kimi-K2.5's 277 tok/s, a 51% improvement. And critically, it used only 4 GPUs (TP=4), leaving the other 4 GPUs idle at 33W each, compared to Kimi-K2.5's full 8-GPU deployment consuming ~680W at idle.

Assumptions Made

Several assumptions underpin this benchmark. First, the assistant assumed that the benchmark script was correctly configured for the MiniMax-M2.5 model — specifically, that the prompt length, max_tokens, and other parameters matched the model's capabilities. The benchmark used 512 max_tokens with chat completions, which was appropriate for this non-reasoning model. Second, the assistant assumed that the model had been fully loaded and was in a steady state before benchmarking; the "Warming up..." line in the output confirms that a warmup phase was executed. Third, the assistant assumed that the TP=4 configuration was optimal — a reasonable assumption given that the TP=8 attempt ([msg 2298]) would later fail due to FP8 block quantization alignment issues.

There was also an implicit assumption about the reproducibility of benchmarks. The first run ([msg 2288]) showed 84.4 tok/s at C=1 and 353.2 tok/s at C=8. The second run ([msg 2290]) showed 84.2 tok/s at C=1 and 417.7 tok/s at C=8. While the single-user result was nearly identical (84.4 vs 84.2, within noise), the C=8 result differed significantly (353.2 vs 417.7). This discrepancy was likely due to the first run having a different benchmark configuration (perhaps fewer total requests per concurrency level) or system state differences. The assistant did not comment on this discrepancy, instead treating the second run as a straightforward improvement.

Input Knowledge Required

To interpret this message, one needs substantial context about the hardware and software stack. The system runs Ubuntu 24.04 with NVIDIA drivers 590.48.01 and CUDA Toolkit 13.1, with eight RTX PRO 6000 Blackwell GPUs (each with 97.8 GB of HBM4 memory). The vLLM installation is a nightly build with custom patches for SM120 support. The model is MiniMax-M2.5, a 230B-parameter FP8 Mixture-of-Experts model with 10B active parameters per token, using Grouped Query Attention. The benchmark script sends chat completion requests to the vLLM OpenAI-compatible API server running on localhost:8000, measuring throughput and latency across multiple concurrent client connections.

One also needs to understand the significance of the "C=" column: it represents the number of concurrent requests being sent to the server simultaneously. The benchmark ramps up concurrency to test how the system handles increasing load, measuring both throughput (tok/s) and latency (seconds per request). The "4/4 ok" notation means all 4 requests at that concurrency level completed successfully.

Output Knowledge Created

This message created concrete, actionable knowledge. It demonstrated that:

  1. MiniMax-M2.5 achieves 84.2 tok/s single-stream on 4 Blackwell GPUs, well above the 40-50 tok/s target.
  2. Throughput scales nearly linearly with concurrency up to C=4 (84 → 253 tok/s, approximately 3x), then begins to show diminishing returns at C=8 (253 → 418 tok/s, approximately 1.65x).
  3. Latency remains remarkably stable, growing only from 6.08s at C=1 to 9.80s at C=8, indicating efficient batching.
  4. The model produces zero errors across all test runs, confirming stability.
  5. The TP=4 configuration is the correct choice — not just empirically but provably, as TP=8 would later be shown to be incompatible with the model's FP8 block quantization. These numbers became the baseline for all subsequent optimization work. The team would later push throughput to nearly 4,000 tok/s by combining TP=8 with Expert Parallelism (EP=8), but that was only possible after understanding the TP=4 baseline and the FP8 alignment constraints discovered through this benchmark.## Mistakes and Incorrect Assumptions While the benchmark itself was accurate, there were subtle issues worth noting. The most significant was the discrepancy between the two benchmark runs. The first run ([msg 2288]) showed 353.2 tok/s at C=8, while the second run ([msg 2290]) showed 417.7 tok/s — a 18% difference. This suggests that the benchmark methodology was not perfectly controlled. Possible explanations include: different numbers of total requests per concurrency level (the first run may have used fewer iterations), different system thermal states, or interference from other processes. The assistant did not investigate this discrepancy, which means the benchmark results should be interpreted with some caution. Another potential issue is that the benchmark used a single prompt type. All requests used the same chat completion format with similar input lengths. Real-world workloads would have variable input lengths, which could affect throughput and latency differently. The benchmark also did not measure time-to-first-token (TTFT), which is critical for user-facing applications. The reported latency is end-to-end for the full generation, which is useful for throughput but doesn't capture responsiveness.

The Thinking Process Visible in the Message

Although the message itself is just a benchmark output, the surrounding messages reveal the assistant's thinking process. In [msg 2289], the assistant wrote "These results are spectacular. Let me push even higher" — this shows a mindset of iterative optimization, where good results are not accepted as final but rather as a starting point for further improvement. The assistant immediately edited the benchmark script to test higher concurrency levels, demonstrating a systematic approach to exploring the performance envelope.

The choice to benchmark at concurrency levels 1, 2, 4, and 8 reveals an understanding of how throughput scales with load. These are powers of two, a common choice for benchmarking because they create a clean logarithmic progression. The assistant likely chose these levels to observe how the system transitions from latency-bound (C=1) to throughput-bound (C=8) regimes.

The fact that the assistant ran a second benchmark after modifying the script, rather than simply reporting the first run's results, shows a commitment to rigor. The first run had already produced excellent numbers, but the assistant wanted to explore higher concurrency before declaring success. This iterative approach — benchmark, analyze, adjust, benchmark again — is characteristic of effective performance engineering.

Conclusion

Message [msg 2290] is a pivotal moment in a complex deployment journey. It represents the moment when a strategic bet paid off: the pivot from a problematic NVFP4 MLA model to a well-suited FP8 GQA model was validated by hard numbers. The benchmark showed 84 tok/s single-stream and 418 tok/s at moderate concurrency, using only half the GPUs and half the power of the previous setup. These numbers would serve as the foundation for further optimization, including Expert Parallelism configurations that eventually pushed throughput to nearly 4,000 tok/s. In the broader narrative of this session, this message is the quiet confirmation that hardware-aware model selection — choosing the right architecture for the available interconnect — matters as much as raw model capability.