The Benchmark That Changed Everything: MiniMax-M2.5 vs Kimi-K2.5 on 8x Blackwell GPUs

In the high-stakes world of large language model deployment, a single benchmark comparison can reshape an entire infrastructure strategy. Message [msg 2292] in this opencode session is precisely that kind of document: a concise, data-packed summary that crystallizes hours of debugging, configuration, and optimization into a single compelling narrative. The assistant, having just finished deploying and stress-testing the MiniMax-M2.5 model on an 8x RTX PRO 6000 Blackwell GPU system, delivers a head-to-head comparison against the previously deployed Kimi-K2.5 model that is both a technical report and a strategic recommendation.

The Context: A Long Road to This Moment

To understand why this message exists, one must appreciate the journey that preceded it. The session had been wrestling with large-scale model deployment for hours. Earlier segments (see [msg 2262] through [msg 2291]) document a painful pivot from the NVFP4 variant of Kimi-K2.5 — a 1-trillion-parameter model using DeepSeek V3's MLA (Multi-Head Latent Attention) architecture — to MiniMax-M2.5, a 230-billion-parameter model (10B active) using GQA (Grouped Query Attention). The NVFP4 Kimi had been benchmarked at roughly 61 tok/s single-stream, but the team had identified a fundamental bottleneck: PCIe allreduce across 8 GPUs for the 61-layer MLA architecture. The MLA architecture, while elegant, required expensive allreduce operations that saturated the PCIe bus, limiting throughput.

The pivot to MiniMax-M2.5 was not arbitrary. It was a calculated bet: GQA models use standard FlashAttention, which has none of the SM120 backend limitations that plagued MLA on these Blackwell GPUs. Moreover, MiniMax-M2.5 is a native FP8 model — no NVFP4 quirks, no config modifications needed. The assistant had downloaded the model (discovering a HuggingFace shard numbering quirk in the process — see [msg 2262] through [msg 2266]), deployed it as a systemd service, survived an OOM crash during sampler warmup (fixed by adding --max-num-seqs 256 in [msg 2277]), and watched it load in just 75 seconds — a staggering 10x improvement over Kimi-K2.5's 13-minute cold start.

What the Message Actually Says

The message is structured as a clean comparison table followed by qualitative analysis. Here is the exact content:

Setup: 512 max_tokens, chat completions, same prompt, same hardware

>

| Concurrency | Kimi-K2.5 tok/s | MiniMax-M2.5 tok/s | Speedup | |---|---|---|---| | 1 | 61 | 84 | 1.4x | | 2 | 58 | 143 | 2.5x | | 4 | 94 | 253 | 2.7x | | 8 | 277 | 418 | 1.5x | | 16 | 544 | 549 | ~1.0x | | 32 | 464-800 | 841 | ~1.3x | | 64 | 803 | 1,429 | 1.8x | | 128 | 1,239 | 2,261 | 1.8x | | 256 | (not tested) | 2,586 | -- |

>

Single-user latency: 6.1s vs 8.4s (37% faster)

The table is followed by a list of key advantages and a model quality comparison.

The Reasoning Behind Every Number

The assistant's thinking process is visible in how the data is presented. The table is ordered by concurrency level, which reveals a fascinating pattern. At low concurrency (1-8), MiniMax dominates with 1.4x to 2.7x speedups. At concurrency 16, the two models are nearly tied (544 vs 549 tok/s). Then at higher concurrency (32-128), MiniMax pulls ahead again with 1.3x to 1.8x speedups. This non-monotonic pattern tells a story about the different bottlenecks each model hits as concurrency increases.

At low concurrency, MiniMax's advantage comes from its smaller active parameter count (10B vs Kimi's ~37B active) and its efficient GQA attention. The MLA architecture in Kimi, while theoretically more parameter-efficient, requires complex allreduce operations that become a smaller fraction of total work at higher batch sizes — hence the convergence at concurrency 16. But at very high concurrency, MiniMax's simpler architecture and smaller memory footprint per sequence allow it to pack more sequences into the same GPU memory, yielding better throughput.

The assistant chose to include the Kimi-K2.5 data point at concurrency 32 as a range ("464-800") rather than a single number. This is a subtle but important signal: the Kimi model was showing instability at that concurrency level, with throughput varying by nearly 2x. MiniMax, by contrast, produced a clean 841 tok/s. This stability difference is a real operational concern for production deployments.

Assumptions Embedded in the Analysis

The message makes several assumptions worth examining. First, it assumes that "not a reasoning model by default" is an advantage, arguing that "reasoning tokens are shorter/more efficient, so more of the token budget goes to actual content." This is a pragmatic engineering perspective, but it glosses over the fact that Kimi-K2.5's reasoning capabilities are precisely what make it valuable for code and math tasks. The assistant acknowledges this tension in the model quality comparison section, but the framing clearly favors MiniMax for throughput-oriented workloads.

Second, the message assumes that using only 4 GPUs (TP=4) is an unambiguous win. The assistant notes that "GPUs 4-7 are free for another workload" and that idle power is halved (~330W vs ~680W). This is correct for the current setup, but it implicitly assumes that the remaining 4 GPUs can actually be utilized for something else — which may not be straightforward given the shared PCIe topology and memory constraints.

Third, the benchmark methodology assumes that 512 max_tokens is a representative workload. This is a reasonable choice for chat completions, but it may not generalize to long-context tasks where Kimi-K2.5's 131K context window could be an advantage.

Input Knowledge Required

To fully understand this message, a reader needs familiarity with several concepts:

• Tensor Parallelism (TP) and how it distributes model layers across GPUs
• Grouped Query Attention (GQA) vs Multi-Head Latent Attention (MLA) — the architectural difference that drives the performance gap
• NVFP4 vs FP8 quantization — the former is NVIDIA's 4-bit floating point format that required custom patches, while the latter is standard and well-supported
• Allreduce operations and why PCIe bandwidth becomes a bottleneck for MLA architectures
• Concurrency benchmarking and the relationship between concurrency, latency, and throughput
• SM120 — the compute capability of Blackwell GPUs, which had compatibility issues with certain attention backends The message also references benchmarks like SWE-bench (80.2%) and BrowseComp (76.3%), which require domain knowledge of LLM evaluation to interpret.

Output Knowledge Created

This message creates several valuable artifacts:

1. A concrete, reproducible benchmark comparing two 1T-class models on identical hardware — rare and valuable data for anyone deploying LLMs on Blackwell GPUs.
2. A decision framework for model selection: the message implicitly argues that for throughput-sensitive applications (chat, coding assistants, tool use), MiniMax-M2.5 is the better choice despite having 4x fewer total parameters.
3. Operational insights: the 75-second startup time, the 4-GPU footprint, and the power consumption data are all directly useful for capacity planning and cost analysis.
4. A cautionary tale about MLA on PCIe-bound systems: the message provides strong evidence that MLA's theoretical advantages are neutralized by real-world hardware constraints.

The Thinking Process Visible in the Message

The assistant's reasoning is most visible in the "Key advantages" bullet list. Each bullet addresses a specific pain point from the earlier Kimi-K2.5 deployment:

• "75s startup vs 13 min" — directly addresses the painful 13-minute cold start that plagued the Kimi deployment
• "Uses only 4 GPUs" — addresses the resource waste of TP=8
• "GQA instead of MLA — standard FlashAttention, no SM120 backend limitations" — addresses the custom Triton MLA sparse attention backend that had to be written for Kimi
• "Native FP8 — no NVFP4 quirks, no config modifications needed" — addresses the weight_utils.py patches and dequantization fixes required for NVFP4 The message is essentially saying: "Every problem we fought with Kimi-K2.5 is solved by MiniMax-M2.5." It's a retrospective justification for the pivot, presented as objective data.

What the Message Does Not Say

Notably, the message does not address the quality difference in model outputs beyond a brief comparison table. The assistant tested MiniMax with a "capital of France" query and a prime number function request ([msg 2283], [msg 2284]), both of which produced correct results. But there is no systematic evaluation of whether MiniMax matches Kimi on complex code generation, mathematical reasoning, or long-context tasks. The message implicitly assumes that the throughput advantage outweighs any quality gap — an assumption that may or may not hold depending on the application.

Conclusion

Message [msg 2292] is a masterclass in data-driven decision-making for ML infrastructure. It distills hours of complex engineering work into a clear, actionable comparison. The assistant demonstrates a sophisticated understanding of the tradeoffs between model architectures, quantization formats, and parallelism strategies, all while keeping the focus on practical outcomes: throughput, latency, startup time, power consumption, and GPU utilization. For anyone deploying large language models on multi-GPU systems, this message is a template for how to evaluate and compare deployment options — and a powerful reminder that sometimes the best model is not the biggest one, but the one that best fits your hardware.