The Benchmark That Confirmed a Hypothesis: TP=8+EP on MiniMax-M2.5

In the middle of a marathon session deploying and benchmarking trillion-parameter models on eight NVIDIA RTX PRO 6000 Blackwell GPUs, a single message captures the moment a risky architectural bet paid off. Message [msg 2331] is deceptively brief — just a few lines of benchmark output — but it represents the climax of an extended chain of reasoning about tensor parallelism, expert parallelism, and the fundamental hardware constraints of PCIe-bound multi-GPU inference.

The Context: A Search for the Optimal Parallelism Strategy

To understand why this message was written, we must trace the reasoning that led to it. The session had been exploring the MiniMax-M2.5 model, a 230B-parameter FP8 Mixture-of-Experts (MoE) architecture with 256 experts, an intermediate size of 1536, and Grouped-Query Attention (GQA) with 48 attention heads and 8 key-value heads. Earlier benchmarks with TP=4 (tensor parallelism across 4 GPUs) had shown respectable performance — 84 tok/s single-stream and over 2,500 tok/s at high concurrency — but left 4 GPUs idle.

The assistant had been systematically investigating whether utilizing all 8 GPUs with Expert Parallelism (EP) could improve throughput. This was not a straightforward decision. The path was littered with obstacles:

1. FP8 block quantization alignment: With TP=8 alone, the expert intermediate size (1536) divided by 8 gives 192, which is not divisible by the FP8 block size of 128, causing a hard crash. This was the initial blocker that made TP=8 seem impossible.
2. OOM with TP=2+EP: A first attempt at EP with TP=2 failed because vLLM's EP implementation ties EP size to TP size — TP=2 meant only 2 GPUs were used, and the 230GB model couldn't fit in 2×96GB.
3. The critical insight about EP semantics: Through careful source code inspection ([msg 2318] through [msg 2325]), the assistant discovered that when EP is enabled, the MoE layers use TP_moe=1 — expert weights are distributed whole across EP ranks, not tensor-sharded. This means the FP8 alignment constraint on intermediate_size / TP simply doesn't apply to expert weights when EP is on. The check only fires for attention/dense layers, which have different dimensions. This last insight was the key that unlocked the TP=8+EP configuration. The assistant launched it ([msg 2327]), waited through the ~90-second load time ([msg 2328]), verified all 8 GPUs were utilized at ~96.6GB each ([msg 2329]), and confirmed the model produced coherent output ([msg 2330]). Only then did it run the benchmark that produced [msg 2331].

What the Message Actually Contains

The message is a direct transcript of running a benchmark script:

Working. Now let me benchmark TP=8+EP and compare against TP=4:
python3 /home/theuser/glm-kimi-sm120-rtx6000bw/benchmark.py
Warming up...

Benchmark results:
  --------------------------------------------------------------------------------
  C=   1 |   4/  4 ok (0 err) | wall=  28.8s |    71.1 tok/s | lat avg=  7.20 p50=  7.20 p99=  7.20
  C=   2 |   8/  8 ok (0 err) | wall=  31.8s |   128.7 tok/s | lat avg=  7.95 p50=  7.96 p99=  7.97
  C=   4 |  12/ 12 ok (0 err) | wall=  25.7s |   239.3 tok/s | lat avg=  8.55 p50=  8.56 p99=  8.58
  C=   8 |  16/ 16 ok (0 err) | wall=  23.1s |   354.6 tok/s | lat avg= 11.54 p50= ...

The benchmark tests concurrency levels (C) from 1 to 8, sending requests in parallel and measuring throughput in tokens per second. The results show a clean progression: 71.1 tok/s at C=1, scaling to 354.6 tok/s at C=8. All requests succeeded (0 errors), and latency remains tightly clustered (p50 and p99 are nearly identical), indicating stable behavior under load.

The message is truncated — the full benchmark continued to higher concurrency levels, and the next message ([msg 2332]) reveals the complete picture. But even this partial output was enough to confirm the server was working correctly and producing reasonable numbers.

The Assumptions and Knowledge Required

This message rests on a substantial foundation of input knowledge:

• Understanding of vLLM's parallelism model: The assistant needed to know how TP and EP interact in vLLM's MoE implementation — specifically that TP_moe becomes 1 when EP is enabled, bypassing the FP8 alignment constraint. This required reading the source code of FusedMoEParallelConfig and understanding the sharding logic.
• Hardware constraints: The 96GB GPU memory limit, PCIe bandwidth for allreduce operations, and the Blackwell SM120 architecture's support for FP8 dequantization were all known quantities from earlier in the session.
• Model architecture: The MiniMax-M2.5 model's specific dimensions (hidden_size=3072, intermediate_size=1536, num_experts=256, num_attention_heads=48, num_key_value_heads=8) were critical for calculating which parallelism configurations were viable.
• The benchmark script: The script at /home/theuser/glm-kimi-sm120-rtx6000bw/benchmark.py was presumably written earlier in the session and reused here for consistent comparison. The key assumption the assistant made was that TP=8+EP would actually work — that vLLM's implementation would correctly handle the combination of tensor parallelism for attention layers and expert parallelism for MoE layers without running into the FP8 alignment issue that blocked plain TP=8. This assumption was validated by the successful server launch in [msg 2328], but the benchmark in [msg 2331] was the first quantitative confirmation.

What the Message Created: Output Knowledge

This message produced several important pieces of knowledge:

1. Quantitative confirmation that TP=8+EP is viable: The model loaded successfully, all 8 GPUs were utilized, and the server produced correct output at competitive throughput.
2. Baseline numbers for low-concurrency performance: At C=1, TP=8+EP achieves 71.1 tok/s — lower than the 84 tok/s seen with TP=4. This is expected because TP=8 introduces more allreduce overhead per token (synchronizing across 8 GPUs instead of 4), and the benefit of EP only manifests at higher batch sizes where the additional KV cache capacity matters.
3. Evidence of good scaling behavior: The throughput scales almost linearly from C=1 (71 tok/s) to C=8 (355 tok/s), suggesting the system is not bottlenecked by any single resource at these concurrency levels.
4. Latency characteristics: The tight p50/p99 spread (e.g., 7.20/7.20 at C=1, 11.54/... at C=8) indicates predictable, low-variance performance — important for production serving. The full significance of these numbers only becomes clear in the next message ([msg 2332]), where the assistant presents the complete comparison table showing that TP=8+EP reaches nearly 4,000 tok/s at C=256 — a 54% improvement over TP=4's 2,586 tok/s. The crossover point is around C=12-16, after which the additional KV cache capacity from 8 GPUs allows much larger batch sizes.

The Reasoning Process Visible in the Message

The message itself is terse, but the reasoning is embedded in its structure. The assistant writes "Working. Now let me benchmark TP=8+EP and compare against TP=4" — this single sentence reveals a deliberate experimental methodology:

• "Working" confirms the prerequisite: the server is up, the model is loaded, and basic functionality has been verified. This is a checkpoint acknowledging that the risky TP=8+EP configuration didn't crash.
• "benchmark TP=8+EP" identifies the experimental variable being tested.
• "compare against TP=4" establishes the control condition. The assistant isn't just collecting numbers in isolation — it's running a comparative experiment designed to answer a specific question: does TP=8+EP outperform TP=4, and under what conditions? The choice to run the benchmark at multiple concurrency levels (C=1, 2, 4, 8, and presumably higher) reflects an understanding that the answer isn't binary — the optimal configuration depends on the workload. This is the thinking of someone who knows that parallelism strategies have different regimes of effectiveness.

The Broader Significance

This message is a turning point in the session. Before it, the assistant had been struggling with the NVFP4 Kimi-K2.5 model, which maxed out at ~60 tok/s single-stream and ~1,200 tok/s at high concurrency due to its MLA (Multi-head Latent Attention) architecture requiring expensive PCIe allreduce operations across all 8 GPUs. The pivot to MiniMax-M2.5 was itself a strategic decision — choosing a GQA model with smaller active parameters that could benefit from EP.

The benchmark in [msg 2331] validates that pivot and confirms that the assistant's deep dive into vLLM's source code to understand the TP/EP interaction was not academic curiosity but practical necessity. Without that investigation, the FP8 alignment error would have remained an unsolved blocker, and the 4,000 tok/s peak throughput achieved in [msg 2332] would never have been realized.

In the broader arc of the conversation, this message represents the moment when hardware-aware model selection and parallelism engineering converge to produce a result that fundamentally changes the deployment strategy — from a single-user coding assistant (where TP=4's 84 tok/s is better) to a high-throughput API serving architecture (where TP=8+EP's 4,000 tok/s dominates). The assistant's next action, deploying the INT4 Kimi-K2.5 as a production service, builds directly on the methodology validated here.