The Question That Unlocked 4,000 tok/s: "can we do tp/ep or tp6?"

In the middle of a marathon coding session spanning multiple model deployments, hardware debugging, and performance benchmarking across 8x Blackwell GPUs, a single short question from the user marked a critical turning point. The message, appearing at index 2303, reads simply:

can we do tp/ep or tp6?

This three-word question — barely a sentence — belies the depth of reasoning, hardware knowledge, and architectural insight it required. It arrived at a moment of apparent resolution, when the assistant had just declared a problem "solved" and recommended a configuration that left half the hardware idle. The user's question challenged that conclusion and sent the session down a path that would ultimately achieve the highest throughput of the entire project: nearly 4,000 tokens per second.

The Context: A Problem Declared Solved

To understand why this question was so pivotal, we need to examine what had just happened. The session had been experimenting with MiniMax-M2.5, a 230B-parameter FP8 Mixture-of-Experts (MoE) model with a Grouped-Query Attention (GQA) architecture. The assistant had attempted to run the model with Tensor Parallelism size 8 (TP=8) — distributing the model across all 8 GPUs — but the launch failed with a cryptic error:

ValueError: The output_size of gate's and up's weight = 192 is not divisible by weight quantization block_n = 128.

The assistant correctly diagnosed the root cause ([msg 2299]): MiniMax-M2.5 uses FP8 block quantization with a block size of [128, 128]. The MoE intermediate size is 1536. When sharded across 8 GPUs, each GPU gets 1536 / 8 = 192 elements, and 192 is not divisible by 128. With TP=4, each GPU gets 1536 / 4 = 384 = 3 × 128, which works perfectly.

The assistant then concluded ([msg 2302]) that "TP=4 is the correct choice" and argued it was "arguably better" because it halves the allreduce overhead, leaves 4 GPUs free for other workloads, and the model's 10B active parameters don't need more compute parallelism.

The User's Challenge: A Question of Hardware Utilization

The user's question — "can we do tp/ep or tp6?" — rejected this comfortable conclusion. The user had 8 RTX PRO 6000 Blackwell GPUs, each with 96GB of memory, and the idea of leaving half of them idle was apparently unacceptable. The question proposed two alternative strategies:

1. TP/EP (Tensor Parallelism + Expert Parallelism): A hybrid approach where attention and dense layers use tensor parallelism while the MoE experts are distributed across GPUs using expert parallelism. This is the theoretically ideal configuration for MoE models because it avoids replicating all experts on every GPU.
2. TP6 (Tensor Parallelism with 6 GPUs): A different sharding factor that might avoid the FP8 alignment issue while still using more hardware than TP=4. The user's assumptions were implicit but significant. They assumed that vLLM supported these configurations, that the FP8 alignment constraint might not apply under EP, and that there was a path to using all available hardware. They also assumed — correctly — that the assistant's "TP=4 is best" conclusion might have been premature.

The Investigation: Uncovering Hidden Constraints

The assistant's response ([msg 2304]) immediately began evaluating both proposals, revealing a sophisticated understanding of model architecture and parallelism strategies.

For TP=6, the math looked promising at first: 1536 / 6 = 256, and 256 / 128 = 2 — divisible by the FP8 block size. But the assistant quickly identified a second constraint: the number of key-value attention heads. MiniMax-M2.5 has num_key_value_heads = 8, and 8 / 6 = 1.33 — not an integer. TP requires the number of attention heads to be evenly divisible by the TP size, so TP=6 was dead on arrival.

For TP/EP, the investigation was more involved. The assistant had to dig into vLLM's source code to understand how expert parallelism interacts with tensor parallelism. Through a series of grep commands across the vLLM codebase (<msg id=2305-2318>), the assistant discovered that:

• vLLM supports --enable-expert-parallel (aliased as -ep)
• When EP is enabled, the MoE layer uses EP for expert distribution and TP only for non-expert layers (attention, dense)
• Critically, with EP, the expert weights are not tensor-sharded — each EP rank holds a subset of experts in full
• This means the FP8 block alignment constraint on intermediate_size / TP does not apply to expert weights when EP is enabled The assistant also discovered a subtle behavior: in vLLM, when EP is enabled, the EP group equals the TP group by default. So --tensor-parallel-size 8 --enable-expert-parallel would create TP=8 for attention and EP=8 for MoE, with the MoE layers using TP=1 internally (no tensor sharding of expert weights).

The Breakthrough: TP=8+EP Works

The assistant's analysis revealed that TP=8 with EP should bypass the FP8 alignment error. An initial attempt with TP=2+EP failed due to OOM (the model's 230GB couldn't fit on 2 GPUs), but TP=8+EP launched successfully in about 90 seconds ([msg 2328]).

The results were transformative. At low concurrency (single user), TP=4 was still faster (84 vs 71 tok/s) due to less allreduce overhead. But at high concurrency, TP=8+EP dominated:

| Concurrency | TP=4 tok/s | TP=8+EP tok/s | Improvement | |---|---|---|---| | 256 | 2,586 | 3,982 | +54% | | 1024 | — | 3,997 | — |

The system achieved a flat ~4,000 tok/s throughput from C=256 through C=1024, with zero errors and no degradation. This was the highest throughput recorded in the entire session — 3.2× what the earlier Kimi-K2.5 NVFP4 model achieved.

What This Question Reveals

The user's question at [msg 2303] is a masterclass in how to drive a technical investigation forward. Rather than accepting the assistant's "solved" conclusion, the user:

1. Identified the core tension: 8 GPUs available, only 4 being used
2. Proposed specific alternatives: TP/EP and TP6 — showing familiarity with parallelism strategies
3. Pushed past the first answer: The assistant had declared TP=4 "arguably better," but the user wanted to explore the tradeoff space more thoroughly The question also reveals the user's mental model: they understood that MoE models have a natural affinity for expert parallelism, and that the FP8 alignment failure with TP=8 might be a limitation of pure tensor parallelism rather than a fundamental hardware constraint. This turned out to be exactly correct.

Input and Output Knowledge

To understand this message, one needs knowledge of:

• Tensor parallelism (TP) and how it shards model weights across GPUs
• Expert parallelism (EP) and how it distributes MoE experts
• FP8 block quantization and its alignment constraints
• The MiniMax-M2.5 model architecture (intermediate_size=1536, num_key_value_heads=8)
• vLLM's parallelism configuration options The message created new knowledge:
• TP=6 is impossible due to KV head divisibility
• TP=8+EP works and bypasses the FP8 alignment constraint
• The crossover point where EP outperforms pure TP is around C=12-16
• Peak throughput of ~4,000 tok/s is achievable on this hardware

Conclusion

"can we do tp/ep or tp6?" is a question that looks simple but carries enormous weight. It challenged a premature conclusion, drove deep codebase investigation, uncovered the interaction between FP8 quantization and parallelism strategies, and ultimately unlocked the highest throughput of the entire project. It's a reminder that in complex systems engineering, the best answers often come from questioning the "obvious" solution and exploring the full design space — even when that means asking a question that seems to have already been answered.