The Geometry of Parallelism: Reasoning Through TP and EP Constraints for MiniMax-M2.5 on 8 Blackwell GPUs

Introduction

In the high-stakes world of large language model deployment, choosing the right parallelism strategy is not merely a matter of preference—it is a constraint-satisfaction problem governed by the arithmetic of tensor dimensions, quantization block sizes, and hardware topology. Message [msg 2309] captures a pivotal moment in an extended coding session where an AI assistant, having just benchmarked the MiniMax-M2.5 FP8 model at impressive throughputs on 4 GPUs, is asked by the user whether the model can exploit all 8 available GPUs through tensor parallelism (TP) or expert parallelism (EP). What follows is a masterclass in systematic reasoning: the assistant enumerates the architectural constraints, checks divisibility conditions, evaluates the interaction between parallelism strategies and quantization alignment, and arrives at a ranked set of viable configurations—all before executing a single command.

This article examines this single message in depth, unpacking the reasoning process, the assumptions made, the knowledge required and generated, and the broader significance of this kind of analytical thinking in production ML engineering.

Context: The Problem Space

To understand why this message matters, we must situate it within the larger narrative of the session. The team has been deploying and benchmarking trillion-parameter-scale models on a machine equipped with 8 NVIDIA RTX PRO 6000 Blackwell GPUs. Earlier, they had struggled with the NVFP4 variant of Kimi-K2.5, identifying a fundamental bottleneck: PCIe allreduce across 8 GPUs for the 61-layer MLA (Multi-head Latent Attention) architecture. They pivoted to MiniMax-M2.5, a 230B-parameter Mixture-of-Experts model with only 10B active parameters per token, using Grouped-Query Attention (GQA) instead of MLA. With TP=4, it loaded in 75 seconds (versus 13 minutes for Kimi-K2.5) and achieved up to 2,586 tok/s at high concurrency, using only half the GPUs and half the power.

The user's question—"can we do tp/ep or tp6?" ([msg 2303])—is a natural next step: if the model is already performing well on 4 GPUs, can we engage all 8 GPUs to push throughput even higher? The assistant's initial investigation ([msg 2304] through [msg 2308]) revealed that TP=6 is problematic because 8 key-value heads are not divisible by 6, and that vLLM does support expert parallelism via --enable-expert-parallel. Message [msg 2309] is where the assistant synthesizes these findings into a comprehensive analysis.

The Reasoning Process: A Step-by-Step Deconstruction

The message opens with a confirmation: "EP is supported via --enable-expert-parallel / -ep." This establishes the tooling available. Then the assistant immediately pivots to the core analytical task: checking the math for TP+EP combinations.

Step 1: Enumerating the Constraints

The assistant identifies three key constraints:

  1. intermediate_size = 1536 (per expert in the MoE layers)
  2. FP8 block_size = [128, 128] — the sharded dimension must be divisible by 128
  3. TP * EP <= total_GPUs — the product of tensor parallelism and expert parallelism cannot exceed the available 8 GPUs This is a classic constraint-satisfaction framing. The assistant is treating the parallelism configuration as a set of integer parameters that must satisfy divisibility conditions imposed by the model architecture and the quantization scheme.

Step 2: Understanding the EP/TP Interaction

A crucial insight follows: "When EP is enabled, the MoE layer uses EP for expert distribution and TP for non-expert layers (attention). The expert weights are NOT tensor-sharded by TP when EP is on—each EP rank holds a subset of experts in full."

This is the key that unlocks the solution space. With EP, the FP8 block alignment constraint on intermediate_size / TP does not apply to expert weights because they are not tensor-sharded. The constraint only applies to attention and dense layers, which are still handled by TP. This means that configurations previously ruled out by the FP8 alignment issue (like TP=8, which failed because 1536 / 8 = 192 is not divisible by 128) may become viable when combined with EP, because the expert weights are no longer being tensor-sharded.

Step 3: Checking Attention Head Divisibility

The assistant then checks the attention dimensions: hidden_size=3072, num_attention_heads=48, num_key_value_heads=8. The divisibility analysis proceeds:

Step 4: Enumerating Viable Configurations

With the constraints understood, the assistant enumerates all viable options for 8 GPUs:

Step 5: Making a Decision

The assistant selects TP=2, EP=4 as the first configuration to try, describing it as "a good balance." This decision reflects engineering judgment: TP=2 provides some parallelism for attention layers (reducing memory per GPU for the non-expert weights) while EP=4 distributes the 256 experts across 4 groups of GPUs, reducing the allreduce overhead compared to TP=8 or TP=4.

The message concludes by executing the command to stop the current service and free the GPUs, preparing for the new configuration.

Assumptions and Their Validity

The assistant makes several assumptions in this analysis:

  1. That EP removes the FP8 alignment constraint on expert weights. This is correct given the description: with EP, each GPU holds complete expert weights (not sharded), so the intermediate_size dimension is not being divided by TP for experts. However, this assumes the vLLM implementation matches the conceptual model—a non-trivial assumption when dealing with complex distributed inference engines.
  2. That the attention dimensions are the only TP constraints for non-expert layers. The assistant checks hidden_size, num_attention_heads, and num_key_value_heads but does not check other potential TP-sensitive dimensions like ffn_hidden_size or any layer normalization parameters. For this model architecture, these are likely fine, but the analysis is not exhaustive.
  3. That TP=2, EP=4 is a good starting point. This is a heuristic judgment. The assistant could have started with TP=1, EP=8 (which maximizes expert distribution) or TP=4, EP=2 (which keeps the attention parallelism that was already working well). The choice reflects a preference for balance.
  4. That the current service can be safely stopped and restarted. The assistant assumes no other clients are using the service, which is reasonable in a benchmarking context but would be risky in production. These assumptions are generally sound, but they highlight the gap between analytical reasoning and empirical validation. The assistant is about to test TP=2, EP=4 empirically—the true test of whether the assumptions hold will come in subsequent messages.

Input Knowledge Required

To follow this analysis, a reader needs to understand:

Output Knowledge Created

This message produces several valuable pieces of knowledge:

  1. A complete enumeration of viable parallelism configurations for MiniMax-M2.5 on 8 Blackwell GPUs, with explicit reasoning for why each configuration works or fails
  2. A clear explanation of why TP=6 is impossible (KV head divisibility) and why TP=8 failed earlier (FP8 block alignment on expert weights)
  3. The insight that EP bypasses the FP8 alignment constraint on expert weights, opening up configurations that were previously ruled out
  4. A ranked recommendation with TP=2, EP=4 as the first candidate to test
  5. The mathematical framework for evaluating parallelism strategies, which is reusable for any model with known dimensions This knowledge is immediately actionable—the assistant proceeds to implement the recommended configuration. But it also has lasting value as documentation of the reasoning process, which could inform future deployment decisions for similar models.

Broader Significance

What makes this message noteworthy is not the specific configuration chosen, but the systematic reasoning process it demonstrates. The assistant does not guess or rely on trial and error. Instead, it:

  1. Identifies all constraints (architectural, quantization, hardware)
  2. Understands the interaction between parallelism strategies (how EP changes the constraint surface)
  3. Checks divisibility exhaustively across all TP values
  4. Enumerates the full solution space before selecting a candidate
  5. Documents the reasoning so that the decision is transparent and auditable This is the kind of analytical thinking that separates effective ML engineering from haphazard configuration. In production environments where model deployment decisions have significant cost and performance implications, the ability to reason through parallelism constraints before running experiments is invaluable. The message also illustrates a broader principle: that deploying large models is fundamentally a constraint-satisfaction problem. The model architecture defines a set of integer dimensions; the hardware defines a set of resource limits; the quantization scheme defines alignment requirements. The engineer's job is to find configurations that satisfy all constraints simultaneously, then empirically validate that the theoretical configuration works in practice.

Conclusion

Message [msg 2309] is a compact demonstration of analytical reasoning in ML engineering. In a few hundred words, the assistant works through architectural constraints, quantization alignment, parallelism interactions, and hardware topology to enumerate viable configurations for deploying MiniMax-M2.5 on 8 Blackwell GPUs. The reasoning is clear, the assumptions are stated, and the decision is justified. Whether the chosen configuration (TP=2, EP=4) delivers the expected performance is a question for subsequent messages—but the analytical framework established here would remain valid regardless of the empirical outcome.

This is the kind of thinking that turns a collection of GPUs into a well-utilized inference cluster: not by throwing hardware at the problem, but by understanding the geometry of the model and finding the configuration that fits.