Diagnosing Parallelism Failure: When Expert Parallelism Reuses the Tensor Parallelism Group

Introduction

In the high-stakes world of deploying large language models on multi-GPU hardware, few moments are as instructive as a well-diagnosed failure. Message <msg id=2313> captures one such moment in an ongoing session to deploy and benchmark 1-trillion-parameter models on 8× NVIDIA RTX PRO 6000 Blackwell GPUs. The assistant had just attempted to launch MiniMax-M2.5, a 230B-parameter FP8 model with Mixture-of-Experts (MoE) architecture, using a combination of tensor parallelism (TP) and expert parallelism (EP). The attempt failed with an out-of-memory (OOM) error. This message is the diagnostic pivot — the moment the assistant stops, examines the logs, performs the arithmetic, and discovers that the failure was not due to insufficient total GPU memory, but due to a fundamental misunderstanding of how vLLM's parallelism groups are constructed.

The message is a masterclass in systematic debugging: observe the symptom, inspect the evidence, calculate the constraints, and formulate the next hypothesis. It also reveals a subtle but critical design detail in vLLM's distributed execution model that any practitioner deploying MoE models must understand.

The Message

Here is the message in full:

So with --tensor-parallel-size 2 --enable-expert-parallel, vLLM uses world_size=2, TP=2, EP=2. The EP is just reusing the TP group — experts are split across the same 2 GPUs. With 256 experts split over 2 GPUs = 128 experts per GPU, each 1536 intermediate * FP8 = too much for 2 GPUs.

>

The model is 230GB total. Split over 2 GPUs = 115GB per GPU, which exceeds 96GB. That's the root cause.

>

To get true TP+EP with more GPUs, I need to specify the total world size differently. Let me check if vLLM supports separate EP sizing:

>

``bash ssh root@10.1.230.174 "/root/ml-env/bin/python3 -m vllm.entrypoints.openai.api_server --help 2>/dev/null | grep -B1 -A3 'expert-parallel\|all2all'" ``

This is deceptively brief. In just a few lines, the assistant performs a complete root-cause analysis and sets up the next investigative step. Let us unpack each layer.

Context and Background

To understand this message, we must trace back through the preceding conversation. The session had been a whirlwind tour of model deployment. Earlier, the team had pivoted from the NVFP4 variant of Kimi-K2.5 (a 1T-parameter DeepSeek V3 architecture model) to MiniMax-M2.5, a 230B-parameter FP8 model using Grouped-Query Attention (GQA) instead of Multi-Head Latent Attention (MLA). The MiniMax had proven dramatically faster — 84 tok/s single-stream versus 61 tok/s for Kimi-K2.5, and up to 2,586 tok/s at high concurrency — while using only 4 GPUs with TP=4.

The user then asked in <msg id=2303>: "can we do tp/ep or tp6?" This was a natural question. If the model already performs well on 4 GPUs, could it perform even better by utilizing all 8 GPUs through a combination of tensor parallelism and expert parallelism? Expert parallelism distributes the MoE experts across GPUs rather than replicating them, potentially reducing memory pressure and improving throughput.

The assistant spent several messages investigating the viability. In <msg id=2304>, it checked vLLM's help flags. In <msg id=2305> through <msg id=2308>, it searched the vLLM source code for EP-related configuration, discovering --enable-expert-parallel / -ep flag and the all2all-backend option. In <msg id=2309>, it performed the mathematical analysis:

WHY This Message Was Written

The message was written to answer a single, urgent question: Why did the TP=2+EP configuration fail with OOM when we have 8 GPUs with 96GB each?

The surface-level answer — "not enough GPU memory" — was unsatisfying because the math should have worked. With 8 GPUs × 96GB = 768GB total capacity, and a model requiring only 230GB, there should have been ample room. The OOM error on "GPU 1" suggested something was fundamentally wrong with how the model was distributed.

The assistant needed to reconcile two conflicting facts:

  1. The model is 230GB and we have 768GB total — it should fit
  2. The process OOM'd on GPU 1 with only 2 workers The resolution of this contradiction required understanding vLLM's parallelism group construction, which is precisely what this message delivers.

HOW Decisions Were Made

The decision-making in this message follows a clear diagnostic pattern:

Step 1: Observe the evidence. The assistant had already retrieved the logs in <msg id=2312> showing world_size=2 rank=0 and world_size=2 rank=1, with TP rank 0 and EP rank 0 on the same worker. The logs also showed the assignment: "rank 0 in world size 2 is assigned as DP rank 0, PP rank 0, PCP rank 0, TP rank 0, EP rank 0."

Step 2: Interpret the evidence. The assistant concludes: "The EP is just reusing the TP group — experts are split across the same 2 GPUs." This is the key insight. When --enable-expert-parallel is used without a separate EP size specification, vLLM creates a single process group where TP and EP share the same world. The world size is determined by --tensor-parallel-size, and EP is layered on top of the same group.

Step 3: Calculate the consequence. With 2 GPUs and a 230GB model, each GPU must hold 115GB. But each GPU only has ~96GB (97887 MiB). The OOM is inevitable.

Step 4: Formulate the next hypothesis. If the problem is that EP is tied to TP's world size, then the solution is to find a way to specify EP separately. The assistant decides to check vLLM's help output for any flags that might allow independent EP sizing.

Step 5: Execute the investigation. The assistant runs a grep command on vLLM's help output to search for expert-parallel and all2all-related flags.

This is textbook root-cause analysis: observe → interpret → calculate → hypothesize → investigate.

Assumptions Made

This message reveals several assumptions, some correct and some incorrect:

Correct assumption: The assistant correctly assumes that the OOM is not a transient or random error but has a structural cause that can be identified through log inspection. This assumption is validated immediately.

Correct assumption: The assistant assumes that vLLM's parallelism configuration is deterministic and that the log output will reveal the exact group assignments. This is correct — the logs clearly show world_size=2.

Incorrect assumption (earlier, now corrected): The assistant had assumed in <msg id=2309> that --enable-expert-parallel combined with --tensor-parallel-size 2 would distribute work across all 8 GPUs. The reasoning was: "TP=2, EP=4: 8 GPUs total, experts split 4 ways, attention TP=2." But this assumed that EP would create its own independent group spanning all GPUs, which vLLM does not do by default.

Implicit assumption: The assistant assumes that there is a way to specify EP independently in vLLM. This is not yet confirmed — the grep command is checking this very question. If no such flag exists, the team would need a different approach (e.g., using TP=4 on 4 GPUs as before, or patching vLLM).

Assumption about model size: The assistant assumes the model is "230GB total." This is a rough figure — the actual memory footprint depends on activation memory, KV cache, and other overheads. But as a first-order approximation, it correctly explains the OOM.

Mistakes and Incorrect Assumptions

The most significant mistake was the assumption about how vLLM constructs EP groups. This is a subtle but critical detail in vLLM's distributed execution model.

In many distributed training and inference frameworks, tensor parallelism and expert parallelism are orthogonal dimensions. TP shards individual tensor operations across GPUs, while EP distributes different experts to different GPUs. In an ideal implementation, you could specify TP=2 and EP=4 independently, and the framework would create 8 workers total (2 TP groups × 4 EP replicas).

However, vLLM's implementation (at least in this version) does not work this way. The --enable-expert-parallel flag, when used without additional configuration, simply tags the existing TP group with EP responsibilities. The world size remains --tensor-parallel-size, and EP is layered on top of the same group. This means each GPU holds both a TP shard of the attention layers AND a subset of experts — but the total number of GPUs is still just the TP size.

This is a design choice that prioritizes simplicity and avoids the complexity of managing multiple communication groups. But it means that EP does not increase the total number of GPUs used — it only changes how experts are distributed within the existing TP group.

The assistant's earlier analysis in <msg id=2309> had considered "TP=2, EP=4" as a viable option, but the actual implementation does not support this without additional configuration. This is not necessarily a mistake in the assistant's reasoning — it was exploring possibilities — but the launch in <msg id=2310> should have specified the desired world size if vLLM supported it.

Another subtle point: the assistant says "With 256 experts split over 2 GPUs = 128 experts per GPU, each 1536 intermediate * FP8 = too much for 2 GPUs." This is slightly imprecise — the OOM is not just about expert weights but about the total model footprint including attention layers, embeddings, and KV cache. But the conclusion is correct: 115GB > 96GB.

Input Knowledge Required

To fully understand this message, the reader needs:

  1. vLLM architecture knowledge: Understanding that vLLM uses process groups for distributed execution, with world_size determining the number of worker processes. Knowledge of TP (tensor parallelism), EP (expert parallelism), and how they relate to MoE models.
  2. MoE model architecture: Understanding that MiniMax-M2.5 has 256 experts with intermediate_size=1536, and that FP8 quantization uses 128×128 blocks. Knowledge that experts can be distributed across GPUs (EP) or sharded across GPUs (TP).
  3. GPU memory constraints: Knowing that each RTX PRO 6000 Blackwell GPU has ~96GB of usable memory (97887 MiB), and that a 230GB model cannot fit on 2 GPUs.
  4. The history of the session: Understanding that this is part of a larger exploration of parallelism configurations, and that TP=4 on 4 GPUs was the previously working configuration.
  5. Linux/bash basics: Understanding the grep command structure, the vLLM help output format, and the SSH remote execution pattern.

Output Knowledge Created

This message produces several valuable pieces of knowledge:

  1. Root cause identification: The OOM is definitively traced to world_size=2 with a 230GB model, not to any other factor.
  2. vLLM EP behavior documented: The message documents that --enable-expert-parallel with --tensor-parallel-size 2 creates world_size=2 with TP=2 and EP=2, where EP reuses the TP group. This is a non-obvious behavior that would otherwise require reading vLLM source code or extensive experimentation.
  3. Memory calculation: The message provides a clear calculation: 230GB / 2 GPUs = 115GB/GPU > 96GB. This confirms the OOM is structurally inevitable with this configuration.
  4. Next investigative direction: The message identifies that separate EP sizing might be the solution and initiates a search for the relevant vLLM flags.
  5. A diagnostic methodology: The message demonstrates a reproducible pattern for debugging distributed inference failures: check world_size, compare against model size and per-GPU memory, calculate the expected memory per GPU, and verify against the observed failure.

The Thinking Process

The thinking process visible in this message is a model of structured reasoning. Let me trace it step by step:

Step 1 — Synthesize the evidence: The assistant begins by stating the discovered fact: "So with --tensor-parallel-size 2 --enable-expert-parallel, vLLM uses world_size=2, TP=2, EP=2." The word "So" signals that this is a conclusion drawn from the log output examined in the previous message.

Step 2 — Interpret the implication: "The EP is just reusing the TP group — experts are split across the same 2 GPUs." This is the crucial insight. The assistant recognizes that EP is not creating a separate, larger group but is overlaid on the existing TP group.

Step 3 — Quantify the consequence: "With 256 experts split over 2 GPUs = 128 experts per GPU, each 1536 intermediate * FP8 = too much for 2 GPUs." The assistant does the arithmetic to show why this configuration fails.

Step 4 — Generalize to total model size: "The model is 230GB total. Split over 2 GPUs = 115GB per GPU, which exceeds 96GB. That's the root cause." This is the definitive conclusion. The assistant steps back from the expert-specific calculation to the total model memory, confirming the diagnosis.

Step 5 — Formulate the solution direction: "To get true TP+EP with more GPUs, I need to specify the total world size differently." The assistant identifies that the fix requires a different way of configuring vLLM, not a different model or different hardware.

Step 6 — Execute the investigation: "Let me check if vLLM supports separate EP sizing." The assistant then runs a grep command to search vLLM's help output for relevant flags.

This thinking process is notable for its economy — each step follows logically from the previous one, and the assistant never speculates beyond what the evidence supports. When the evidence runs out (after identifying that separate EP sizing is needed), the assistant immediately pivots to gathering more data.

Broader Significance

This message, while brief, illuminates a fundamental challenge in deploying large MoE models on multi-GPU hardware: the interaction between different parallelism strategies is often non-obvious and framework-specific.

In an ideal world, tensor parallelism and expert parallelism would be orthogonal knobs that can be tuned independently. TP shards individual layers' weights and activations across GPUs to reduce per-GPU memory and enable larger batch sizes. EP distributes different experts to different GPUs to reduce the memory footprint of the expert weights. The two can be combined: TP for the attention and dense layers, EP for the MoE experts.

But in practice, frameworks like vLLM make design choices that couple these dimensions. In this case, EP is implemented as a property of the TP group rather than as an independent parallelism dimension. This means that enabling EP does not increase the total number of GPUs — it only changes how experts are distributed within the existing TP group.

This has profound implications for deployment strategy. If you have 8 GPUs and want to use EP, you might need to set TP=4 (to get 4 GPUs in the group) and then hope EP distributes experts across those 4 GPUs. But if the model's attention layers require TP=4 for memory reasons (as MiniMax does with its 230GB footprint), then EP on 4 GPUs might not provide enough expert distribution to be beneficial.

Alternatively, if vLLM supports a separate --expert-parallel-size flag (which the assistant is about to discover), you could set TP=2 and EP=4 to get 8 workers total. But this requires the framework to support multi-dimensional process groups, which adds significant complexity to the distributed runtime.

Conclusion

Message <msg id=2313> is a small but perfectly formed example of diagnostic reasoning in the context of large-scale ML deployment. In just a few sentences, the assistant:

  1. Identifies the root cause of an OOM failure
  2. Explains why the expected behavior (8 GPUs) did not occur
  3. Calculates the memory arithmetic that confirms the diagnosis
  4. Formulates the next investigative step
  5. Executes that step with a targeted command The message also reveals a critical design detail in vLLM's parallelism implementation that any practitioner deploying MoE models must understand: expert parallelism, when enabled without separate configuration, reuses the tensor parallelism group rather than creating an independent, larger group. This knowledge is immediately actionable. It tells the team that they cannot simply add --enable-expert-parallel to their existing TP=2 configuration and expect it to use 8 GPUs. They need either a different configuration flag, a different version of vLLM, or a different approach entirely (such as TP=4 on 4 GPUs, which was already working well). The message ends with the assistant actively investigating the next step — checking vLLM's help for separate EP sizing flags. This forward momentum, even in the face of a failed experiment, is characteristic of effective engineering: treat every failure as data, extract the lesson, and move to the next hypothesis.