The EP That Wasn't: Debugging vLLM's Expert Parallelism on 8x Blackwell GPUs

In the high-stakes world of deploying trillion-parameter language models on multi-GPU systems, every decision about parallelism strategy carries weight. One wrong assumption about how a framework distributes work can turn a promising configuration into an out-of-memory error, wasting hours of compute time. Message [msg 2312] captures exactly such a moment — a diagnostic pivot point where the assistant, having just watched a carefully-planned TP=2+EP launch crash with an OOM error, stops to investigate why before blindly trying another configuration.

The Context: A Search for the Optimal Parallelism Strategy

The session had been a whirlwind tour of model deployment on 8x NVIDIA RTX PRO 6000 Blackwell GPUs. After struggling with the NVFP4 variant of Kimi-K2.5 (limited by PCIe allreduce bottlenecks for its 61-layer MLA architecture), the team had pivoted to MiniMax-M2.5, a 230B-parameter MoE model with only 10B active parameters per token. Using TP=4 (tensor parallelism across 4 GPUs), it achieved an impressive 84 tok/s single-stream and over 2,500 tok/s at high concurrency — a massive improvement over the Kimi-K2.5 NVFP4 results.

But 4 of the 8 GPUs sat idle. The user naturally asked: can we use all 8 GPUs? Specifically, could they use TP=6 or a combination of tensor parallelism and expert parallelism (TP+EP)?

The assistant investigated TP=6 first ([msg 2304]) and quickly ruled it out: MiniMax-M2.5 has 8 key-value heads, and 8 is not divisible by 6. TP=8 was also ruled out earlier ([msg 2299]) due to FP8 block quantization alignment — the MoE intermediate size of 1536 divided by 8 gives 192, which isn't divisible by the 128-block quantization size.

The remaining option was TP+EP. With expert parallelism, the 256 experts in MiniMax-M2.5's MoE layers would be distributed across GPUs instead of replicated. The assistant calculated that TP=2 with EP=4 would use all 8 GPUs: 2 GPUs per tensor-parallel group, 4 such groups, each holding a subset of experts. This seemed ideal — it would halve the allreduce overhead for attention layers while distributing expert memory across all GPUs.

The Message: A Diagnostic Deep-Dive

Message [msg 2312] opens with the assistant confronting the failure:

OOM with TP=2 + EP. The issue is that with only TP=2, attention layers need hidden_size/2 = 1536 which is fine, but the problem is that EP with TP=2 only uses 2 workers (TP=2), and the EP is automatic.

The assistant is reasoning through the OOM in real-time. It correctly identifies that the attention dimensions are fine with TP=2 (hidden_size=3072, divided by 2 gives 1536, which is well within GPU memory limits). The problem must be elsewhere. The key insight comes next: "EP with TP=2 only uses 2 workers (TP=2), and the EP is automatic."

This is the crucial realization. The assistant had assumed that --enable-expert-parallel combined with --tensor-parallel-size 2 would distribute work across all 8 GPUs — 2 GPUs handling tensor parallelism, with expert parallelism distributing the remaining 4 groups. But the OOM suggests vLLM interpreted the configuration differently.

The assistant then runs a diagnostic command to check the parallel state:

ssh root@10.1.230.174 "grep -E 'world_size|EP rank|TP rank|ep_size|num_gpu' /tmp/vllm_minimax_tp2ep.log | head -15"

The results are revealing:

INFO 02-21 00:55:55 [parallel_state.py:1307] world_size=2 rank=0 local_rank=0 ...
INFO 02-21 00:55:55 [parallel_state.py:1307] world_size=2 rank=1 local_rank=1 ...
INFO 02-21 00:55:55 [parallel_state.py:1535] rank 0 ... TP rank 0, EP rank 0 ...
INFO 02-21 00:55:55 [parallel_state.py:1535] rank 1 ... TP rank 1, EP rank 0 ...

world_size=2. Only 2 GPUs were used. And both workers have EP rank 0 — meaning expert parallelism wasn't actually distributing experts at all. The --enable-expert-parallel flag was accepted silently, but vLLM's parallel state configuration assigned both workers to the same EP group.

The Root Cause: A Gap Between Intention and Implementation

The message reveals a critical misunderstanding about vLLM's parallelism model. The assistant had assumed that --enable-expert-parallel would automatically distribute experts across all available GPUs, with the tensor parallelism size determining how many GPUs share attention weights. But vLLM's behavior is different: the total world size (number of GPUs used) is determined by tensor_parallel_size * data_parallel_size, where data_parallel_size defaults to 1 if not specified. With --tensor-parallel-size 2 and no --data-parallel-size flag, vLLM launched only 2 GPUs.

Within those 2 GPUs, EP was "enabled" in the sense that the code path for expert parallelism was active, but with only 2 GPUs and both assigned EP rank 0, no actual expert distribution occurred. The model weights were simply split across 2 GPUs via tensor parallelism — each GPU holding roughly half the weights. With the TP=4 configuration that worked earlier, each GPU held roughly a quarter of the weights. Doubling the per-GPU weight footprint pushed memory over the limit, causing the OOM.

This is a classic framework-specific gotcha. The concept of "expert parallelism" in the abstract means distributing experts across GPUs. But in vLLM's implementation, expert parallelism is a mode that operates within the existing data-parallel topology — it doesn't automatically expand the topology. To use all 8 GPUs with TP=2 and EP=4, the assistant would have needed to specify --tensor-parallel-size 2 --data-parallel-size 4 (or an equivalent EP-specific parameter), which would have set world_size=8.

The Thinking Process Visible in the Message

What makes this message particularly valuable is the visible reasoning process. The assistant doesn't just report the OOM and move on — it pauses to diagnose. The structure of the reasoning is:

  1. Observation: The launch failed with OOM.
  2. Hypothesis: The problem might be that only 2 workers were launched, not 8.
  3. Evidence gathering: Check the vLLM logs for world_size and EP rank assignments.
  4. Confirmation: world_size=2, EP rank=0 for both workers. This is textbook debugging methodology. The assistant resists the temptation to simply try a different configuration (TP=4 with EP=2, or TP=1 with EP=8) and instead seeks to understand why the current configuration failed. This understanding is essential for choosing the right next step — without it, the assistant might try TP=1+EP=8 and encounter the same issue (vLLM defaulting to world_size=1).

Input Knowledge Required

To fully understand this message, the reader needs:

Output Knowledge Created

This message creates several pieces of actionable knowledge:

  1. vLLM's --enable-expert-parallel does not automatically use all GPUs. The total number of GPUs must be explicitly or implicitly specified through the parallelism configuration.
  2. EP rank 0 for all workers means no expert distribution occurred. The model loaded with TP=2 but without actual expert parallelism, causing OOM from the increased per-GPU memory pressure.
  3. The diagnostic approach is validated. Checking the parallel state logs (parallel_state.py) is the correct way to understand vLLM's actual parallelism topology.
  4. The next steps are constrained. TP=6 is impossible (KV head divisibility), TP=8 is impossible (FP8 quantization alignment), and TP=2+EP needs explicit data_parallel_size to work. The remaining options are TP=4+EP=2, TP=1+EP=8, or sticking with TP=4.

Conclusion

Message [msg 2312] is a masterclass in diagnostic reasoning during complex ML infrastructure work. It demonstrates the critical gap between theoretical parallelism strategies and their implementation in specific frameworks — a gap that can only be bridged by careful log analysis and a willingness to challenge one's own assumptions. The assistant's methodical approach — observe, hypothesize, gather evidence, confirm — turned a frustrating OOM error into a deeper understanding of vLLM's parallelism model, paving the way for the correct configuration in the next attempt.