The Epiphany: Rethinking Tensor and Expert Parallelism for MoE Models on Blackwell GPUs

In the high-stakes world of large language model deployment, few moments are as satisfying as the sudden flash of insight that reframes an entire problem. Message 2314 of this opencode session captures exactly such a moment — a brief but pivotal exchange where the assistant, after a series of frustrating failures with tensor parallelism (TP) and expert parallelism (EP) configurations for the MiniMax-M2.5 model, has a critical realization that could unlock the full potential of an 8-GPU Blackwell system.

The Context: A Model Deployment Puzzle

The conversation leading up to this message had been a whirlwind tour of model benchmarking and hardware-boundary exploration. The team had already pivoted from NVFP4 Kimi-K2.5 (hampered by PCIe allreduce bottlenecks for its 61-layer MLA architecture) to MiniMax-M2.5, a 230B-parameter FP8 model with Grouped Query Attention (GQA) and a Mixture-of-Experts (MoE) architecture. The MiniMax had proven spectacularly fast at TP=4, achieving 84 tok/s single-stream and over 2,500 tok/s at high concurrency. But the user wanted more — specifically, they asked if TP=8 or TP+EP combinations could work ([msg 2303]).

The assistant had systematically explored the options. TP=8 failed because the FP8 block quantization (128×128 blocks) required the sharded intermediate size to be divisible by 128, and 1536 / 8 = 192 failed that check ([msg 2299]). TP=6 was ruled out because 8 key-value heads aren't divisible by 6 ([msg 2309]). A TP=2 + EP attempt crashed with an OOM error because vLLM's EP implementation tied EP size to TP size, resulting in only 2 GPUs being used for a 230GB model — far exceeding the 96GB per-GPU memory limit (<msg id=2311-2313>).

The Message: A Moment of Reassessment

Message 2314 opens with the assistant absorbing a hard-learned lesson:

No separate --expert-parallel-size. In vLLM, EP size = TP size when EP is enabled. So to get 8 GPUs with EP, we need TP=8 + EP, but TP=8 fails the FP8 alignment.

This is the voice of someone who has just been burned by an incorrect assumption. The assistant had assumed that --enable-expert-parallel would somehow distribute work across all 8 GPUs independently of TP size. The reality was more constrained: vLLM's implementation uses the same process group for both TP and EP, meaning EP size is always equal to TP size. To use all 8 GPUs with EP, you must set TP=8 — but TP=8 was already proven incompatible due to FP8 quantization alignment.

The assistant then considers a fallback:

Let me think about this differently. What about TP=4, EP enabled? That would give TP=4, EP=4, using all 4 GPUs with experts split across them. But that's the same 4 GPUs we're already using...

There's a palpable sense of frustration here. TP=4 with EP would use the same 4 GPUs they're already running on, just with experts distributed instead of replicated. It wouldn't unlock the 4 idle GPUs (GPUs 4-7, sitting at 33W each). The assistant is stuck — every path seems to lead back to the same 4-GPU configuration.

The Breakthrough: Questioning the Assumption

Then comes the pivot — the moment of genuine insight:

Actually wait — let me reread. In vLLM with EP, the EP group equals the TP group. So --tensor-parallel-size 8 --enable-expert-parallel would give TP=8 for attention and EP=8 for MoE. The FP8 alignment check on intermediate_size / TP might not apply with EP because experts are distributed whole, not sharded.

This is the key realization. The assistant had been operating under an assumption that the FP8 block quantization alignment error would apply regardless of EP. But EP fundamentally changes how weights are distributed: with tensor parallelism, each expert's weights are sharded across TP GPUs (so each GPU gets intermediate_size / TP columns). With expert parallelism, each GPU holds entire experts — no sharding across the intermediate dimension. The FP8 alignment constraint on intermediate_size / TP only applies when weights are actually being tensor-sharded.

If EP distributes experts whole, then the FP8 alignment check on the intermediate dimension simply doesn't apply to the expert weights. The attention layers would still be TP-sharded, but those use different dimensions (hidden_size, head counts) which are divisible by 8.

The assistant immediately recognizes the need to verify this hypothesis:

Let me check that more carefully: [bash] grep -n 'output_size.gate.up.*not divisible\|block_n' /root/ml-env/lib/python3.12/site-packages/vllm/model_executor/layers/fused_moe/ -r 2>/dev/null | head -5

This is the hallmark of rigorous engineering: don't just assume your insight is correct — go verify it against the source code.

Input Knowledge Required

To fully understand this message, one needs several layers of knowledge:

Hardware constraints: The system has 8 NVIDIA RTX PRO 6000 Blackwell GPUs (each with 97.8GB VRAM) connected via PCIe. PCIe allreduce is a known bottleneck for models requiring heavy cross-GPU communication, particularly for MLA architectures with many attention layers.

Model architecture: MiniMax-M2.5 is a 230B-parameter MoE model with 10B active parameters per token. It uses FP8 block quantization with 128×128 blocks, GQA with 48 attention heads and 8 KV heads, and a hidden size of 3072. The intermediate (expert feed-forward) size is 1536. It has 256 experts.

vLLM parallelism concepts: Tensor parallelism shards weight matrices across GPUs (each GPU holds a slice of every layer). Expert parallelism distributes different experts to different GPUs (each GPU holds a subset of experts in full). The two can be combined, but vLLM's implementation ties EP group size to TP group size.

FP8 quantization mechanics: Block quantization divides weight matrices into blocks (here 128×128) and stores scaling factors per block. When sharding a matrix across GPUs, the shard boundaries must align with block boundaries — hence the divisibility constraint.

Assumptions and Their Corrections

The message reveals several assumptions that were either corrected or questioned:

Assumption: EP would automatically use all available GPUs. Correction: EP size = TP size in vLLM, so to use 8 GPUs you need TP=8.
Assumption: The FP8 alignment error (intermediate_size / TP must be divisible by 128) applies universally. Questioned: With EP, experts are distributed whole, not sharded, so the alignment constraint may not apply to expert weights.
Assumption: TP=4 + EP would use 8 GPUs. Correction: TP=4 + EP gives TP=4, EP=4 on 4 GPUs — the same as TP=4 without EP in terms of GPU count.
Implicit assumption: The error from TP=8 was in the MoE layers specifically. This is confirmed by the earlier error message: "The output_size of gate's and up's weight = 192 is not divisible by weight quantization block_n = 128" ([msg 2298]).

The Thinking Process

What makes this message remarkable is the visible arc of the assistant's reasoning. It begins in a state of constraint-satisfaction problem-solving: "Given these divisibility rules and vLLM's API, what configurations are valid?" When all paths seem blocked, there's a moment of reconsideration — "Actually wait — let me reread" — where the assistant re-examines its fundamental assumptions.

The key insight is recognizing that the FP8 alignment constraint is not a universal property of the model or hardware, but rather a consequence of how weights are sharded. By changing the distribution strategy (EP instead of TP for experts), the constraint domain changes. This is classic systems thinking: understanding that a constraint applies to a specific operation (tensor sharding) rather than to the model as a whole.

The assistant also demonstrates good engineering judgment by immediately planning to verify the hypothesis against source code rather than blindly trying another launch. After multiple failed attempts (TP=8 crash, TP=6 ruled out, TP=2+EP OOM), a more cautious, verification-first approach is warranted.

Output Knowledge Created

This message creates several valuable pieces of knowledge:

vLLM's EP implementation detail: EP size equals TP size; there is no separate --expert-parallel-size flag. This is non-obvious and not clearly documented in the help output.
A testable hypothesis: TP=8 + EP might work because EP distributes experts whole, bypassing the FP8 alignment constraint on the intermediate dimension.
A verification strategy: Check the fused_moe source code to confirm where the alignment check occurs and whether it's conditional on TP sharding vs EP distribution.
A refined understanding of the problem space: The constraints are not independent — TP divisibility (for attention heads/KV heads), FP8 block alignment (for expert intermediate size), and memory capacity interact in complex ways.

Broader Significance

This message exemplifies a pattern that recurs throughout the entire opencode session: the assistant repeatedly encounters hardware-software boundary issues where model architecture, quantization scheme, parallelism strategy, and hardware topology interact in unexpected ways. Each time, the response is the same — systematic debugging, hypothesis formation, and source code verification.

The specific insight here — that EP changes which divisibility constraints apply — is the kind of deep architectural understanding that separates effective ML engineers from those who merely follow tutorials. It requires understanding not just what vLLM does, but why the error occurs at the level of weight matrix operations.

Whether the hypothesis proves correct (the next message would reveal the answer) is almost beside the point. The value lies in the reasoning process itself: the willingness to question assumptions, the ability to reframe the problem, and the discipline to verify before acting. In the complex world of large-scale model deployment, where documentation often lags behind implementation and hardware constraints multiply, this kind of thinking is the difference between success and endless frustration.