Breaking the FP8 Barrier: How Expert Parallelism Unlocked All 8 GPUs for MiniMax-M2.5

In the sprawling, high-stakes world of large language model deployment, a single line of output can represent hours of debugging, pages of source code analysis, and a cascade of failed experiments. Such is the case with message [msg 2329] in this opencode session, where the assistant reports simply: "TP=8 + EP works! Ready in ~90s." Behind this terse announcement lies one of the most technically nuanced breakthroughs in the entire conversation — the successful deployment of the MiniMax-M2.5 FP8 model across all 8 NVIDIA RTX PRO 6000 Blackwell GPUs using a combination of Tensor Parallelism (TP) and Expert Parallelism (EP) that had previously seemed impossible due to FP8 quantization alignment constraints.

The Problem: Stuck at 4 GPUs

The session had been on a rollercoaster of model pivots and hardware bottlenecks. After struggling with the 1T-parameter GLM-5 and NVFP4 Kimi-K2.5 models, the team had pivoted to MiniMax-M2.5, a 230B-parameter FP8 model using a Grouped-Query Attention (GQA) architecture with 256 experts. Initial results with TP=4 were promising — 84 tok/s single-stream and over 2,500 tok/s at high concurrency — but this configuration left 4 of the 8 available GPUs completely idle. The hardware investment was being half-wasted.

The obvious next step was TP=8, using all 8 GPUs. But when the assistant tried this in earlier messages, it crashed with a cryptic error about FP8 block quantization alignment. The root cause, uncovered through painstaking source code analysis in messages [msg 2305] through [msg 2328], was that TP=8 required sharding each expert's intermediate dimension (intermediate_size = 1536) across 8 GPUs, giving 1536 / 8 = 192 per partition. The FP8 quantization uses a block size of [128, 128], and 192 is not divisible by 128. This was a hard constraint in vLLM's fp8_utils.py — the check at line 1444 explicitly raises a ValueError when the sharded dimension isn't divisible by the quantization block size.

The Insight: EP Changes the Rules

The breakthrough came when the assistant dug into vLLM's fused MoE layer implementation. In messages [msg 2318] through [msg 2325], the assistant traced through the source code to understand how Expert Parallelism interacts with Tensor Parallelism. The critical discovery was in FusedMoEParallelConfig:

When EP is enabled, the MoE layers use EP for expert distribution and TP=1 for MoE. The expert weights are NOT tensor-sharded when EP is on — each EP rank holds a subset of experts in full.

This means the FP8 alignment constraint on intermediate_size / TP simply doesn't apply to expert weights when EP is enabled. The attention and dense layers still use TP=8, but the MoE experts are distributed whole across the 8 GPUs. Each GPU holds 256 / 8 = 32 complete experts, each with its full 1536 intermediate dimension — no sharding, no alignment issue.

This was a subtle architectural point that required reading vLLM's internal documentation (lines 1020-1090 of config.py) to fully understand. The assistant even quoted the comment: "When TP = 2, EP = True: TP_moe = {1, 0}, EP = {2, 0/1}" — confirming that EP replaces TP for MoE layers.

The Message: A Milestone Announced

With this understanding, the assistant launched TP=8 with --enable-expert-parallel in message [msg 2327], and by message [msg 2329] the result was in:

TP=8 + EP works! Ready in ~90s. Let me verify GPU usage and run a quick test:

`` 0, 96601 MiB, 97887 MiB 1, 96601 MiB, 97887 MiB 2, 96601 MiB, 97887 MiB 3, 96601 MiB, 97887 MiB 4, 96599 MiB, 97887 MiB 5, 96599 MiB, 97887 MiB 6, 96599 MiB, 97887 MiB 7, 96599 MiB, 97887 MiB ``

The numbers tell a remarkable story. Each of the 8 GPUs is using approximately 96.6 GiB out of 97.9 GiB total — a staggering 98.7% utilization. The memory distribution is nearly perfectly balanced: GPUs 0-3 show 96,601 MiB, GPUs 4-7 show 96,599 MiB. This 2 MiB difference across 8 GPUs is essentially noise, indicating that vLLM's EP distribution algorithm achieved near-ideal balance.

The 90-second load time is also noteworthy. For a 230B-parameter model spread across 8 GPUs with expert parallelism, loading in 90 seconds demonstrates efficient I/O and weight distribution. This contrasts sharply with the 36-minute load time later seen for the INT4 Kimi-K2.5 model (547GB), highlighting how model size and quantization format dramatically affect deployment speed.

What This Message Reveals About the Thinking Process

The brevity of message [msg 2329] belies the depth of reasoning that preceded it. The assistant's thinking process, visible across the preceding messages, shows several hallmarks of expert debugging:

Systematic hypothesis testing: Each failed attempt (TP=8 without EP, TP=2+EP) generated a new hypothesis about the root cause, which was then tested against the source code.
Reading the source, not the docs: The assistant went directly to vLLM's Python source files — fused_moe/layer.py, fused_moe/config.py, fp8_utils.py, modelopt.py — to understand the actual constraints rather than relying on documentation or error messages alone.
Understanding the abstraction layers: The key insight required understanding that TP and EP operate at different levels of the model architecture — TP shards individual tensor operations, while EP distributes entire experts. These are not interchangeable parallelism strategies.
Mathematical precision: The assistant checked exact divisibility: intermediate_size / TP = 1536 / 8 = 192, 192 % 128 = 64 ≠ 0. This mathematical rigor was essential to identifying the real constraint.

Assumptions and Their Validation

The message validates several assumptions the assistant made:

That EP would properly separate from TP for MoE layers: Confirmed. The model loaded and distributed memory evenly.
That the FP8 alignment check was the only barrier to TP=8: Confirmed. With EP bypassing that check, TP=8 worked.
That 8 GPUs would have sufficient aggregate memory: The model is ~230GB, and 8 × 96GB = 768GB total, so this was never the concern — the issue was always about shard alignment, not capacity.
That vLLM's EP implementation correctly handles the MiniMax architecture: Confirmed. The model loaded without architecture-specific errors. One earlier incorrect assumption was that --enable-expert-parallel with TP=2 would use more than 2 GPUs. In message [msg 2312], the assistant discovered that vLLM's EP group equals the TP group — with TP=2, EP=2, only 2 workers are launched. This led to OOM because 230GB / 2 GPUs = 115GB per GPU, exceeding the 96GB limit. The correct approach was TP=8 + EP, which launches 8 workers and distributes both attention (via TP) and experts (via EP) across all GPUs.

The Broader Significance

This message represents a turning point in the session. With TP=8+EP working, the MiniMax-M2.5 model could now leverage all 8 Blackwell GPUs. The subsequent benchmarks (described in the segment summary) showed throughput reaching nearly 4,000 tok/s at high concurrency — a dramatic improvement over the ~2,500 tok/s with TP=4. This validated the entire line of reasoning: that GQA architectures with smaller active parameters, combined with expert parallelism, are vastly superior on PCIe-bound multi-GPU systems compared to the MLA-based models (like Kimi-K2.5 NVFP4) that were bottlenecked by allreduce communication.

The message also demonstrates a crucial lesson in ML infrastructure engineering: understanding the interaction between parallelism strategies is often more important than understanding individual strategies. TP and EP are not independent knobs to be tuned separately — their interaction determines what configurations are even valid. The FP8 alignment constraint that blocked TP=8 was silently eliminated by enabling EP, because EP changes the fundamental geometry of how weights are distributed.

Conclusion

Message [msg 2329] is a classic example of a "small message, big result" moment in systems engineering. Its two lines — the success announcement and the GPU memory table — encode hours of debugging, pages of source code analysis, and a deep understanding of distributed model serving architecture. The near-perfect memory balance across 8 GPUs (within 2 MiB) is a testament to vLLM's EP implementation and the correctness of the assistant's reasoning. For anyone deploying large MoE models on multi-GPU systems, this message encapsulates a critical insight: when FP8 quantization blocks your path to higher tensor parallelism, expert parallelism might be the key that unlocks the door.