The TP=8 + EP Gamble: Navigating vLLM's Parallelism Constraints for MiniMax-M2.5 on 8 Blackwell GPUs

Introduction

In the span of a single bash command, an entire hypothesis about model parallelism was put to the test. Message <msg id=2327> in this opencode session captures the moment when the assistant, after an exhaustive exploration of vLLM's source code, launched the MiniMax-M2.5 model with --tensor-parallel-size 8 --enable-expert-parallel on an 8-GPU Blackwell system. This seemingly simple command represented the culmination of a deep investigation into the interplay between tensor parallelism (TP), expert parallelism (EP), FP8 quantization block alignment, and vLLM's internal architecture. The message is a snapshot of a critical decision point: the pivot from a working but suboptimal TP=4 configuration to an ambitious TP=8+EP setup that promised to harness all 8 GPUs for maximum throughput.

The Reasoning: Why This Message Was Written

The message was born from a chain of failures and discoveries. The assistant had been benchmarking MiniMax-M2.5, a 230B-parameter FP8 Mixture-of-Experts model, on a machine with 8 NVIDIA RTX PRO 6000 Blackwell GPUs (96 GB each). The initial TP=4 configuration worked well—it loaded in 75 seconds and achieved strong throughput—but it left 4 GPUs completely idle. The natural question was: could all 8 GPUs be utilized?

The path to TP=8 was blocked by a seemingly hard constraint. MiniMax-M2.5 uses FP8 block quantization with a block size of 128×128. When tensor parallelism shards the expert intermediate weights, the sharded dimension (intermediate_size / TP) must be a multiple of 128. With intermediate_size = 1536, TP=8 gives 1536 / 8 = 192, which is 192 / 128 = 1.5—not evenly divisible. This caused vLLM to raise a ValueError about block quantization alignment. TP=6 was also ruled out because the 8 key-value heads aren't divisible by 6.

The assistant then explored expert parallelism as an alternative. The key insight, uncovered through extensive code spelunking in messages <msg id=2304> through <msg id=2325>, was that when EP is enabled, vLLM's MoE layers use EP for expert distribution and TP=1 for MoE weight sharding. In other words, expert weights are not tensor-sharded at all when EP is active—each EP rank holds a complete set of a subset of experts. This means the FP8 block alignment constraint on intermediate_size / TP simply does not apply to the MoE layers when EP is on. The constraint only applies to the attention and dense layers, which have different dimension sizes.

This discovery opened up a new possibility: TP=8 with EP enabled would give TP=8 for attention layers (which have compatible dimensions) and EP=8 for MoE layers (distributing the 256 experts across 8 GPUs without TP sharding). The FP8 alignment issue that blocked TP=8 without EP would be sidestepped entirely.

The Decision Process: How the Command Was Constructed

The command in <msg id=2327> reflects several deliberate choices. First, the cleanup of /dev/shm/psm_*, /dev/shm/sem.mp-*, and /dev/shm/*vllm* files was necessary because previous failed launches (like the TP=2+EP attempt in <msg id=2310> that OOM'd) left stale shared memory segments that could interfere with NCCL initialization.

The environment variables NCCL_PROTO=LL and NCCL_P2P_LEVEL=SYS were carried over from earlier successful runs. These configure NCCL to use the Low Latency protocol and system-level P2P (PCIe), which is appropriate for the 8-GPU system where GPUs are connected via PCIe rather than NVLink.

The model path /shared/minimax-m2.5 points to a local copy of the model, avoiding repeated downloads. The --trust-remote-code flag was necessary because MiniMax uses custom model code. The --max-model-len 131072 matches the model's native context length. The --gpu-memory-utilization 0.95 leaves 5% headroom for CUDA kernels and temporary allocations—a conservative choice given that the model is 230 GB and 8×96 GB = 768 GB total, so memory pressure should be manageable.

The --tool-call-parser and --reasoning-parser flags were specific to MiniMax's architecture, enabling function calling and reasoning capabilities. These had been discovered and tested in earlier segments of the conversation.

Assumptions Made

The command rested on several critical assumptions:

Assumption 1: EP would use all 8 GPUs. The previous TP=2+EP attempt in <msg id=2310> had failed because vLLM launched only 2 workers (world_size=2), with EP reusing the TP group. The assistant assumed that with TP=8, EP would naturally scale to 8 GPUs. This was supported by code inspection showing that when EP is enabled, the MoE layer uses ep_size independently of tp_size.

Assumption 2: The FP8 alignment check would not fire for MoE layers with EP. This was the core hypothesis. The assistant had traced the alignment check in fp8_utils.py and found it only triggers when is_tp_split = True. With EP, MoE weights are not TP-split, so the check should be bypassed.

Assumption 3: Attention layer dimensions are compatible with TP=8. The assistant had verified that hidden_size=3072 divided by 8 gives 384, num_attention_heads=48 divided by 8 gives 6, and num_key_value_heads=8 divided by 8 gives 1—all clean integers. This was confirmed in <msg id=2309>.

Assumption 4: 230 GB fits comfortably across 8 GPUs. With 8×96 GB = 768 GB total, and the model at 230 GB, the memory utilization would be roughly 30% of capacity, well within the 95% limit.

Assumption 5: The NCCL configuration from earlier runs would work for 8 GPUs. The NCCL_PROTO=LL and NCCL_P2P_LEVEL=SYS settings had been tuned for 4-GPU PCIe communication. Whether they would scale efficiently to 8 GPUs was an open question.

Potential Mistakes and Incorrect Assumptions

The most significant risk was Assumption 1. The assistant's code inspection in <msg id=2325> revealed that vLLM's documentation describes a scenario where "TP=2, EP=True" gives TP_moe={1, 0}, EP={2, 0/1}—meaning EP=2 and TP_moe=1. But this was documentation for a specific case, and the assistant had not verified that the same logic would produce EP=8 with TP=8. There was a real possibility that vLLM would still constrain EP size to equal TP size (as it did with TP=2+EP), resulting in EP=8 but only using 8 GPUs if the world size was properly set.

A subtler issue: the assistant had not verified that the attention layers' FP8 block quantization would be compatible with TP=8. While the dimension sizes divide evenly, the FP8 block alignment check in fp8_utils.py applies to all linear layers, not just MoE ones. The attention layers use qkv_proj and o_proj with hidden_size=3072. With TP=8, each shard gets 3072/8=384 hidden units. The FP8 block size is 128×128, so 384 % 128 = 0—this should work. But the assistant had not explicitly verified this path.

Another assumption worth questioning: the assistant assumed that the TP=8+EP configuration would actually improve throughput over TP=4. While using all 8 GPUs reduces per-GPU memory pressure, it also increases allreduce overhead for attention layers (which still use TP=8). The PCIe allreduce bottleneck that plagued the NVFP4 Kimi-K2.5 model (identified in <msg id=2318> of the segment summary) would also affect MiniMax's attention layers. The assistant was implicitly betting that the EP benefit for MoE layers would outweigh the TP overhead for attention layers.

Input Knowledge Required

To understand this message, one needs:

  1. vLLM parallelism architecture: Understanding the difference between tensor parallelism (sharding individual weight matrices across GPUs) and expert parallelism (distributing entire experts across GPUs). Knowledge that vLLM's EP reuses the TP group by default, and that EP size can differ from TP size for MoE layers.
  2. FP8 block quantization: Understanding that FP8 weights are stored in blocks (128×128) and that tensor parallelism must respect block boundaries. A sharded dimension that isn't a multiple of the block size causes an error.
  3. MiniMax-M2.5 architecture: Knowledge that it's a 230B MoE model with 256 experts, intermediate_size=1536, hidden_size=3072, 48 attention heads, 8 KV heads, and FP8 quantization.
  4. Blackwell GPU memory: The RTX PRO 6000 has 96 GB of VRAM. Eight GPUs provide 768 GB total.
  5. NCCL configuration: NCCL_PROTO=LL selects the Low Latency protocol, and NCCL_P2P_LEVEL=SYS restricts P2P communication to system (PCIe) paths, avoiding NVLink. These are appropriate for a PCIe-connected multi-GPU system.
  6. Linux shared memory: The /dev/shm/ cleanup targets NCCL shared memory segments (psm_*, sem.mp-*) and vLLM's own shared memory (*vllm*, *nccl*). Stale segments from crashed processes can prevent new NCCL initialization.

Output Knowledge Created

The immediate output of this message was the launch of a vLLM server process with PID 233186. The log file /tmp/vllm_minimax_tp8ep.log would contain the startup sequence, model loading progress, and any errors encountered.

The broader output was a test of the hypothesis that TP=8+EP would work for MiniMax-M2.5. If successful, it would unlock the full 8-GPU throughput potential. If it failed, the failure mode would reveal whether the FP8 alignment issue was truly bypassed by EP, or whether other constraints (memory, NCCL, attention layer quantization) would block the configuration.

This message also created knowledge about vLLM's EP behavior. The assistant had spent significant effort reading vLLM source code to understand how EP interacts with TP. The command in <msg id=2327> was the experimental validation of that code analysis. Whether the experiment succeeded or failed, the results would inform future model deployment decisions.

The Thinking Process

The assistant's thinking process, visible across the preceding messages, reveals a methodical approach to debugging parallelism constraints. The chain of reasoning went:

  1. Problem identification: TP=8 fails with FP8 alignment error on intermediate_size / 8 = 192 % 128 != 0.
  2. Constraint mapping: TP=6 fails because KV heads (8) aren't divisible by 6. TP=4 works but uses only 4 GPUs. TP=2+EP OOMs because vLLM uses world_size=2 (only 2 GPUs).
  3. Code exploration: The assistant dives into vLLM's source code, examining fused_moe/layer.py, fused_moe/config.py, fp8_utils.py, and modelopt.py to understand the exact conditions that trigger the FP8 alignment check and how EP affects MoE weight sharding.
  4. Discovery: The documentation in config.py reveals that with EP enabled, MoE layers use TP_moe=1 (no tensor sharding of expert weights). This means the FP8 alignment check on intermediate_size / TP doesn't apply to MoE layers when EP is on.
  5. Hypothesis formation: TP=8 with EP enabled should work because attention layers handle TP=8 fine (dimensions divide evenly), and MoE layers use EP=8 without TP sharding (avoiding the FP8 alignment issue).
  6. Experimental design: Launch vLLM with TP=8, EP enabled, using the same NCCL configuration as previous successful runs. Clean up shared memory to avoid interference from failed previous attempts. The assistant's thinking shows a strong preference for understanding the why before attempting the what. Rather than blindly trying different TP sizes, the assistant read source code to understand the exact constraints, then designed an experiment to test the hypothesis. This is characteristic of effective systems engineering: diagnose the root cause before applying the fix.

Conclusion

Message <msg id=2327> is a pivotal moment in the session—a carefully constructed experiment that tests a hypothesis born from deep code analysis. It represents the transition from investigation to action, from understanding vLLM's parallelism internals to deploying a model configuration that could potentially double the throughput of the previous setup. Whether the command succeeded or failed (the result is not visible in this message alone), the reasoning behind it demonstrates a systematic approach to navigating the complex constraints of large model deployment on multi-GPU systems. The message encapsulates the essence of the engineering challenge: balancing model architecture constraints, quantization requirements, parallelism strategies, and hardware capabilities to find the optimal deployment configuration.