The TP=2, EP=4 Launch: Navigating Parallelism Constraints for MiniMax-M2.5 on Blackwell GPUs

Introduction

In the high-stakes world of large language model deployment, the choice of parallelism strategy can make the difference between a model that hums along efficiently and one that crashes on startup or leaves expensive GPU hardware idle. Message [msg 2310] captures a pivotal moment in an extended benchmarking session where the assistant, responding to a user's query about alternative parallelism configurations, launches the MiniMax-M2.5 model with a carefully reasoned TP=2, EP=4 setup on an 8-GPU Blackwell machine. This single bash command — deceptively simple in appearance — represents the culmination of a deep analysis of tensor parallelism (TP), expert parallelism (EP), FP8 block quantization constraints, and hardware topology. The message is a microcosm of the engineering trade-offs that define modern LLM serving.

The Message

The assistant executes the following command over SSH:

ssh root@10.1.230.174 "rm -f /dev/shm/psm_* /dev/shm/sem.mp-* /dev/shm/*vllm* /dev/shm/*nccl* 2>/dev/null; nohup env NCCL_PROTO=LL NCCL_P2P_LEVEL=SYS HF_HOME=/shared/huggingface TRANSFORMERS_CACHE=/shared/huggingface/hub /root/ml-env/bin/python3 -m vllm.entrypoints.openai.api_server --model /shared/minimax-m2.5 --tensor-parallel-size 2 --enable-expert-parallel --trust-remote-code --max-model-len 131072 --max-num-seqs 256 --gpu-memory-utilization 0.95 --port 8000 --disable-log-requests --tool-call-parser minimax_m2 --reasoning-parser minimax_m2 --enable-auto-tool-choice > /tmp/vllm_minimax_tp2ep.log 2>&1 &
echo PID=\$!"

The command does several things in sequence: it cleans up stale shared memory artifacts from previous runs, sets NCCL environment variables for inter-GPU communication, and launches the vLLM API server with a specific set of parallelism and model configuration flags.

Why This Message Was Written: The Reasoning and Motivation

The immediate trigger for this message was the user's question in [msg 2303]: "can we do tp/ep or tp6?" This came after a failed attempt to run the MiniMax-M2.5 model with TP=8 ([msg 2297]), which crashed with a ValueError about FP8 block quantization alignment. The root cause, identified in [msg 2299], was that the MoE intermediate size of 1536 divided by 8 TP shards gives 192, which is not divisible by the FP8 quantization block size of 128. TP=4 worked because 1536 / 4 = 384 = 3 × 128.

The user's question opened up two avenues: TP=6 (which would avoid the FP8 alignment issue since 1536 / 6 = 256, divisible by 128) and TP+EP combinations (which would distribute experts across GPUs without tensor-sharding them, sidestepping the alignment constraint entirely). The assistant recognized this as an opportunity to explore whether the remaining 4 idle GPUs (GPUs 4-7 were sitting unused with the TP=4 configuration) could be put to work.

In the messages immediately preceding the subject ([msg 2304] through [msg 2309]), the assistant conducted a thorough feasibility analysis. It checked vLLM's command-line flags for expert parallelism support, inspected the model's source code for EP-related logic, examined the distributed communication infrastructure, and performed the divisibility arithmetic for every plausible TP/EP combination. This analytical chain is the direct intellectual predecessor of the command in [msg 2310].

How Decisions Were Made

The decision to use TP=2, EP=4 rather than any other combination was the result of systematic elimination:

TP=6 was ruled out because the model has 8 key-value attention heads (num_key_value_heads=8), and 8 is not divisible by 6. Tensor parallelism requires the number of attention heads to be evenly divisible by the TP size, so TP=6 would fail at the attention layer before even reaching the expert layers.

TP=8 was already known to fail due to the FP8 block quantization issue on the expert intermediate dimension.

TP=4, EP=2 was viable but would leave the attention allreduce overhead on 4 GPUs, which the assistant judged as suboptimal given that the attention layers are the primary source of communication overhead in this model.

TP=1, EP=8 was the most aggressive option — no tensor parallelism at all, with experts distributed across all 8 GPUs. This would minimize allreduce communication for attention (none needed) but would place the full attention computation on a single GPU, potentially creating a bottleneck for the 3072-dimensional hidden states.

TP=2, EP=4 emerged as the "Goldilocks" choice. With TP=2, the attention computation is split across two GPUs, halving the per-GPU memory and compute for the dense layers. With EP=4, the 256 experts are distributed across 4 groups of 2 GPUs each, meaning each group holds 64 experts in full (not sharded). This configuration uses all 8 GPUs, avoids the FP8 alignment problem entirely (experts are not tensor-sharded when EP is enabled), and keeps the allreduce groups small (just 2 GPUs per TP group), which is critical on a PCIe-interconnected system where allreduce bandwidth is the primary bottleneck.

The assistant also made a deliberate choice about NCCL configuration. The environment variables NCCL_PROTO=LL and NCCL_P2P_LEVEL=SYS were carried forward from earlier successful runs. NCCL_PROTO=LL selects the Low Latency protocol for NVIDIA's collective communications library, which is appropriate for the relatively small message sizes in attention allreduce. NCCL_P2P_LEVEL=SYS forces peer-to-peer communication through the system fabric (PCIe) rather than attempting NVLink, which is correct for these RTX PRO 6000 Blackwell GPUs that lack NVLink bridges.

Assumptions Made

The command embeds several assumptions, some explicit and some implicit:

That EP is well-supported for the MiniMax-M2.5 architecture in this version of vLLM. The assistant verified that --enable-expert-parallel exists as a flag and that the vLLM distributed infrastructure references EP groups, but it did not run a dry test or inspect the model's weight-loading code path for EP-specific logic. The MiniMax-M2.5 model uses a Grouped-Query Attention (GQA) architecture with Mixture-of-Experts (MoE) layers — if the vLLM model implementation (minimax_m2.py) does not properly handle expert parallelism for weight loading or routing, the server could crash at startup or produce incorrect outputs.

That cleaning /dev/shm artifacts is sufficient to avoid inter-process conflicts. The command removes shared memory files from previous vLLM runs (psm_*, sem.mp-*, *vllm*, *nccl*). This assumes that all stale state is in these well-known locations and that no other processes are using shared memory segments that might conflict.

That the NCCL environment variables are optimal for this configuration. The NCCL_PROTO=LL setting was tuned for the TP=4 configuration. With TP=2 and EP=4, the communication patterns change — the allreduce groups are smaller (2 GPUs instead of 4), but there is now additional all-to-all communication for expert routing. The Low Latency protocol might not be optimal for the expert all-to-all transfers, which involve larger payloads.

That the model can be loaded within the 95% GPU memory utilization limit. With EP distributing experts, each GPU holds fewer total parameters (roughly 1/4 of the experts plus 1/2 of the dense layers), so memory pressure should be lower than the TP=4 configuration. But the KV cache is still replicated across TP groups, and the 131072-token context window requires substantial cache memory.

That the user's intent is to maximize throughput by utilizing all 8 GPUs. The assistant implicitly assumes that the user wants to put the idle GPUs to work rather than keeping them available for a separate workload. This is a reasonable inference from the question "can we do tp/ep or tp6?" but it represents a value judgment about resource allocation.

Potential Mistakes and Incorrect Assumptions

The most significant risk in this message is the assumption that EP is fully functional for the MiniMax-M2.5 model in this vLLM build. The assistant's investigation in [msg 2305] and [msg 2306] found references to EP infrastructure in the distributed utilities and the model's weight-mapping code, but it did not verify that the expert routing logic, the all-to-all communication backend, and the weight loading path all correctly handle the EP=4 case. If the model implementation has a bug in its EP support — for example, if it incorrectly indexes experts or misroutes tokens — the server might start successfully but produce garbage output, a failure mode that is much harder to diagnose than a crash.

Another subtle issue is the interaction between --enable-expert-parallel and --tensor-parallel-size 2. When EP is enabled, vLLM typically creates TP groups within each EP group. With 8 GPUs, TP=2 and EP=4, the expected topology is 4 EP groups of 2 GPUs each. But the command does not specify --expert-parallel-size explicitly — it only enables EP with a boolean flag. The assistant assumed that vLLM would automatically compute EP=4 from the available GPUs (8) divided by TP=2. If vLLM instead defaults to EP=1 (no expert parallelism) or EP=2 (dividing GPUs differently), the actual configuration could be different from intended.

The shared memory cleanup (rm -f /dev/shm/psm_* ...) is aggressive and could interfere with other processes running on the machine. If another vLLM instance or any NCCL-using process is running concurrently (unlikely given the systemd service was stopped, but possible), this command would destroy their communication channels.

Input Knowledge Required

To understand this message fully, one needs knowledge of several domains:

Tensor Parallelism (TP): The splitting of individual weight matrices across GPUs, so that each GPU holds a slice of every layer. TP requires divisibility constraints on model dimensions (hidden size, attention heads, intermediate size) by the TP count.

Expert Parallelism (EP): The distribution of MoE experts across GPUs, where each GPU holds a complete subset of experts. EP avoids the divisibility constraints of TP for expert weights but introduces all-to-all communication for token routing.

FP8 Block Quantization: A compression scheme where weights are stored in 8-bit floating point format with per-block scaling factors. The block size (here 128×128) creates alignment constraints when sharding — each shard's dimensions must be a multiple of the block size.

NCCL Protocols: NVIDIA's Collective Communications Library supports multiple protocols (LL for Low Latency, LL128 for higher bandwidth, Simple for fallback). The choice affects throughput and latency for allreduce operations.

vLLM Architecture: Understanding the relationship between TP, EP, pipeline parallelism, and data parallelism in vLLM's distributed execution model is essential. The --enable-expert-parallel flag changes how MoE layers are partitioned across the GPU topology.

MiniMax-M2.5 Model Architecture: The model uses GQA with 48 query heads and 8 key-value heads, an MoE with 256 experts and intermediate size 1536, and FP8 quantization. These specific numbers drive the divisibility analysis.

Output Knowledge Created

This message creates several concrete outputs:

A running vLLM API server on port 8000 with the TP=2, EP=4 configuration. The server logs are written to /tmp/vllm_minimax_tp2ep.log. The process ID (232203) is captured for monitoring.

A testable hypothesis about whether TP=2, EP=4 is a viable and performant configuration for MiniMax-M2.5 on 8 Blackwell GPUs. The subsequent messages in the conversation will reveal whether the server starts successfully, how long it takes to load, and what throughput it achieves.

A benchmark data point for the community about the interaction between FP8 quantization, MoE expert parallelism, and PCIe-bound Blackwell GPUs. If successful, this configuration demonstrates that EP can circumvent the FP8 alignment limitations that make TP=8 impossible for this model.

An updated system state where all 8 GPUs are utilized (compared to the previous TP=4 configuration that left 4 GPUs idle), and the shared memory segments have been cleaned for a fresh start.

The Thinking Process Visible in the Reasoning

The chain of reasoning leading to this message is a textbook example of systematic constraint satisfaction. The assistant enumerated all possible TP/EP combinations for 8 GPUs, checked each against three constraints — attention head divisibility, FP8 block alignment, and total GPU count — and then ranked the survivors by expected performance.

The reasoning in [msg 2309] is particularly revealing. The assistant explicitly notes that "with EP, the 128-block alignment constraint on intermediate_size / TP doesn't apply to experts (they're not TP-sharded)." This insight is the key that unlocks EP as a solution. The FP8 alignment problem that killed TP=8 is not a fundamental hardware limitation — it's an artifact of how TP interacts with block quantization. By switching to EP for the expert layers, the assistant sidesteps the constraint entirely.

The choice of TP=2, EP=4 over TP=1, EP=8 also shows nuanced thinking. TP=1 would eliminate allreduce entirely for attention, but it would also place the full attention computation on a single GPU, potentially creating a memory bandwidth bottleneck for the 3072-dimensional hidden states. TP=2 provides a balance: the attention allreduce is small (just 2 GPUs), and the per-GPU attention compute is halved. On a PCIe system where allreduce bandwidth is scarce but compute is abundant (each GPU has 96GB of VRAM and substantial compute capacity), this trade-off makes sense.

The command's structure also reveals operational thinking. The cleanup of /dev/shm artifacts is a learned behavior from earlier debugging sessions — shared memory leaks from crashed NCCL processes can cause mysterious failures in subsequent runs. The use of nohup and output redirection to a log file shows an expectation that the server will run as a background process, to be monitored later. The echo PID=$! at the end provides a capture of the process ID for potential debugging or cleanup.

Conclusion

Message [msg 2310] is far more than a simple bash command. It is the executable conclusion of a multi-step reasoning process that navigated hardware constraints, software capabilities, and mathematical divisibility rules to arrive at a specific parallelism configuration. The command embodies the assistant's understanding of tensor parallelism's divisibility requirements, expert parallelism's freedom from those constraints, FP8 quantization's block alignment rules, NCCL protocol selection, and vLLM's distributed execution model. Whether the TP=2, EP=4 configuration succeeds or fails, the reasoning behind it represents a master class in the art of model deployment on modern multi-GPU systems.