The Arithmetic of Inference: Reasoning Through TP, EP, and FP8 Alignment Constraints

In the middle of a high-stakes benchmarking session on an 8× Blackwell GPU machine, a single question from the user — "can we do tp/ep or tp6?" — triggered a cascade of mental arithmetic, architectural reasoning, and a targeted tool invocation that would reshape the entire deployment strategy. The assistant's response at <msg id=2304> is a masterclass in how systems thinking, quantitative reasoning, and deep knowledge of GPU hardware constraints combine to navigate the treacherous landscape of large-scale model serving.

The Question and Its Context

The conversation leading up to this message had been a whirlwind tour of model deployment on 8× NVIDIA RTX PRO 6000 Blackwell GPUs. The team had pivoted from the NVFP4 Kimi-K2.5 model (a 1T-parameter behemoth with 61-layer MLA architecture that was bottlenecked by PCIe allreduce) to MiniMax-M2.5, a 230B-parameter FP8 MoE model with only 10B active parameters per token. The MiniMax model had proven spectacularly successful at TP=4: it loaded in just 75 seconds (versus 13 minutes for Kimi-K2.5), used only 4 GPUs, and achieved 84 tok/s single-stream throughput with over 2,500 tok/s at high concurrency.

However, the user had just attempted TP=8 (<msg id=2296>–<msg id=2299>) and discovered a hard constraint: the FP8 block quantization alignment. With intermediate_size = 1536 per expert and FP8 quantization blocks of [128, 128], sharding the expert weights across 8 GPUs gave 1536 / 8 = 192, which is not divisible by 128. The model crashed with a ValueError: The output_size of gate's and up's weight = 192 is not divisible by weight quantization block_n = 128.

The assistant had just explained this failure and concluded that TP=4 was "arguably better" (<msg id=2302>). But the user wasn't satisfied — they wanted to know if there were other configurations that could use all 8 GPUs: TP=6, or a combination of tensor parallelism (TP) and expert parallelism (EP).

The Reasoning Process

The assistant's response at <msg id=2304> opens with a clear, structured analysis. It tackles the two proposals separately, and the thinking is immediately visible in the mathematical checks.

For TP=6: The assistant performs a quick divisibility check. intermediate_size / TP = 1536 / 6 = 256. Then 256 / 128 = 2 — perfectly divisible. The FP8 alignment constraint would be satisfied. But then a second, subtler constraint emerges: NUMA topology. The machine has 8 GPUs arranged as 4+4 across two NUMA nodes. TP=6 doesn't divide evenly into either node, meaning 3 GPUs would be used on each NUMA node. The assistant notes this is "fine for PCIe," acknowledging that while not ideal, it's workable. This reveals an assumption: that PCIe interconnect is the binding constraint, and that NUMA imbalance is tolerable compared to outright incompatibility.

For TP+EP: The assistant explains expert parallelism in clear terms: "Expert parallelism would shard the 256 experts across GPUs instead of replicating them. This is the ideal approach for MoE — less memory per GPU (experts distributed), with TP only for the attention/dense layers." This explanation reveals a deep understanding of MoE architecture: that the attention layers (which are dense, not sparse) still benefit from TP, while the MoE feed-forward layers can be distributed via EP without tensor sharding each expert's weights.

The Tool Invocation: A Targeted Probe

Rather than speculating further, the assistant then executes a targeted bash command to probe vLLM's actual capabilities:

ssh root@10.1.230.174 "/root/ml-env/bin/python3 -m vllm.entrypoints.openai.api_server --help 2>/dev/null | grep -E 'expert.parallel|tensor.parallel|pipeline.parallel|data.parallel' | head -10"

This is not a random exploration. It's a precise query designed to answer a specific question: does vLLM expose a separate --expert-parallel-size flag, or is EP implicitly tied to TP? The assistant already suspects the answer — the grep pattern deliberately excludes --enable-expert-parallel (which they already know about from the TP=8 attempt) and instead searches for size parameters.

The result reveals only --tensor-parallel-size, --pipeline-parallel-size, and --data-parallel-size — no --expert-parallel-size. This is a critical finding that will shape everything that follows.

Assumptions and Knowledge Required

To fully understand this message, one needs:

  1. MoE architecture knowledge: Understanding that Mixture-of-Experts models have both dense layers (attention) and sparse layers (expert feed-forward networks), and that these can be parallelized differently.
  2. FP8 quantization mechanics: Knowing that FP8 weights are stored in blocks (here 128×128) and that tensor parallelism must shard dimensions evenly across block boundaries.
  3. vLLM's parallelism model: Understanding that vLLM's --tensor-parallel-size controls both the TP group for attention and, when EP is enabled, also determines the EP group size. There is no separate EP size parameter.
  4. NUMA topology awareness: Recognizing that GPU interconnect topology (4 GPUs per NUMA node) imposes constraints on parallelism configurations.
  5. The specific model architecture: MiniMax-M2.5 has intermediate_size=1536, hidden_size=3072, num_attention_heads=48, num_key_value_heads=8, and 256 experts — all of which factor into the divisibility calculations.

What This Message Creates

This message generates several forms of output knowledge:

  1. A confirmed constraint: TP=6 is mathematically viable for FP8 alignment (1536/6=256, divisible by 128), but its NUMA implications need consideration.
  2. A hypothesis about EP behavior: The assistant correctly intuits that with EP enabled, the FP8 alignment check on expert weights might not apply because experts are distributed whole rather than tensor-sharded. This hypothesis will be tested in subsequent messages.
  3. A discovered limitation: vLLM does not expose a separate --expert-parallel-size flag — EP size equals TP size. This means you cannot independently scale EP and TP, which constrains the configuration space.
  4. A plan for investigation: The command output sets the stage for deeper probing into vLLM's EP implementation, which the assistant immediately follows up on in <msg id=2305>–<msg id=2309>.

Mistakes and Incorrect Assumptions

The assistant makes one notable assumption that turns out to be incorrect: that EP in vLLM truly separates expert distribution from tensor sharding. The assistant assumes that with --enable-expert-parallel, the MoE layers would not TP-shard expert weights, thus bypassing the FP8 alignment check. However, subsequent investigation (<msg id=2312>) reveals that vLLM's EP implementation simply reuses the TP group — world_size=2 with TP=2, EP=2 — meaning the experts are still sharded across the same 2 GPUs, not distributed whole across 8. This leads to an OOM failure in the next attempt.

This is a forgivable error. The vLLM codebase is complex, and the distinction between "EP distributes experts whole" (the ideal) and "EP reuses TP group" (the actual implementation) is subtle. The assistant's methodology — forming a hypothesis, testing it with a targeted command, and iterating — is exactly the right approach.

The Broader Significance

This message represents a pivotal moment in the session. The user has just hit a hard wall (TP=8 fails due to FP8 alignment) and is asking "what else can we try?" The assistant's response demonstrates how to systematically enumerate the configuration space:

Conclusion

Message <msg id=2304> is a beautiful example of how inference serving at scale is as much about arithmetic as it is about software. The assistant must hold in its working memory: the model's architectural dimensions (intermediate_size, hidden_size, num_heads, num_experts), the quantization parameters (FP8 block size), the hardware topology (8 GPUs, 2 NUMA nodes), and the software framework's parallelism model (vLLM's TP/EP semantics). From these, it must compute viable configurations, predict their behavior, and design experiments to validate those predictions.

The message also reveals a crucial meta-skill: knowing when to stop reasoning and start probing. The assistant could have continued speculating about EP behavior indefinitely, but instead it ran a targeted command to check vLLM's actual API. This balance between analytical reasoning and empirical validation is the hallmark of effective systems engineering in the ML deployment space.