The Four Words That Uncovered a Hardware Constraint: "run and bench tp8"

In the middle of an intense ML deployment session on an 8× Blackwell GPU system, a user typed just four words: "run and bench tp8" ([msg 2293]). This terse command — barely a sentence — would trigger a cascade of investigation that revealed a fundamental hardware-software compatibility constraint, forced the team to rethink their parallelism strategy, and ultimately led to a deeper understanding of how FP8 quantization interacts with tensor parallelism on modern GPU architectures.

The Context: A Model That Was Already Winning

To understand why this message matters, we need to appreciate what had just happened. The team had spent the previous hour pivoting from the problematic NVFP4 variant of Kimi-K2.5 (a 1-trillion-parameter model) to MiniMax-M2.5, a 230-billion-parameter FP8 model using Grouped-Query Attention (GQA) with only 10 billion active parameters per token. The results were spectacular: the model loaded in just 75 seconds (compared to 13 minutes for Kimi-K2.5), used only 4 of the 8 available GPUs via tensor parallelism of size 4 (TP=4), and delivered 84 tokens per second for single-user requests — a 37% improvement over Kimi-K2.5's 61 tok/s. At high concurrency, it reached an astonishing 2,586 tok/s ([msg 2292]).

The assistant had just presented a comprehensive benchmark comparison table, celebrating MiniMax-M2.5's advantages: half the power consumption, faster startup, lower latency, and superior throughput. The model was running happily on GPUs 0–3, leaving GPUs 4–7 completely idle.

The Reasoning Behind the Request

The user's command — "run and bench tp8" — was motivated by a natural and entirely reasonable instinct: if 4 GPUs are good, wouldn't 8 GPUs be better? The system had 8 Blackwell GPUs, each with 96GB of HBM2e memory. Four of them were sitting idle, drawing minimal power (around 33W each). The user wanted to know: could we double the tensor parallelism from 4 to 8 and get even higher throughput?

This is the kind of question any engineer would ask. In the world of large-scale ML inference, more GPUs generally means more memory bandwidth, lower latency, and higher throughput. The user's assumption was that TP=8 would be a straightforward scaling of the current TP=4 configuration — just change a flag and reap the benefits.

But this assumption concealed a hidden constraint that would only become visible through failure.

The Execution: A Crash and Its Aftermath

The assistant immediately acted on the request. The first step was to stop the running TP=4 service and free GPU memory ([msg 2294]). After a brief struggle with a stuck systemd service (requiring a SIGKILL to terminate), the GPUs were freed and a TP=8 launch was attempted ([msg 2296]).

The launch command used the same parameters as the TP=4 configuration, with only --tensor-parallel-size 8 changed. The assistant launched the server in the background and began polling for readiness.

The process died within about 30 seconds ([msg 2297]).

The Discovery: FP8 Block Quantization Alignment

The crash log revealed the root cause ([msg 2298]):

ValueError: The output_size of gate's and up's weight = 192 
is not divisible by weight quantization block_n = 128.

This error message is dense with meaning. The MiniMax-M2.5 model uses FP8 quantization with a block size of [128, 128] — meaning weights are quantized in 128-element blocks along each dimension. The model's MoE (Mixture of Experts) intermediate size is 1,536. With TP=8, each GPU gets 1,536 / 8 = 192 elements of the intermediate dimension. But 192 is not divisible by 128, so the quantization blocks cannot be evenly distributed across GPUs.

With TP=4, the shard size was 1,536 / 4 = 384, and 384 / 128 = 3 — a perfect fit. TP=8 broke that alignment.

This is a classic example of a hardware-software co-design constraint that is invisible until you hit it. The FP8 quantization format was designed with a specific block size that works well for common sharding configurations, but the combination of this particular model architecture (intermediate_size=1,536) with TP=8 creates an edge case where the math doesn't work.

The Assumptions That Were Challenged

Several assumptions were tested and disproven by this experiment:

  1. "More GPUs always helps": The assumption that scaling from TP=4 to TP=8 would improve performance was wrong. Not only did it fail to run, but the failure revealed that TP=4 was actually the maximum viable tensor parallelism for this model given its FP8 quantization scheme.
  2. "TP is a simple scaling parameter": The user (and initially the assistant) treated --tensor-parallel-size as a knob that could be freely adjusted. The crash demonstrated that TP interacts with model architecture details — specifically quantization block sizes — in ways that must be carefully validated.
  3. "The model will run identically on any number of GPUs": The model's architecture (intermediate_size=1,536, num_key_value_heads=8) creates specific divisibility constraints that limit viable TP configurations. This is not a bug in the model or the framework — it's a fundamental mathematical constraint of the quantization scheme.

Input Knowledge Required

To understand this message and its consequences, one needs:

Output Knowledge Created

This single message produced several important insights:

  1. TP=8 is impossible for MiniMax-M2.5: The FP8 quantization alignment constraint is a hard blocker. No amount of tuning or optimization can fix it — it's a mathematical incompatibility.
  2. TP=4 is the optimal TP configuration: Since 1,536 / 4 = 384 is divisible by 128, and TP=6 fails because 8 KV heads aren't divisible by 6, TP=4 is the only viable TP configuration that uses a meaningful number of GPUs.
  3. The need for Expert Parallelism (EP): The failure of TP=8 led directly to exploring EP, which distributes experts across GPUs without sharding individual expert weights. This avoids the FP8 alignment issue entirely because each EP rank holds full experts, not shards.
  4. A deeper understanding of the hardware bottleneck: The TP=8 failure was the first hint that PCIe allreduce bandwidth — not compute — was the primary bottleneck. This understanding would later inform the decision to pivot to models with smaller active parameter counts and GQA architectures.

The Broader Significance

The "run and bench tp8" message is a perfect example of how negative results drive progress in ML engineering. The request was simple, the execution was straightforward, and the failure was informative. Rather than being a setback, the TP=8 crash opened up a new line of investigation into expert parallelism, which would eventually yield even better results (nearly 4,000 tok/s with TP=8+EP).

It also demonstrates the importance of understanding the full stack — from model architecture details (intermediate_size, quantization block size) to hardware topology (8 GPUs, PCIe interconnect) to framework internals (vLLM's parallelism implementation). The user's four-word command was the spark, but the real value came from the assistant's systematic investigation of why it failed, turning a crash into a learning opportunity.

In the end, the team learned that sometimes the best configuration is not "use all the hardware" but "use the right amount of hardware for the model." TP=4 on 4 GPUs was not a compromise — it was the optimal configuration, and the TP=8 experiment proved it by failing.