The Three-Model Gauntlet: Systematic Evaluation of 1T-Parameter LLMs on 8× Blackwell GPUs

Introduction

In the high-stakes world of production LLM inference, the difference between a deployment that merely works and one that excels often comes down to a single decision: which model to serve. In a single marathon session spanning dozens of hours, a team systematically evaluated three different trillion-parameter-scale language models on a machine equipped with eight NVIDIA RTX PRO 6000 Blackwell GPUs, ultimately arriving at a production deployment that would serve as the backbone of their inference infrastructure. The journey took them from a 1-trillion-parameter NVFP4 model struggling at 61 tok/s single-stream throughput, through a 230B-parameter FP8 model that shattered expectations at nearly 4,000 tok/s, and finally to a native INT4 variant that delivered the best single-stream performance of the entire session.

This article examines the complete arc of that session — from the clean reinstallation of vLLM and the benchmarking of NVFP4 Kimi-K2.5, through the discovery of a fundamental PCIe allreduce bottleneck, to the pivot toward MiniMax-M2.5 FP8 and its astonishing 4,000 tok/s peak throughput, and finally to the deployment of the native INT4 Kimi-K2.5 as a persistent systemd service. Each phase of this journey reveals a different facet of what it means to deploy large language models at the frontier of hardware capability.

The Opening Move: Clean State and NVFP4 Benchmarking

The session began with a necessary housekeeping operation: a clean vLLM reinstall that removed stale GLM-5 debug patches. These patches — torch.save blocks in deepseek_v2.py, force-dequantization logic in weight_utils.py, and other accumulated artifacts from earlier experiments — could have interfered with subsequent deployments. The assistant's decision to start with a clean installation reflects a disciplined engineering approach: when debugging complex distributed inference systems, the first step is always to eliminate artifacts from previous experiments.

With a fresh vLLM installation, the team turned to the NVFP4 variant of Kimi-K2.5, a 1-trillion-parameter Mixture-of-Experts (MoE) model using Multi-head Latent Attention (MLA). The model, hosted by NVIDIA on Hugging Face, used NVIDIA's proprietary NVFP4 quantization — a 4-bit format with group_size=16 that compressed the 1T parameters into 540GB on disk. The architecture was DeepSeek V3 with 61 layers, 384 routed experts (top-8 active), and a single shared expert.

The benchmark results were sobering. At single-stream (C=1), the model delivered approximately 61 tok/s with a latency of 8.4 seconds for 512 tokens ([msg 2226]). At high concurrency (C=128), throughput scaled to roughly 1,200 tok/s. These numbers were competitive for a 1T-parameter MoE model on PCIe-interconnected GPUs, but they hinted at a deeper limitation.

The root cause was identified with surgical precision: PCIe allreduce across 8 GPUs for the 61-layer MLA architecture. The assistant's analysis quantified the impact: approximately 65–70% of decode time was spent in NCCL allreduce ([msg 2174]). The MLA architecture, while elegant in its memory efficiency, requires expensive allreduce operations at every attention layer to synchronize the latent projections across devices. With 61 layers, this created a staggering number of PCIe transactions per forward pass. On a system with no NVLink — where all inter-GPU communication flows through PCIe Gen5 — this was a fundamental hardware ceiling that no amount of software tuning could overcome.

The team had reached a plateau. The NVFP4 Kimi-K2.5 was working correctly, producing coherent output with proper reasoning chains. But the hardware was telling a clear story: this model architecture was not the right fit for this machine.

The First Pivot: MiniMax-M2.5 and the GQA Revolution

The turning point came with a single message from the user: "Try https://huggingface.co/MiniMaxAI/MiniMax-M2.5, which is native fp8, smaller activation so should be faster" ([msg 2232]). This was not a casual suggestion — it was a hypothesis grounded in architectural awareness. The user understood that the bottleneck was MLA's allreduce overhead, and they proposed a model with a fundamentally different attention mechanism: Grouped Query Attention (GQA).

The assistant immediately recognized the strategic significance of this pivot. Through detailed architectural analysis ([msg 2237]), it identified six reasons MiniMax-M2.5 should outperform Kimi-K2.5 on this specific hardware:

  1. GQA instead of MLA — standard FlashAttention works, eliminating the need for custom attention backends and their associated allreduce overhead
  2. 10B active parameters vs ~37B — 3.7× less compute per token
  3. FP8 native weights — half the memory bandwidth of BF16 for weight loads
  4. 3 MTP heads — built-in speculative decoding for multi-token prediction
  5. 230GB vs 540GB — more comfortable fit with more room for KV cache
  6. No cross-GPU attention — with GQA and 8 KV heads across 8 GPUs, each GPU gets exactly 1 KV head, eliminating the allreduce bottleneck entirely The pivot was executed decisively. The Kimi-K2.5 service was stopped and disabled ([msg 2239]), freeing 540GB of GPU memory. The MiniMax-M2.5 model — a 230B-parameter FP8 MoE model with 256 experts — was downloaded (a process that revealed a fascinating HuggingFace shard-numbering quirk where %05d formatting created phantom "missing" shards; see [msg 2262] through [msg 2266]). A new systemd service was created, and the model was deployed with Tensor Parallelism size 4 (TP=4). The results were immediate and dramatic. The model loaded in just 75 seconds — compared to Kimi-K2.5's 13-minute cold start ([msg 2282]). Single-stream throughput reached 84 tok/s, a 38% improvement over Kimi-K2.5's 61 tok/s. At high concurrency, the system delivered 2,586 tok/s at C=256, more than double the previous peak ([msg 2290], [msg 2292]). The assistant's reaction was uncharacteristically enthusiastic: "These results are spectacular" ([msg 2289]).

The TP=8 Failure: When Quantization Blocks Collide

With the TP=4 deployment validated, the natural next question was: can we use all 8 GPUs? The user asked to try TP=8 ([msg 2293]), hoping to squeeze even more performance from the hardware.

The attempt crashed immediately with a cryptic error:

ValueError: The output_size of gate's and up's weight = 192 is not divisible by weight quantization block_n = 128.

The assistant diagnosed the root cause with remarkable speed ([msg 2299]). MiniMax-M2.5 uses FP8 block quantization with a block size of [128, 128]. The model's MoE intermediate size is 1536. With TP=8, each GPU gets 1536 / 8 = 192 elements per shard, and 192 is not divisible by 128. With TP=4, each GPU gets 1536 / 4 = 384 = 3 × 128, which aligns perfectly. The quantization block structure was fundamentally incompatible with 8-way tensor parallelism for this model.

The assistant initially concluded that TP=4 was the correct choice, arguing that it was "arguably better" because it halved the allreduce overhead and left 4 GPUs free for other workloads ([msg 2302]). But the user pushed back with a question that would unlock the session's highest throughput: "can we do tp/ep or tp6?" ([msg 2303]).

The Breakthrough: Expert Parallelism at 4,000 tok/s

The user's question opened two parallel lines of investigation. The assistant first analyzed TP=6 mathematically: 1536 / 6 = 256, and 256 / 128 = 2 — divisible, so the FP8 alignment constraint would be satisfied. But a second constraint emerged: MiniMax-M2.5 has num_key_value_heads = 8, and 8 / 6 = 1.33 — not an integer. TP requires the number of attention heads to be evenly divisible by the TP size, so TP=6 was dead on arrival.

The second line of investigation — TP/EP (Tensor Parallelism combined with Expert Parallelism) — proved far more fruitful. The assistant embarked on a systematic exploration of vLLM's source code to understand how expert parallelism interacts with tensor parallelism ([msg 2305] through [msg 2308]). Through targeted grep commands across the codebase, the assistant discovered that:

ssh root@10.1.230.174 "/root/ml-env/bin/python3 -m vllm.entrypoints.openai.api_server --help 2>/dev/null | grep -A2 'expert.parallel'"

The grep pattern 'expert.parallel' used a regex wildcard dot to match both expert-parallel (CLI format) and expert_parallel (Python source format) in a single query. The -A2 flag captured the related --all2all-backend and --enable-dbo options, revealing the full EP configuration surface.

The breakthrough came when the assistant launched MiniMax-M2.5 with TP=8 and EP enabled. The model started successfully in about 90 seconds ([msg 2328]), and the benchmark results were transformative:

| Concurrency | TP=4 tok/s | TP=8+EP tok/s | Improvement | |---|---|---|---| | 256 | 2,586 | 3,982 | +54% | | 1024 | — | 3,997 | — |

The system achieved a flat ~4,000 tok/s throughput from C=256 through C=1024, with zero errors and no degradation. This was the highest throughput recorded in the entire session — 3.2× what the NVFP4 Kimi-K2.5 model achieved at its peak, and a staggering 65× the single-stream throughput of the original deployment.

The Final Pivot: Native INT4 Kimi-K2.5

With the MiniMax-M2.5 benchmarks complete, the session made one final pivot — back to the Kimi-K2.5 model family, but this time using the native INT4 quantization variant (moonshotai/Kimi-K2.5). This model, despite its 547GB size and 36-minute load time, delivered an impressive 82 tok/s single-stream — well exceeding the 40–50 tok/s target that had been set earlier — and scaled to 2,276 tok/s at high concurrency.

The INT4 variant significantly outperformed the NVFP4 variant (82 vs 61 tok/s single-stream), demonstrating that quantization format choice matters as much as model architecture. The native INT4 quantization, with its smaller memory footprint and more efficient GPU utilization, allowed the model to overcome some of the PCIe bottleneck that had plagued the NVFP4 version.

Extensive NCCL tuning experiments — testing Ring vs LL algorithms, varying channel counts, and adjusting thread configurations — confirmed that the bottleneck was fundamental hardware bandwidth, not algorithm choice. The session concluded by deploying the INT4 Kimi-K2.5 as a persistent systemd service (vllm-kimi-k25-int4.service), leaving the system with a clean, production-ready setup.

The Architecture of Discovery: What This Session Reveals

This session is far more than a sequence of model deployments and benchmarks. It is a case study in hardware-aware model selection — the art of understanding how a model's architectural properties interact with a specific hardware configuration to determine real-world performance.

The MLA vs GQA Lesson

The most important lesson from this session is the dramatic performance difference between MLA and GQA on PCIe-bound multi-GPU systems. MLA, while theoretically more parameter-efficient, requires allreduce operations at every attention layer. On a system with 8 GPUs connected via PCIe (no NVLink), this creates a communication bottleneck that dominates inference time. GQA, by contrast, avoids cross-GPU communication entirely when the number of KV heads matches the TP size.

This is not a judgment about which attention mechanism is "better" in the abstract — it is a concrete finding about which architecture performs better on this specific hardware. On an NVLink-connected DGX system, the results might be entirely different. The key insight is that model selection must be hardware-aware.

The Quantization Alignment Trap

The FP8 block quantization alignment failure with TP=8 is a cautionary tale about the hidden constraints that quantization formats impose on parallelism strategies. The [128, 128] block size of FP8 quantization creates a divisibility requirement that may not be satisfied by all TP configurations. This constraint is invisible until deployment time, when it manifests as a cryptic error message.

The solution — expert parallelism — elegantly bypasses the constraint by changing which weights are sharded. With EP, the expert weights are not tensor-sharded, so the FP8 alignment constraint on intermediate_size / TP simply does not apply. This is a powerful example of how understanding the interaction between quantization and parallelism can unlock configurations that would otherwise seem impossible.

The Value of Systematic Investigation

Throughout the session, the assistant demonstrated a methodical approach to problem-solving that is worth emulating. When the TP=8 launch failed, it did not guess at fixes — it read the error message, traced it to the FP8 quantization code, and calculated the divisibility constraint. When the user asked about TP/EP, it did not guess at support — it grepped the source code, discovered the flag, and verified its behavior. When the benchmark results came in, it did not just report them — it compared them against the previous model, calculated speedups, and identified the architectural reasons for the differences.

This systematic approach — hypothesis, investigation, experiment, analysis — is the hallmark of effective ML engineering. It transforms what could be a chaotic trial-and-error process into a disciplined exploration of the design space.

The Geometry of Parallelism: A Deeper Analysis

Throughout this session, the assistant's reasoning about parallelism configurations followed a systematic pattern. At each step, it enumerated the constraints — attention head divisibility, FP8 block alignment, GPU memory capacity, PCIe bandwidth — and then searched the space of viable configurations. This is a constraint-satisfaction problem where the model architecture defines a set of integer dimensions, the hardware defines resource limits, and the quantization scheme defines alignment requirements.

The key insight that unlocked TP=8+EP for MiniMax came from understanding that EP changes the constraint surface. When EP is enabled, the MoE layers use TP=1 internally — expert weights are not tensor-sharded. This means the FP8 alignment constraint on intermediate_size / TP simply doesn't apply to experts. The constraint only applies to attention and dense layers, which have different dimensions that are divisible by 8.

This understanding required deep knowledge of vLLM's distributed execution model. The assistant didn't just know that EP exists — it read the source code to understand exactly how EP interacts with TP for MoE layers. It found the documentation comment in config.py that explicitly states "TP for MoE becomes 1" when EP is enabled. This level of understanding is rare and valuable; most practitioners would have stopped at "EP is supported" without verifying the implementation details.

Lessons Learned

Several overarching themes emerge from this session:

Hardware-aware model selection is paramount. The choice between MLA and GQA attention architectures had a more significant impact on throughput than any software tuning parameter. On PCIe-bound multi-GPU systems, GQA's reduced communication overhead makes it vastly superior to MLA, regardless of model quality.

Quantization format matters independently of model architecture. The INT4 Kimi-K2.5 outperformed its NVFP4 sibling by 30%+ on the same hardware, despite having the same architectural characteristics. This demonstrates that quantization format selection is a first-order performance decision.

Source code is the ultimate documentation. When the assistant encountered the FP8 alignment error, it didn't stop at the error message. It traced the error to its source in vLLM's codebase, understood the guard condition, and realized that EP would bypass it. This source-level understanding was essential to the breakthrough.

The ceiling is physics, not software. After exhaustive NCCL tuning and configuration optimization, the system reached a flat throughput plateau at approximately 4,000 tok/s for MiniMax and 2,276 tok/s for Kimi-K2.5 INT4. No further software optimization could improve these numbers — they represent the fundamental hardware limits of the system.

Systematic benchmarking reveals the full picture. The decision to push beyond the initial test range (C=256) to C=1,024 revealed that the system was already saturated at C=256. Without that data, the team might have assumed further headroom existed and continued optimizing in the wrong direction.

Conclusion

The session that began with a 1T-parameter NVFP4 model struggling at 61 tok/s ended with a 230B-parameter FP8 model achieving 4,000 tok/s — a 65× improvement in throughput. In between, the team navigated PCIe bottlenecks, quantization alignment constraints, version mismatches, OOM crashes, and the discovery of expert parallelism infrastructure hidden in vLLM's source code.

The overarching lesson is clear: in production ML inference, the model is not the only variable. The hardware topology, the quantization format, the parallelism strategy, and the attention mechanism all interact in complex ways to determine real-world performance. Understanding these interactions — and being willing to pivot decisively when the data demands it — is the difference between a deployment that works and one that soars.

The final deployment — the INT4 Kimi-K2.5 running as a systemd service on 8 Blackwell GPUs — represents the culmination of dozens of hours of reasoning, debugging, and optimization. But the true legacy of this session is not the specific configuration chosen, but the methodology demonstrated: systematic constraint analysis, hypothesis-driven experimentation, source-level verification, and the relentless pursuit of hardware limits. These are the skills that separate effective ML engineering from trial-and-error configuration.## References

[1] "The Great Model Pivot: How Hardware-Aware Architecture Selection Unlocked 4,000 tok/s on 8× Blackwell GPUs" — Chunk article covering the pivot from NVFP4 Kimi-K2.5 to MiniMax-M2.5 FP8 and the EP breakthrough.

[2] "The Architecture of a Pivot: Deploying 1T-Parameter Models Across 8 Blackwell GPUs" — Chunk article providing detailed analysis of the parallelism geometry, NCCL tuning, and systemd deployment.