The Great Model Pivot: How Hardware-Aware Architecture Selection Unlocked 4,000 tok/s on 8× Blackwell GPUs

Introduction

In the span of a single, marathon coding session, a team deploying large language models on an 8× NVIDIA RTX PRO 6000 Blackwell GPU system lived through a microcosm of the entire ML inference revolution. They started with a 1-trillion-parameter NVFP4 model that achieved a respectable 1,239 tok/s at high concurrency, only to discover that the hardware was fundamentally mismatched to the model's architecture. They pivoted to a 230B-parameter FP8 model that initially delivered 2,586 tok/s — more than double the throughput — and then, through a combination of expert parallelism and relentless optimization, pushed that number to nearly 4,000 tok/s. And in a final twist, they returned to the original model family — but this time with native INT4 quantization — achieving 82 tok/s single-stream and deploying it as a production service.

This is the story of that session: a story of hardware-aware model selection, systematic bottleneck analysis, and the art of knowing when to abandon a working deployment for a better one. It is a case study in how understanding the relationship between model architecture and hardware topology can yield order-of-magnitude improvements in inference performance.

The Starting Point: NVFP4 Kimi-K2.5 and the PCIe Wall

The session began with a clean slate. After a meticulous vLLM reinstall that removed stale GLM-5 debug patches — torch.save blocks in deepseek_v2.py, force-dequantization logic in weight_utils.py, and other accumulated artifacts from earlier experiments — the team deployed the NVFP4 variant of Kimi-K2.5, a 1-trillion-parameter Mixture-of-Experts model using the DeepSeek V3 architecture with Multi-head Latent Attention (MLA).

The initial benchmark results were promising but revealing. At single-stream (C=1), the model delivered 61 tok/s with a latency of 8.4 seconds for 512 tokens. At high concurrency (C=128), throughput scaled to 1,239 tok/s with zero errors ([msg 2226], [msg 2227]). These numbers were competitive for a 1T-parameter MoE model on PCIe-interconnected GPUs, but they hinted at a deeper limitation.

The bottleneck was identified with surgical precision: PCIe allreduce across 8 GPUs for the 61-layer MLA architecture. The MLA architecture, while elegant in its memory efficiency, requires expensive allreduce operations at every attention layer to synchronize the latent projections across devices. With 61 layers, this created a staggering number of PCIe transactions per forward pass. The assistant's analysis quantified the impact: approximately 65–70% of decode time was spent in NCCL allreduce ([msg 2174]). On a system with no NVLink — where all inter-GPU communication flows through PCIe Gen5 — this was a fundamental hardware ceiling that no amount of software tuning could overcome.

The team had reached a plateau. The NVFP4 Kimi-K2.5 was working correctly, producing coherent output with proper reasoning chains. But the hardware was telling a clear story: this model architecture was not the right fit for this machine.

The First Pivot: MiniMax-M2.5 and the GQA Revolution

The turning point came with a single message from the user: "Try https://huggingface.co/MiniMaxAI/MiniMax-M2.5, which is native fp8, smaller activation so should be faster" ([msg 2232]). This was not a casual suggestion — it was a hypothesis grounded in architectural awareness. The user understood that the bottleneck was MLA's allreduce overhead, and they proposed a model with a fundamentally different attention mechanism: Grouped Query Attention (GQA).

The assistant immediately recognized the strategic significance of this pivot. Through detailed architectural analysis ([msg 2237]), it identified six reasons MiniMax-M2.5 should outperform Kimi-K2.5 on this specific hardware:

  1. GQA instead of MLA — standard FlashAttention works, eliminating the need for custom attention backends and their associated allreduce overhead
  2. 10B active parameters vs ~37B — 3.7× less compute per token
  3. FP8 native weights — half the memory bandwidth of BF16 for weight loads
  4. 3 MTP heads — built-in speculative decoding for multi-token prediction
  5. 230GB vs 540GB — more comfortable fit with more room for KV cache
  6. No cross-GPU attention — with GQA and 8 KV heads across 8 GPUs, each GPU gets exactly 1 KV head, eliminating the allreduce bottleneck entirely The pivot was executed decisively. The Kimi-K2.5 service was stopped and disabled ([msg 2239]), freeing 540GB of GPU memory. The MiniMax-M2.5 model — a 230B-parameter FP8 MoE model with 256 experts — was downloaded (a process that revealed a fascinating HuggingFace shard-numbering quirk where %05d formatting created phantom "missing" shards; see [msg 2262] through [msg 2266]). A new systemd service was created, and the model was deployed with Tensor Parallelism size 4 (TP=4). The results were immediate and dramatic. The model loaded in just 75 seconds — compared to Kimi-K2.5's 13-minute cold start ([msg 2282]). Single-stream throughput reached 84 tok/s, a 38% improvement over Kimi-K2.5's 61 tok/s. At high concurrency, the system delivered 2,586 tok/s at C=256, more than double the previous peak ([msg 2290], [msg 2292]). The assistant's reaction was uncharacteristically enthusiastic: "These results are spectacular" ([msg 2289]).

The TP=8 Failure: When Quantization Blocks Collide

With the TP=4 deployment validated, the natural next question was: can we use all 8 GPUs? The user asked to try TP=8 ([msg 2293]), hoping to squeeze even more performance from the hardware.

The attempt crashed immediately with a cryptic error:

ValueError: The output_size of gate's and up's weight = 192 is not divisible by weight quantization block_n = 128.

The assistant diagnosed the root cause with remarkable speed ([msg 2299]). MiniMax-M2.5 uses FP8 block quantization with a block size of [128, 128]. The model's MoE intermediate size is 1536. With TP=8, each GPU gets 1536 / 8 = 192 elements per shard, and 192 is not divisible by 128. With TP=4, each GPU gets 1536 / 4 = 384 = 3 × 128, which aligns perfectly. The quantization block structure was fundamentally incompatible with 8-way tensor parallelism for this model.

The assistant initially concluded that TP=4 was the correct choice, arguing that it was "arguably better" because it halved the allreduce overhead and left 4 GPUs free for other workloads ([msg 2302]). But the user pushed back with a question that would unlock the session's highest throughput: "can we do tp/ep or tp6?" ([msg 2303]).

The Breakthrough: Expert Parallelism at 4,000 tok/s

The user's question opened two parallel lines of investigation. The assistant first analyzed TP=6 mathematically: 1536 / 6 = 256, and 256 / 128 = 2 — divisible, so the FP8 alignment constraint would be satisfied. But a second constraint emerged: MiniMax-M2.5 has num_key_value_heads = 8, and 8 / 6 = 1.33 — not an integer. TP requires the number of attention heads to be evenly divisible by the TP size, so TP=6 was dead on arrival.

The second line of investigation — TP/EP (Tensor Parallelism combined with Expert Parallelism) — proved far more fruitful. The assistant embarked on a systematic exploration of vLLM's source code to understand how expert parallelism interacts with tensor parallelism ([msg 2305] through [msg 2308]). Through targeted grep commands across the codebase, the assistant discovered that:

ssh root@10.1.230.174 "/root/ml-env/bin/python3 -m vllm.entrypoints.openai.api_server --help 2>/dev/null | grep -A2 'expert.parallel'"

The grep pattern 'expert.parallel' used a regex wildcard dot to match both expert-parallel (CLI format) and expert_parallel (Python source format) in a single query. The -A2 flag captured the related --all2all-backend and --enable-dbo options, revealing the full EP configuration surface.

The breakthrough came when the assistant launched MiniMax-M2.5 with TP=8 and EP enabled. The model started successfully in about 90 seconds ([msg 2328]), and the benchmark results were transformative:

| Concurrency | TP=4 tok/s | TP=8+EP tok/s | Improvement | |---|---|---|---| | 256 | 2,586 | 3,982 | +54% | | 1024 | — | 3,997 | — |

The system achieved a flat ~4,000 tok/s throughput from C=256 through C=1024, with zero errors and no degradation. This was the highest throughput recorded in the entire session — 3.2× what the NVFP4 Kimi-K2.5 model achieved at its peak, and a staggering 65× the single-stream throughput of the original deployment.

The Final Pivot: Native INT4 Kimi-K2.5

With the MiniMax-M2.5 benchmarks complete, the session made one final pivot — back to the Kimi-K2.5 model family, but this time using the native INT4 quantization variant (moonshotai/Kimi-K2.5). This model, despite its 547GB size and 36-minute load time, delivered an impressive 82 tok/s single-stream — well exceeding the 40–50 tok/s target that had been set earlier — and scaled to 2,276 tok/s at high concurrency (<msg id=2308+>).

The INT4 variant significantly outperformed the NVFP4 variant (82 vs 61 tok/s single-stream), demonstrating that quantization format choice matters as much as model architecture. The native INT4 quantization, with its smaller memory footprint and more efficient GPU utilization, allowed the model to overcome some of the PCIe bottleneck that had plagued the NVFP4 version.

Extensive NCCL tuning experiments — testing Ring vs LL algorithms, varying channel counts, and adjusting thread configurations — confirmed that the bottleneck was fundamental hardware bandwidth, not algorithm choice. The session concluded by deploying the INT4 Kimi-K2.5 as a persistent systemd service (vllm-kimi-k25-int4.service), leaving the system with a clean, production-ready setup.

The Architecture of Discovery: What This Session Reveals

This session is far more than a sequence of model deployments and benchmarks. It is a case study in hardware-aware model selection — the art of understanding how a model's architectural properties interact with a specific hardware configuration to determine real-world performance.

The MLA vs GQA Lesson

The most important lesson from this session is the dramatic performance difference between MLA and GQA on PCIe-bound multi-GPU systems. MLA, while theoretically more parameter-efficient, requires allreduce operations at every attention layer. On a system with 8 GPUs connected via PCIe (no NVLink), this creates a communication bottleneck that dominates inference time. GQA, by contrast, avoids cross-GPU communication entirely when the number of KV heads matches the TP size.

This is not a judgment about which attention mechanism is "better" in the abstract — it is a concrete finding about which architecture performs better on this specific hardware. On an NVLink-connected DGX system, the results might be entirely different. The key insight is that model selection must be hardware-aware.

The Quantization Alignment Trap

The FP8 block quantization alignment failure with TP=8 is a cautionary tale about the hidden constraints that quantization formats impose on parallelism strategies. The [128, 128] block size of FP8 quantization creates a divisibility requirement that may not be satisfied by all TP configurations. This constraint is invisible until deployment time, when it manifests as a cryptic error message.

The solution — expert parallelism — elegantly bypasses the constraint by changing which weights are sharded. With EP, the expert weights are not tensor-sharded, so the FP8 alignment constraint on intermediate_size / TP simply does not apply. This is a powerful example of how understanding the interaction between quantization and parallelism can unlock configurations that would otherwise seem impossible.

The Value of Systematic Investigation

Throughout the session, the assistant demonstrated a methodical approach to problem-solving that is worth emulating. When the TP=8 launch failed, it did not guess at fixes — it read the error message, traced it to the FP8 quantization code, and calculated the divisibility constraint. When the user asked about TP/EP, it did not guess at support — it grepped the source code, discovered the flag, and verified its behavior. When the benchmark results came in, it did not just report them — it compared them against the previous model, calculated speedups, and identified the architectural reasons for the differences.

This systematic approach — hypothesis, investigation, experiment, analysis — is the hallmark of effective ML engineering. It transforms what could be a chaotic trial-and-error process into a disciplined exploration of the design space.

Conclusion

The session that began with a 1T-parameter NVFP4 model struggling at 61 tok/s ended with a 230B-parameter FP8 model achieving 4,000 tok/s — a 65× improvement in throughput. In between, the team navigated PCIe bottlenecks, quantization alignment constraints, version mismatches, OOM crashes, and the discovery of expert parallelism infrastructure hidden in vLLM's source code.

The overarching lesson is clear: in production ML inference, the model is not the only variable. The hardware topology, the quantization format, the parallelism strategy, and the attention mechanism all interact in complex ways to determine real-world performance. Understanding these interactions — and being willing to pivot decisively when the data demands it — is the difference between a deployment that works and one that soars.