Segment 18

In this sub-session, we systematically evaluated three large language models on the 8x RTX PRO 6000 Blackwell GPU system to find the optimal deployment for production inference. After a clean vLLM reinstall that removed stale GLM-5 debug patches, the NVFP4 Kimi-K2.5 was benchmarked at ~61 tok/s single-stream, revealing a fundamental PCIe allreduce bottleneck for the 61-layer MLA architecture. We pivoted to MiniMax-M2.5, a 230B FP8 GQA model, which loaded in 75 seconds and achieved 84 tok/s single-stream with TP=4, and nearly 4,000 tok/s with TP=8 using Expert Parallelism (EP8), demonstrating that GQA with smaller active parameters is vastly superior on PCIe-bound Blackwell hardware. Finally, we deployed the native INT4 Kimi-K2.5, which despite its 547GB size and 36-minute load time, delivered 82 tok/s single-stream and scaled to 2,276 tok/s at high concurrency. Extensive NCCL tuning confirmed the bottleneck was fundamental hardware bandwidth. The session concluded by deploying the INT4 Kimi-K2.5 as a persistent systemd service, leaving a clean, production-ready setup.

The Three-Model Gauntlet: Systematic Evaluation of 1T-Parameter LLMs on 8× Blackwell GPUs 2828 words

Chunks

• The Great Model Pivot: How Hardware-Aware Architecture Selection Unlocked 4,000 tok/s on 8× Blackwell GPUs
  2156 words
• The Architecture of a Pivot: Deploying 1T-Parameter Models Across 8 Blackwell GPUs
  3951 words