Chunk 18.0

This chunk was a whirlwind tour of rapid model deployment and hardware-boundary exploration, pivoting from the problematic NVFP4 Kimi-K2.5 to find the fastest viable 1T-parameter model. After a clean vLLM reinstall that removed stale GLM-5 debug patches, the NVFP4 Kimi was benchmarked at ~61 tok/s single-stream, but the team quickly identified the fundamental bottleneck: PCIe allreduce across 8 GPUs for the 61-layer MLA architecture. The user then pivoted to **MiniMax-M2.5**, a 230B FP8 GQA model that loaded in 75 seconds and used only 4 GPUs (TP=4), achieving 84 tok/s single-stream and over 2,500 tok/s at high concurrency. A TP=8 attempt failed due to FP8 block quantization alignment, but TP=8 with Expert Parallelism (EP) succeeded, pushing throughput to nearly **4,000 tok/s**, proving that GQA + smaller active parameters is vastly superior on PCIe-bound Blackwell hardware. The final pivot was to the native **INT4 Kimi-K2.5** (`moonshotai/Kimi-K2.5`). Despite its 547GB size and 36-minute load time, it delivered an impressive **82 tok/s single-stream** (well exceeding the 40–50 target) and scaled to 2,276 tok/s at high concurrency, significantly outperforming the NVFP4 variant. Extensive NCCL tuning (Ring, LL, channels, threads) and vLLM compilation flags confirmed the bottleneck was fundamental hardware bandwidth, not algorithm choice. The session concluded by deploying the Kimi-K2.5 INT4 model as a persistent systemd service (`vllm-kimi-k25-int4.service`), leaving the system with a clean, production-ready setup. The overarching themes were **hardware-aware model selection** (GQA vs MLA, INT4 vs NVFP4), **aggressive benchmarking across concurrency levels**, and **systematic elimination of software artifacts** (stale patches, FP8 alignment issues) to isolate genuine hardware limits.

The Great Model Pivot: How Hardware-Aware Architecture Selection Unlocked 4,000 tok/s on 8× Blackwell GPUs 2156 words

Message Articles