The Limits of PCIe: Diagnosing a 1T-Parameter Model's Throughput Ceiling

Message Overview

In this pivotal message (msg 2377), the assistant pauses after a round of benchmarking to analyze why the newly deployed Kimi-K2.5 INT4 model—a ~1 trillion parameter giant—is achieving 76–82 tok/s single-stream on eight NVIDIA RTX PRO 6000 Blackwell GPUs. Rather than celebrating the fact that this already exceeds the user's 40–50 tok/s target by a wide margin, the assistant performs a careful post-mortem on the vLLM engine configuration, identifies a fundamental hardware bottleneck, and attempts one last NCCL tuning probe before concluding that the ceiling is architectural, not configurable.

The message is a masterclass in performance diagnosis under real-world constraints: it reads the vLLM initialization log to understand what optimizations are and aren't active, reasons about the interaction between quantization format and hardware topology, and correctly identifies that the bottleneck is not software-tunable but rather a fundamental consequence of PCIe allreduce bandwidth for 61-layer MLA attention across eight GPUs.

The Context: What Led to This Message

The preceding messages document a whirlwind of model deployment and benchmarking. After pivoting from the NVFP4 variant of Kimi-K2.5 (which topped out at ~61 tok/s), the assistant downloaded the native INT4 version—a 547GB model spread across 64 safetensor shards—and launched it with TP=8 tensor parallelism. The model took approximately 36 minutes to load ([msg 2360]), consuming 96.7 GB of the 97.9 GB available on each GPU.

Initial benchmarks showed 81.4 tok/s single-stream ([msg 2365]), already well above the user's target. A 2048-token generation confirmed 76.2 tok/s including prefill time ([msg 2375]). The assistant then tried NCCL tuning (Ring algorithm, LL protocol, channel count adjustments) in msg 2370, but the tuned version produced nearly identical results: 81.9 vs 81.4 tok/s ([msg 2373]).

This is the moment msg 2377 arrives. The assistant has just checked the vLLM log for compilation flags ([msg 2376]) and now synthesizes what it sees.

What the Message Actually Says

Let's quote the message in full:

I notice: 'custom_ops': ['none'] and 'fuse_norm_quant': False, 'fuse_act_quant': False. With compressed-tensors INT4, vLLM is NOT using the quant fusion passes that it uses for FP8. Also CUDAGraph is working (AOT compilation loaded from cache).

The model is already at 76-82 tok/s single-stream for a 1T parameter model on PCIe, which is quite good. The bottleneck is fundamental — 61 layers of MLA attention with allreduce across 8 PCIe GPUs. Let me check if NCCL_MIN_NCHANNELS helps or if we can try NCCL_CROSS_NIC:

[bash] ssh root@10.1.230.174 "grep -c 'NCCL' /tmp/vllm_kimi_int4_tuned.log; grep 'NCCL' /tmp/vllm_kimi_int4_tuned.log | head -5" 0

The message has three distinct layers: an observation about vLLM's quantization configuration, a diagnosis of the fundamental bottleneck, and a final empirical check that returns null.

Deep Dive: The Quantization Observation

The first sentence is dense with meaning. The assistant reads the vLLM engine initialization log and spots two critical configuration items:

custom_ops: ['none'] — This tells us that vLLM's compressed-tensors INT4 path is not using any custom CUDA kernels for quantization operations. In contrast, the FP8 path in vLLM has extensive custom kernel support, including fused quantization of normalization and activation outputs.
fuse_norm_quant: False, fuse_act_quant: False — These flags confirm that the INT4 path is not fusing layer normalization with quantization, nor activation with quantization. Fusion is a standard optimization that reduces memory traffic and kernel launch overhead by combining adjacent operations into a single kernel. The implication is significant: the INT4 model is running without the optimization passes that make FP8 models faster. This is not necessarily a bug—compressed-tensors INT4 is a different quantization scheme with different kernel requirements—but it means there is latent headroom if vLLM ever adds INT4 fusion passes. The assistant is filing this observation away for future reference. The message also confirms that CUDAGraph is working with AOT (ahead-of-time) compilation loaded from cache. CUDAGraph is vLLM's mechanism for capturing GPU operations as a static graph and replaying them, eliminating Python interpreter overhead during decode. This is good news: the decode path is already as efficient as vLLM can make it on the software side.

The Bottleneck Diagnosis

The central insight of this message is the bottleneck identification: "61 layers of MLA attention with allreduce across 8 PCIe GPUs."

MLA (Multi-Head Latent Attention) is the attention mechanism used by DeepSeek-derived architectures like Kimi-K2.5. Unlike standard multi-head attention (MHA) or grouped-query attention (GQA), MLA uses a latent projection that compresses the key-value cache. However, it still requires allreduce operations across tensor-parallel GPUs for each attention layer.

With 61 layers and 8 GPUs connected via PCIe (not NVLink), each decode step requires 61 allreduce operations. Each allreduce must transfer the activations for all hidden dimensions across the PCIe bus. On Blackwell GPUs with PCIe Gen5 x16, the theoretical bandwidth is ~64 GB/s per direction, but practical allreduce bandwidth is lower due to protocol overhead and the fact that all 8 GPUs must synchronize.

The assistant's phrasing—"the bottleneck is fundamental"—is a crucial diagnostic conclusion. It means that no amount of NCCL tuning, CUDAGraph optimization, or quantization fusion will materially improve single-stream throughput. The hardware topology is the constraint.

The NCCL Probe and Its Null Result

The assistant then tries one more thing: checking whether NCCL configuration is visible in the vLLM log, and whether NCCL_MIN_NCHANNELS or NCCL_CROSS_NIC could help. The bash command searches for "NCCL" in the log and finds zero matches.

This null result is informative in itself. NCCL (NVIDIA Collective Communications Library) does not log its configuration by default; it only logs when explicitly configured to do so via NCCL_DEBUG=INFO or similar environment variables. The assistant's assumption that NCCL configuration would appear in the vLLM log was incorrect—but the probe was still valuable because it confirmed that the previous tuning attempt (setting NCCL_ALGO=Ring, NCCL_PROTO=LL, etc.) had no visible effect in the log, consistent with the benchmark results showing identical performance.

The Thinking Process Visible in This Message

The assistant's reasoning chain is transparent and methodical:

Observe the configuration: Read the vLLM log to see what optimizations are active.
Compare to known baselines: INT4 lacks the fusion passes that FP8 has; CUDAGraph is working.
Evaluate performance: 76–82 tok/s is "quite good" for a 1T model on PCIe.
Identify the root cause: 61 MLA layers × allreduce across 8 GPUs = fundamental PCIe bottleneck.
Test a remaining hypothesis: Check if NCCL configuration is visible and tunable.
Accept the result: The null NCCL log confirms the bottleneck is not software-configurable. This is textbook performance debugging: start with what you can observe, reason about the architecture, test specific hypotheses, and accept the conclusion when evidence supports it.

Input Knowledge Required

To fully understand this message, one needs:

vLLM architecture knowledge: Understanding of tensor parallelism, CUDAGraph, quantization fusion passes, and the compressed-tensors INT4 path.
NCCL familiarity: Knowledge of NCCL algorithms (Ring, Tree), protocols (LL, Simple), and how they interact with PCIe topology.
MLA attention: Understanding that Multi-Head Latent Attention has different allreduce characteristics than standard MHA or GQA.
Hardware topology awareness: PCIe Gen5 bandwidth limits, the absence of NVLink between these GPUs, and how allreduce scales with GPU count.
The earlier session history: The pivot from NVFP4 to INT4, the NCCL tuning attempts, and the benchmark results that preceded this message.

Output Knowledge Created

This message produces several lasting insights:

INT4 performance ceiling confirmed: The Kimi-K2.5 INT4 model achieves 76–82 tok/s on 8× Blackwell GPUs via PCIe, and this is the hardware limit.
Quantization fusion gap documented: vLLM's compressed-tensors INT4 path lacks the fusion passes available for FP8, representing potential future headroom.
NCCL tuning ruled out: The null NCCL log confirms that NCCL algorithm/protocol tuning has no effect on this topology.
Diagnostic methodology demonstrated: The message shows how to read vLLM logs for configuration, reason about bottlenecks from first principles, and test hypotheses empirically.

Conclusion

Message 2377 is a quiet but crucial turning point in the session. After hours of deployment, debugging, and tuning, the assistant arrives at a clear-eyed assessment: the model is performing at the hardware limit, and further optimization would require architectural changes (NVLink, fewer GPUs, or a different attention mechanism) rather than software configuration. The message is notable for its intellectual honesty—the assistant does not chase diminishing returns or pretend that more tuning will help. Instead, it documents the bottleneck, accepts the constraint, and moves on to production deployment in subsequent messages.

For anyone debugging large-model inference on multi-GPU PCIe systems, this message is a case study in how to recognize when you've hit a fundamental limit versus a configurable one.