When the Data Speaks: How a Single Profiling Result Upended the Optimization Strategy for Kimi-K2.5 on 8x Blackwell GPUs

In the middle of an intense, multi-week engineering campaign to deploy large language models on an 8x NVIDIA RTX PRO 6000 Blackwell GPU system, a single message arrived that fundamentally reshaped the team's understanding of where their performance problems lay. The message, sent by the AI assistant at index 2458 of the conversation, is deceptively brief — a few paragraphs of summary text and a table. But within those paragraphs lies the culmination of an exhaustive three-phase benchmarking campaign, the refutation of a deeply held hypothesis about the system's bottleneck, and the redirection of the entire optimization effort toward a new target.

This article examines that message in depth: the reasoning that produced it, the assumptions it overturned, the knowledge it required, and the knowledge it created.

The Message Itself

The assistant wrote:

All benchmarks are complete and documented. Here's the summary of what we found:

## Key Findings

### The Bottleneck is NOT the GEMMs — It's AllReduce

The torch.profiler revealed the actual bottleneck for Kimi-K2.5 INT4 on vLLM:

| Category | Per-step (ms) | % of Total | |---|---|---| | AllReduce (NCCL + custom) | 11.17 | 51.5% | | Attention | 3.32 | 15.3% | | Other GEMMs (cuBLAS) | 2.90 | 13.4% | | MoE GEMMs (Marlin) | 2.40 | 11.1% | | Routing | 0.99 | 4.5% | | Everything else | 0.92 | 4.2% |

### The Game Has Changed From GLM-5

The biggest surprise: Marlin W4A16 kernels have ZERO dtype-cast overhead. Remember how GLM-5 on SGLang spent 69% of decode time on unrolled_elementwise_kernel (dtype casting)? That's completely gone with Marlin — the INT4→BF16 dequantization is fused directly into the GEMM. This makes Kimi-K2.5 on vLLM 31.8% HBM-efficient (vs 3.4% for GLM-5).

But eliminating the cast bottleneck revealed that AllReduce was always the real problem — it was just hidden behind the even-larger cast overhead. With 127 NCCL AllReduce calls per decode step at 78us each = 9.88ms, and the vllm::all_reduce custom path adding another 1.16ms, communication alone accounts for over half of the 12.1ms TPOT.

### The #1 Optimization Target: Expert Parallelism

EP would replace 60 MoE AllReduces with 60 All-to-All operations (19us vs 78us at decode batch=1), saving ~3.54ms per token (29% of TPOT). Plus the expert GEMMs become 8× wider, better utilizing the GPU. This is a single --enable-expert-parallel flag.

All results are in k25b6000bench1.md. The vLLM service is restarting now (~30 min to load).

Why This Message Was Written: The Culmination of a Three-Phase Investigation

This message was not written in a vacuum. It is the terminal point of a carefully planned investigation that spanned dozens of messages across multiple days. To understand why this particular message exists, one must understand the journey that preceded it.

The team had been working on deploying the Kimi-K2.5 model in its INT4 quantized format (using the compressed-tensors format with W4A16 and group_size=32) on a system with 8x RTX PRO 6000 Blackwell GPUs connected only via PCIe — no NVLink. This is a critical hardware constraint: without NVLink, inter-GPU communication must traverse the PCIe bus, which has significantly higher latency and lower bandwidth than NVLink's direct GPU-to-GPU connections. Earlier in the session, the team had deployed and profiled the GLM-5-NVFP4 model using SGLang, and had found that 69% of decode time was spent on dtype-casting kernels — the overhead of converting NVFP4 weights to BF16 before computation. That bottleneck was so dominant that it masked all other performance issues.

When the team pivoted to Kimi-K2.5 INT4 on vLLM, they brought with them a set of assumptions shaped by the GLM-5 experience. The Marlin kernel, which fuses INT4 dequantization directly into the GEMM, eliminated the dtype-cast bottleneck entirely. But this created a puzzle: the model was still not achieving the throughput they hoped for. The initial hypothesis was that the bottleneck had shifted to the tiny MoE expert GEMMs, which become extremely small when sharded across 8 GPUs via tensor parallelism (TP=8). Each expert's weight matrix is divided into 8 slices, making each individual GEMM too small to fully utilize the GPU's compute units. This hypothesis drove an extensive investigation into SM120 GEMM optimization strategies — Marlin kernels, L2 cache pinning, persistent fused kernels, and column-major tile scheduling.

The assistant and user spent considerable effort researching these GPU-level optimizations before deciding to gather actual data. The message at index 2458 is the result of that data-gathering phase. It was written because the team recognized that they were operating on hypotheses, not facts, and that the only way to make progress was to instrument the system and measure exactly where time was being spent.

The Thinking Process: From Hypothesis to Revelation

The thinking process visible in the messages leading up to index 2458 reveals a rigorous scientific approach. The assistant designed a three-phase benchmarking plan:

Phase 1: Macro benchmarks — Measure end-to-end throughput and latency against the running vLLM server, establishing baseline metrics like tokens-per-output-token (TPOT) at various concurrency levels.

Phase 2: Micro-benchmarks — Isolate individual operations: Marlin W4A16 GEMMs at the exact dimensions used by Kimi-K2.5, NCCL AllReduce operations at the model's hidden size (14KB), and custom vLLM allreduce benchmarks.

Phase 3: Full torch.profiler capture — Run the model under a real inference workload while the PyTorch profiler records every CUDA kernel launch, its duration, and its caller. This produces a complete, unbiased picture of where GPU time is spent.

The assistant executed Phase 3 by launching a special profiler-enabled instance of vLLM (messages 2432-2442), waiting through the lengthy model loading process (~30 minutes for a 540GB model across 119 safetensor shards), sending warmup requests and then profiling requests, and finally retrieving the profiler output files. The profiler produced both detailed trace files (over 80MB each per GPU) and summary text files with aggregate kernel statistics.

The moment of revelation came when the assistant parsed the profiler output (message 2447). The analysis script revealed a startling breakdown: AllReduce accounted for 51.5% of decode time — over half of every inference step was spent on communication, not computation. The assistant's immediate reaction in message 2448 is telling: "This is the definitive data." The tone shifts from speculation to certainty.

The Assumption That Was Overturned

The most significant assumption that this message refutes is that the bottleneck is in the GEMM computations. The entire optimization strategy up to this point had been built around the idea that the MoE expert GEMMs, made tiny by TP=8 sharding, were the primary performance limiter. This was a reasonable hypothesis — MoE models are notorious for having "skinny" GEMMs that underutilize GPU tensor cores, and the Blackwell SM120 architecture has specific characteristics that make small GEMMs particularly problematic.

But the data showed something else entirely. The MoE GEMMs (handled by Marlin kernels) consumed only 2.40ms per decode step — just 11.1% of total CUDA time. The attention mechanism consumed 3.32ms (15.3%). The dominant cost was AllReduce at 11.17ms (51.5%). The assistant's analysis revealed that each decode step involved 127 NCCL AllReduce calls: 61 layers × 2 (one for attention, one for MoE) plus a handful of extras. At 78 microseconds per call, those 127 calls added up to 9.88ms. An additional custom vllm::all_reduce operation (using IPC shared memory for a single large allreduce, likely for the lm_head output) added another 1.16ms at 1125 microseconds per call — 14 times slower than NCCL.

The message's key insight — "eliminating the cast bottleneck revealed that AllReduce was always the real problem" — is a classic example of a masked bottleneck. When one bottleneck is so severe that it dominates all others (the 69% dtype-cast overhead in GLM-5), fixing it doesn't eliminate the performance problem; it simply reveals the next bottleneck in line. The team had experienced this phenomenon before with GLM-5, but they had assumed the next bottleneck would be compute-bound. Instead, it was communication-bound.

Input Knowledge Required to Understand This Message

To fully grasp the significance of this message, a reader needs substantial domain knowledge spanning several areas:

Model architecture knowledge: Understanding that Kimi-K2.5 is a Mixture-of-Experts (MoE) transformer with 61 layers, where each layer has both an attention mechanism and multiple expert feed-forward networks. The "routing" category refers to the gating mechanism that selects which experts to activate for each token.

Quantization knowledge: Understanding the difference between NVFP4 (NVIDIA's 4-bit floating point format, used by GLM-5) and INT4 with Marlin kernels. Marlin is a specialized GPU kernel that performs INT4 matrix multiplication with on-the-fly dequantization to BF16, fusing the dequantization step into the GEMM operation itself. This eliminates the separate dtype-cast kernel that was the bottleneck in GLM-5.

Distributed inference knowledge: Understanding tensor parallelism (TP), where each GPU holds a shard of every weight matrix, and all-reduce operations are needed to sum partial results across GPUs. Also understanding expert parallelism (EP), where each GPU holds a subset of experts, and all-to-all communication routes tokens to the correct expert GPU. The distinction between NCCL allreduce (used for most operations) and the custom vllm::all_reduce IPC path (used for specific large reductions) is also important.

Hardware knowledge: Understanding that the 8x RTX PRO 6000 Blackwell GPUs are connected only via PCIe, without NVLink. This means inter-GPU communication goes through the CPU's PCIe controller, which has limited bandwidth (~32 GB/s per direction for PCIe 5.0 x16) and higher latency than NVLink. The 78 microseconds per NCCL allreduce call reflects this PCIe bottleneck.

Profiling methodology knowledge: Understanding what a torch.profiler capture reveals — the aggregate CUDA kernel time across all streams and GPUs — and how to interpret the per-step breakdown. The assistant notes that the profiler's "per-step CUDA time" (21.7ms) is higher than the measured TPOT (12.1ms), which is expected because the profiler counts overlapping operations across streams while the wall-clock TPOT only counts the critical path.

Output Knowledge Created by This Message

This message creates several pieces of actionable knowledge:

A quantified bottleneck hierarchy: The team now knows exactly where every millisecond of decode time is spent. This replaces guesswork with data. The breakdown — 51.5% AllReduce, 15.3% Attention, 13.4% Other GEMMs, 11.1% MoE GEMMs, 4.5% Routing, 4.2% Everything else — provides a clear optimization priority list.

A validated comparison between GLM-5 and Kimi-K2.5: The message explicitly contrasts the two models' bottleneck profiles, showing how the same hardware produces radically different performance characteristics depending on the model format and engine. GLM-5 on SGLang was 69% dtype-cast overhead (3.4% HBM-efficient); Kimi-K2.5 on vLLM is 31.8% HBM-efficient. This comparison is valuable for future model selection decisions.

A specific optimization target: Expert Parallelism is identified as the single most impactful change. The message quantifies the expected benefit: replacing 60 MoE AllReduce operations with 60 All-to-All operations saves ~3.54ms per token (29% of TPOT). Additionally, EP makes the expert GEMMs 8× wider, which would better utilize the GPU's tensor cores and potentially reduce the "MoE GEMMs" category further.

A call to action: The message ends with the practical step of restarting the vLLM service in its normal configuration (without profiler overhead), setting the stage for the next phase of investigation. The team can now proceed to implement Expert Parallelism with confidence that it targets the actual bottleneck.

The Broader Significance

This message represents a turning point in the engineering campaign. Before it, the team was exploring a wide range of optimization strategies — Marlin kernel tuning, L2 cache pinning, persistent fused kernels, column-major tile scheduling — many of which would have addressed the wrong problem. After it, the optimization effort is sharply focused on reducing communication overhead.

The message also illustrates a fundamental principle of performance engineering: measure before optimizing. The team's initial hypothesis (tiny GEMMs are the bottleneck) was plausible and grounded in real architectural knowledge, but it was wrong. Only by instrumenting the system and collecting empirical data could they discover the true bottleneck. The assistant's disciplined approach — designing a three-phase benchmark plan, executing it methodically, and presenting the results clearly — transformed a guessing game into a data-driven optimization campaign.

The final sentence — "The vLLM service is restarting now (~30 min to load)" — is a reminder of the operational reality of working with large models. Each iteration of the optimization cycle requires waiting 30 minutes for the model to load onto the GPUs. This constraint makes it especially important to get the optimization direction right the first time, which is precisely why the profiling investment was worthwhile.

In the messages that follow (not shown here), the team pivots to investigating speculative decoding as a complementary optimization, recognizing that even with Expert Parallelism, the PCIe allreduce bottleneck will remain a fundamental constraint. But that pivot is only possible because this message first established, with clarity and precision, what the actual problem is.