The Micro-Benchmark That Unmasked the Bottleneck: Profiling Kimi-K2.5 INT4 on Blackwell

In the middle of an intensive profiling campaign targeting a 1-trillion-parameter Mixture-of-Experts model running on eight NVIDIA RTX PRO 6000 Blackwell GPUs, a single command was issued that would reshape the entire optimization strategy. Message [msg 2424] is the execution of a micro-benchmark script on a remote server — a seemingly mundane SSH command that, in the context of the broader conversation, represents a critical pivot from macro-level throughput measurements to granular component-level latency analysis. This message is the moment when the assistant stopped treating the model as a black box and began dissecting it operation by operation.

The Strategic Context: A Three-Phase Profiling Campaign

To understand why this message was written, one must trace back through the preceding conversation. The assistant and user had been engaged in a multi-week effort to deploy and optimize large language models on a system with eight RTX PRO 6000 Blackwell GPUs connected only via PCIe — no NVLink, no high-speed inter-GPU fabric. After successfully deploying the Kimi-K2.5 INT4 model and achieving approximately 60 tokens per second in single-request throughput, the team faced a fundamental question: what is actually consuming the GPU time during each decode step?

The earlier profiling work on a different model (GLM-5 NVFP4) had revealed that 69% of decode time was spent on dtype casting — converting FP4 weights to BF16 before GEMM operations. But Kimi-K2.5 INT4 used a completely different kernel path: Marlin W4A16 kernels that fuse dequantization directly into the matrix multiplication. The assistant hypothesized that Marlin might eliminate the dtype-cast bottleneck entirely, potentially revealing a completely different performance bottleneck underneath.

This hypothesis drove the creation of a three-phase benchmarking plan in [msg 2407]. Phase 1 would measure macro-level throughput and latency against the running vLLM server. Phase 2 would run micro-benchmarks of individual GEMM operations at the exact dimensions used by Kimi-K2.5, plus NCCL AllReduce burst measurements. Phase 3 would capture a full torch.profiler trace for deep timeline analysis. Message [msg 2424] is the execution of Phase 2's GEMM micro-benchmarks.

The Decision to Stop the Production Server

A critical decision preceded this message: the assistant chose to stop the running vLLM production server to free GPU memory for micro-benchmarking. In [msg 2422], the assistant executed systemctl stop vllm-kimi-k25-int4 and killed all residual processes, verifying that all eight GPUs showed 0 MiB memory usage. This was a non-trivial choice — it meant taking down a working inference service, losing the model's loaded state (which took ~30 minutes to restore), and accepting downtime. The alternative would have been to run micro-benchmarks on a separate machine or to use one of the eight GPUs while the others served requests. But the assistant correctly recognized that the micro-benchmarks needed full GPU access and that the Marlin kernel behavior might depend on the exact memory layout and GPU state present during inference.

What the Micro-Benchmark Script Measured

The script bench_k25_micro.py, written in [msg 2416], was designed to measure individual component latencies at the exact dimensions used by Kimi-K2.5 with tensor parallelism of 8. The model has a hidden size of 7168 and an MoE intermediate size of 2048. With TP=8, each GPU handles only a fraction of the full matrix dimensions: the gate and up projections see N=512 (2048/8/2 for the fused projection), and the down projection sees K=256 (2048/8).

The script benchmarked two categories of GEMM operations:

1. BF16 GEMM via torch.mm — the baseline, representing what the computation would look like without quantization. This measured the raw matrix multiplication performance at various batch sizes (M=1 through M=256).
2. Marlin W4A16 grouped GEMM — the actual kernel path used by the deployed model. Marlin is a specialized kernel designed for INT4 weight quantization with BF16 activations, performing dequantization and matrix multiplication in a single fused operation. The script tested this at the exact gate_up and down projection dimensions. The output visible in the message shows only the header and first few lines of results, but the full output (which appears in subsequent messages) would reveal the per-operation latency in microseconds for each combination of M, N, and K dimensions.

The Assumptions Embedded in the Benchmark Design

Several assumptions shaped the design of this micro-benchmark. First, the assistant assumed that individual GEMM operations could be meaningfully isolated from the larger computational graph — that measuring a single torch.mm or Marlin kernel call in isolation would yield latencies that sum to the total decode step time. This is a standard assumption in performance engineering but carries risks: CUDAGraph replay, kernel launch overhead, and memory bandwidth contention can all cause isolated benchmarks to differ from in-situ performance.

Second, the benchmark assumed that the Marlin W4A16 kernel path was the dominant computational component. The script did not measure attention (MLA), RMSNorm, residual connections, or the routing network — only the GEMM operations. This reflected the assistant's prior experience with GLM-5 NVFP4, where GEMM-related operations (including the dtype casting) dominated the profile. As the subsequent profiling would reveal, this assumption was partially wrong: AllReduce, not GEMM, would emerge as the primary bottleneck.

Third, the assistant assumed that running the benchmark on a single GPU (via /root/ml-env/bin/python3, not torchrun) was sufficient. The micro-benchmarks used only one GPU, not all eight. This was a practical decision — the NCCL AllReduce benchmark (which required all eight GPUs) was a separate script — but it meant the micro-benchmarks could not capture the communication overhead that would prove to be the dominant cost.

The Knowledge Required to Interpret This Message

Understanding this message requires substantial domain knowledge. The reader must know what "Marlin W4A16" means — a kernel that operates on INT4-quantized weights with group size 32, dequantizing them on-the-fly during matrix multiplication. They must understand tensor parallelism (TP=8) and how it shards the model's weight matrices across GPUs: the gate projection's N dimension of 2048 becomes 512 per GPU because the fused gate+up projection is split across 8 GPUs and further divided between the two projections. They must know the Kimi-K2.5 architecture: a 1T-parameter MoE model with 61 transformer layers, 60 MoE layers (each with 8 experts, top-2 routing), hidden size 7168, and intermediate size 2048. And they must understand the hardware: SM120 Blackwell architecture, GDDR7 memory, and the critical absence of NVLink between GPUs.

The Output Knowledge Created

This message produced the first concrete data about where decode time actually goes in the Kimi-K2.5 INT4 inference path. The micro-benchmark results would reveal that the Marlin W4A16 kernels were highly efficient — far more so than the FP4→BF16 dtype casting that dominated the GLM-5 profile. The per-operation latencies at various batch sizes would feed directly into the bottleneck analysis, allowing the assistant to estimate how much of each decode step was spent on GEMM versus communication versus other operations.

More importantly, this data would set up the surprise that followed. When the torch.profiler capture (Phase 3) eventually completed, it would reveal that AllReduce consumed 51.5% of decode time — 11.17 milliseconds per step — while the GEMM operations that the micro-benchmarks measured so carefully accounted for a much smaller fraction. The micro-benchmarks were not wrong; they simply measured the wrong thing. The bottleneck was not in computation but in communication.

The Thinking Process: What the Assistant Expected vs. What It Found

The assistant's reasoning in the preceding messages reveals a clear expectation: that the Marlin kernels would eliminate the dtype-cast bottleneck, and the remaining optimization target would be the MoE expert GEMMs themselves. The profiling plan in [msg 2407] listed "Marlin W4A16 GEMM" as the first item under MoE Expert GEMMs, with "dtype cast / dequant overhead" as a sub-item to be measured. The assistant expected to find that the tiny GEMMs caused by TP=8 sharding — gate_up with N=512 and down with K=256 — were the primary performance limiter.

This expectation was reasonable given the GLM-5 NVFP4 experience, where dtype casting dominated. But it overlooked the PCIe communication cost. Each decode step requires 122 allreduce operations (61 layers × 2 for attention + MoE), each moving 14,336 bytes (hidden_size=7168 in BF16) across the PCIe bus. With eight GPUs sharing a single PCIe root complex, these allreduce operations serialize and compete for bandwidth, creating a bottleneck that no amount of GEMM optimization can fix.

The micro-benchmark message represents the moment when the assistant was still operating under this GEMM-centric assumption. The data it produced would be valuable — confirming that Marlin was indeed efficient — but it would also be incomplete, missing the communication cost that would ultimately define the system's performance ceiling.

Conclusion: A Message That Changed the Trajectory

Message [msg 2424] is, on its surface, a simple command execution: SSH into a remote machine, run a Python script, capture the output. But in the narrative of this coding session, it represents the transition from macro-level observation to micro-level investigation, from treating the model as a black box to dissecting it operation by operation. The data it produced would confirm one hypothesis (Marlin eliminates dtype-cast overhead) while setting the stage for a surprising discovery (AllReduce is the real bottleneck). This discovery would, in turn, drive the team toward speculative decoding as a software-only optimization path — a pivot that would define the remainder of the session.