The Art of Measurement: A Profiling Campaign for Blackwell MoE Inference

Introduction

In the high-stakes world of large-scale AI inference, the gap between theoretical GPU peak performance and real-world throughput can be staggering. When a team deploys a 1-trillion-parameter Mixture-of-Experts model like Kimi-K2.5 on eight NVIDIA RTX PRO 6000 Blackwell GPUs, they might expect to see impressive numbers from hardware that boasts nearly 2 petaFLOPs of compute. Instead, they find themselves achieving less than 1% of peak utilization on the very operations that should be the GPU's forte—matrix multiplications. This is the reality that confronts engineers working at the frontier of model deployment, and it is the problem that message [msg 2407] sets out to solve, not by immediately writing optimization code, but by first doing something far more valuable: measuring.

Message [msg 2407] is a pivotal moment in an extended conversation spanning dozens of rounds across multiple sessions. The assistant, having just proposed an ambitious multi-tier optimization plan ranging from quick configuration changes to custom kernel engineering, receives the user's choice: "Plan profiling to truly understand the current bottlenecks and potential." This message is the assistant's response—a comprehensive, three-phase profiling campaign designed to illuminate exactly where every millisecond goes during a single decode step of Kimi-K2.5 INT4 on vLLM. It represents a deliberate pivot from action to understanding, from optimization to diagnosis.

The Context That Demanded Measurement

To appreciate the significance of this message, one must understand the journey that preceded it. The conversation began with deploying GLM-5-NVFP4, a 744B-parameter MoE model, on eight RTX PRO 6000 Blackwell GPUs. The team had resolved countless issues: flash-attn compilation failures, CUDA version mismatches, GGUF loader bugs, Triton MLA attention backend defects, and shard ordering problems. They had achieved a working deployment, optimized single-request throughput from ~20 to ~57 tok/s, and created systemd services for production use.

But then came the pivot to Kimi-K2.5-NVFP4, a 1T-parameter model, and subsequently to its INT4 quantized variant. The team had already conducted extensive profiling during the GLM-5 phase, which revealed a shocking finding: 69% of decode time was spent on dtype casting (unrolled_elementwise_kernel), not on actual computation or communication. This was the NVFP4/SGLang path with FlashInfer CUTLASS kernels, and it shaped everyone's assumptions about where the bottlenecks lay.

When the user asked what it would take to "squeeze out every last FLOP," the assistant dove deep into research, spawning multiple parallel research agents to investigate SM120 GEMM optimization strategies, Blackwell kernel paths, and the state of MoE expert computation in vLLM and SGLang. The resulting analysis ([msg 2403]) was comprehensive: a five-tier optimization plan ranging from Expert Parallelism retries to custom SM120 W4A16 grouped GEMM kernels. But the user wisely chose not to jump into implementation. Instead, they asked for profiling first—a decision that this message responds to with a meticulously structured plan.

The Message: A Profiling Campaign in Three Phases

The subject message opens with a crucial observation that frames the entire investigation: "Kimi-K2.5 INT4 uses a completely different kernel path: W4A16 via Marlin kernels (CompressedTensorsWNA16MarlinMoEMethod) on vLLM." This is the key insight that justifies the profiling effort. The previous bottleneck analysis from GLM-5—where dtype casting dominated—may be entirely irrelevant to the current setup because Marlin kernels fuse INT4 dequantization into the GEMM itself. The assistant is explicitly warning against cargo-culting previous findings onto a different configuration.

The plan is organized into three phases, each with a distinct methodology and set of questions:

Phase 1: Macro-Level Profiling

The first phase uses vLLM's built-in HTTP profiling API—a zero-downtime approach that works with the already-running server. By sending curl commands to /start_profile and /stop_profile, the assistant can capture a kernel-level breakdown of decode steps without restarting the service. This phase answers three fundamental questions:

  1. Does Marlin INT4 suffer from the same dtype-cast overhead that plagued FlashInfer CUTLASS FP4? This is the single most important question, because if Marlin eliminates that overhead, the entire optimization strategy changes.
  2. What is the per-layer breakdown? Attention (MLA) vs MoE expert GEMMs vs allreduce vs routing—each component's contribution to total time.
  3. How many kernel launches per decode step? Kernel launch overhead can be significant when hundreds of tiny GEMMs are launched serially.
  4. Is torch.compile/CUDAGraph hiding individual kernel costs? CUDAGraph replays can amortize launch overhead, but they also make individual kernel costs invisible to naive measurement. The estimated setup time is 30 minutes, with an additional 36 minutes for model loading—a realistic acknowledgment that profiling has a cost.

Phase 2: Micro-Level GEMM Characterization

The second phase moves from the whole-system view to individual operations. The assistant proposes benchmarking the actual Marlin W4A16 grouped GEMM at the exact dimensions Kimi-K2.5 uses. This is where the mathematics of the model architecture meets the physics of the GPU:

Phase 3: nsys Deep Dive

The third phase is the most invasive but potentially the most revealing. Using NVIDIA's nsys profiler with --cuda-graph-trace=node, the assistant proposes to look inside CUDAGraph replays—something that the higher-level torch.profiler cannot do. This requires restarting vLLM with special profiling configuration, or alternatively disabling CUDAGraph entirely with --compilation-config '{"level": 0}' for the clearest possible view. The goal is to capture 5-10 decode steps at batch sizes 1 and 64, then analyze:

The Deliverable: A Bottleneck Breakdown Table

Perhaps the most concrete element of the plan is the proposed deliverable: a detailed breakdown table showing time spent in each component at three concurrency levels (C=1, C=64, C=256). The table includes:

Assumptions and Knowledge Requirements

This message makes several implicit assumptions that are worth examining:

Assumption 1: Marlin kernels eliminate the dtype-cast overhead. The assistant explicitly flags this as a hypothesis to test, not a known fact. The careful phrasing—"Marlin kernels are purpose-built for INT4→BF16 dequant-fused GEMM—they might not have the same dtype-cast overhead"—shows appropriate epistemic humility.

Assumption 2: The HTTP profiling API works correctly with the running vLLM instance. This depends on vLLM version and configuration. If the profiling endpoints are not enabled or if CUDAGraph interferes with the profiler, this approach could fail.

Assumption 3: The model can be reloaded for nsys profiling. The 36-minute model load time is a significant barrier. If the team needs to iterate quickly, this cost could be prohibitive.

Assumption 4: The bottleneck distribution changes with concurrency. The proposed measurement at C=1, C=64, and C=256 assumes that different components scale differently with batch size—a reasonable assumption given that GEMMs become more compute-bound at larger M dimensions while allreduce overhead per token decreases.

The input knowledge required to understand this message is substantial. One must understand:

Decision-Making and Reasoning

The structure of this message reveals a sophisticated decision-making process. The assistant could have proposed jumping straight to optimization—after all, the user had just asked about "squeezing out every last FLOP." Instead, the message builds a case for measurement first, using three rhetorical strategies:

  1. Acknowledging uncertainty. The assistant explicitly notes that "we have no profiling data whatsoever for the current vLLM + Kimi-K2.5 INT4 setup." This honest admission of ignorance justifies the profiling effort.
  2. Leveraging prior knowledge while avoiding overgeneralization. The GLM-5 profiling results are presented as context, but the assistant is careful to note that the kernel path is "completely different." This prevents the team from making decisions based on outdated or irrelevant data.
  3. Providing a concrete deliverable. The bottleneck breakdown table gives the user a clear picture of what the output will look like. It transforms profiling from an abstract "let's see what happens" into a specific "we will produce this table with these columns." The execution order is also carefully considered: start with zero-downtime HTTP profiling, then run microbenchmarks in parallel with the running server, and only if needed restart for nsys. This minimizes disruption to any production workload while maximizing information gain.

Output Knowledge and Significance

What does this message create? On the surface, it creates a plan—a sequence of steps to execute. But more fundamentally, it creates a framework for understanding the performance characteristics of Kimi-K2.5 INT4 on Blackwell GPUs. The plan encodes a theory of where the bottlenecks might be (Marlin GEMMs, allreduce, kernel launch overhead) and provides the tools to test that theory.

The message also creates epistemic discipline. By committing to measurement before optimization, the assistant and user avoid the common trap of optimizing the wrong thing. The 69% dtype-cast overhead from GLM-5 was a red herring—it led the team to focus on communication optimizations that yielded 0-2% improvement. This profiling plan is designed to prevent a repeat of that mistake.

Perhaps most importantly, the message creates reproducibility. The three-phase plan, with its specific curl commands, dimension parameters, and analysis methodology, can be executed by anyone with access to the same hardware and model. The results will be comparable across different configurations, enabling evidence-based decisions about where to invest engineering effort.

Conclusion

Message [msg 2407] is a masterclass in the discipline of performance engineering. In a field where the temptation is always to jump to optimization—to write a kernel, to tune a parameter, to try a new configuration—the assistant instead proposes to measure. The plan is thorough, acknowledging the costs (36 minutes for model reload) and the uncertainties (Marlin might eliminate the old bottleneck). It builds on prior knowledge while remaining open to surprise.

The message also reveals something about the nature of the problem. When you are deploying a 1T-parameter model on eight GPUs connected only by PCIe, with 384 experts sharded across them, the bottlenecks are not where you expect them to be. The previous round of optimization taught the team that allreduce was not the problem—it was only 5-8% of time. The dtype casting was the problem, but that was specific to the NVFP4 path. Now, with a different quantization format and a different kernel backend, the entire bottleneck landscape may have shifted.

This is why measurement matters. Without the profiling campaign proposed in this message, any optimization effort would be guesswork—educated guesswork, perhaps, but guesswork nonetheless. With it, the team will know exactly where every millisecond goes, and they can target their efforts with surgical precision. The message is not just a plan for profiling; it is a commitment to evidence-based engineering in a domain where the evidence is often hidden behind layers of abstraction, from CUDAGraph to NCCL to the MoE routing algorithm itself.

The next step is execution. But the most important step—the decision to measure first—has already been taken.