Squeezing Every FLOP: The Optimization Plan for MoE Inference on Blackwell

Introduction

In the high-stakes world of large language model inference, few challenges are as frustrating as owning eight of the most powerful consumer-grade GPUs ever made—NVIDIA RTX PRO 6000 Blackwell Server Edition cards with 96GB of VRAM each—and watching them deliver less than one percent of their theoretical peak performance. This was the predicament facing the operator of a custom-built inference server running the 1-trillion-parameter Kimi-K2.5 INT4 model, a Mixture-of-Experts (MoE) architecture with 384 routed experts. After weeks of environment setup, driver installation, kernel debugging, and model deployment, the team had achieved a respectable 82 tokens per second for single-stream inference. But the user wanted more—they wanted to "squeeze out every last FLOP."

The message at the center of this analysis, <msg id=2403>, represents a pivotal moment in the conversation: the synthesis of extensive profiling data, prior experimental results, and deep architectural knowledge into a concrete, prioritized optimization plan. It is the moment where scattered observations crystallize into strategy. This article examines that message in depth—its reasoning, its assumptions, its knowledge requirements, and the thinking process that produced it.

The Context: A Conversation at an Inflection Point

To understand why <msg id=2403> was written, we must first understand the conversation that led to it. The session had been running for dozens of rounds across multiple days, covering everything from NVIDIA driver installation on Ubuntu 24.04 to the deployment of three different billion-parameter models. The current production setup was Kimi-K2.5 INT4, a 547GB model using compressed-tensors INT4 quantization, running with tensor parallelism across all eight GPUs (TP=8).

The immediate trigger was a user message that began, "If allreduce is so slow it seems like it would make sense to run two inferences at the same time..." ([msg 2398]). The user had noticed that allreduce—the communication step where GPUs synchronize their partial results after each layer—was consuming a significant portion of each decode step. Their intuition was sound: if one inference could compute while another was communicating, throughput could potentially double.

The assistant's initial response ([msg 2400]) was thorough but sobering. Research into vLLM and SGLang features revealed that while mechanisms like SGLang's Single Batch Overlap (SBO) and Two Batch Overlap (TBO) existed, they were not a magic bullet. More importantly, the assistant revealed a critical insight from prior experiments: allreduce was not the real bottleneck. Earlier profiling on the same hardware had shown that all communication optimizations—MSCCLPP, SBO, allreduce fusion, even halving the allreduce data with TP4+PP2—yielded at most 2% improvement. Allreduce consumed only 5-8% of total per-token time. The real culprit was something far more fundamental.

The user's response was immediate and visceral: "Aaah it's still the GEMMs! What would it take to properly optimize that for this GPU? Want to squeeze out every last FLOP" ([msg 2401]). This was the moment of clarity—the recognition that the bottleneck was not communication but computation, specifically the tiny matrix multiplications (GEMMs) at the heart of the MoE expert layers.

The assistant then launched three parallel research tasks ([msg 2402]): one analyzing the current MoE kernel paths in vLLM for SM120, one researching Blackwell-specific GEMM optimization techniques, and one scanning local project files for any existing profiling data. The results of these tasks, combined with extensive prior work, fed directly into the synthesis presented in <msg id=2403>.

The Message: A Synthesis of Everything Known

The subject message begins with a declaration that sets the tone: "This is a goldmine — we've already done extensive profiling and analysis on this exact problem." This is not a message that starts from scratch. It is a message that synthesizes—pulling together results from weeks of prior experiments, organizing them into a coherent framework, and using that framework to generate actionable next steps.

The message is structured in four distinct sections:

  1. The Problem Statement: A root-cause analysis of why the GPUs are achieving <1% of peak FP4 TFLOPS
  2. What Doesn't Help: A summary of optimization attempts that failed or produced negligible gains
  3. What Does Help: The two proven optimizations (OEA and EP8)
  4. The Optimization Plan: Five tiers of interventions, ranked by effort and expected impact This structure is itself revealing. The assistant is not proposing random experiments. It is building a decision tree based on empirical evidence, theoretical reasoning, and engineering judgment.

The Root Cause: Why Blackwell GPUs Starve on MoE

The problem statement in the message is brutally precise:

1. Kimi-K2.5 has 384 experts, TP=8 → each expert's down_proj GEMM is M×7168×256 where K=256 is pathetically small 2. At decode batch=1, M=1 per expert → 1×7168×256 = 3.7 MFLOP per GEMM. The GPU can do 1,850 TFLOPS. That's 2 nanoseconds of useful compute wrapped in microseconds of overhead. 3. Even at batch=256, M≈5 tokens/expert → still far too small to saturate anything

This analysis reveals a fundamental tension in MoE inference. The model has 384 experts, but only the top-8 are activated per token. With TP=8, each expert's weight matrices are sharded across all eight GPUs, making the K dimension (the inner dimension of the matrix multiplication) equal to the original hidden size divided by 8. For Kimi-K2.5, the down projection has shape 7168×2048 in the unsharded model; with TP=8, each GPU holds a 7168×256 slice. A GEMM of shape M×7168×256 with M=1 is a vector-matrix multiply of negligible arithmetic intensity.

The message's framing of "2 nanoseconds of useful compute wrapped in microseconds of overhead" is particularly effective. It captures the essence of the problem: the GPU's tensor cores, capable of 1,850 teraflops, are spending virtually all their time on kernel launch overhead, memory bandwidth stalls, and synchronization, rather than actual floating-point computation.

The Evidence Base: Learning from Failure

A crucial strength of this message is its honesty about what hasn't worked. The table of failed optimizations is worth examining in detail:

| Optimization | Result | Why | |---|---|---| | MSCCLPP (faster allreduce) | +1.9% | Allreduce is only ~8% of time | | Single Batch Overlap (SBO) | +1.2% | Same—comm isn't the bottleneck | | TP4+PP2 (halve allreduce) | 2x slower | Proved allreduce isn't the problem | | FlashInfer allreduce fusion | -87% (broken) | GDC doesn't work on SM120 | | CuteDSL MoE backend | -2.6 to -16.5% | Slower than CUTLASS on SM120 | | cuBLASLt FP4 | -5-8% vs CUTLASS | CUTLASS wins on SM120 |

This table serves multiple purposes. First, it establishes credibility—the assistant has already tried the obvious things. Second, it prevents wasted effort—no one needs to suggest "have you tried faster allreduce?" because that path has been explored and found wanting. Third, it narrows the search space: if allreduce optimization, pipeline parallelism, and alternative GEMM backends all fail, the remaining options must address the fundamental issue of GEMM dimensions.

The two proven optimizations—OEA (fewer unique experts loaded) and EP8 (expert parallelism)—point in a consistent direction: make the GEMMs larger. OEA reduces the number of experts that need to be loaded, improving memory bandwidth utilization. EP8 distributes experts across GPUs without sharding them, making each expert GEMM 8x wider. Both attack the same root cause from different angles.

The Five-Tier Plan: From Quick Wins to Kernel Engineering

The optimization plan is the heart of the message. It is organized into five tiers, each with specific interventions, estimated effort, and expected impact. The structure reveals a sophisticated understanding of engineering economics: the assistant is not just listing what could be done, but ranking options by return on investment.

Tier 1: Low-Hanging Fruit (Hours)

The first two recommendations are configuration changes that could be tested within hours:

Expert Parallelism retry (--enable-expert-parallel with --tensor-parallel-size 8): This is the most promising quick win. Expert parallelism makes each expert GEMM 8x wider by not sharding experts across GPUs. Instead, each GPU holds complete experts and uses all-to-all communication to route tokens to the correct GPU. The message estimates 20-50% improvement at high concurrency, based on the proven success of this approach for MiniMax-M2.5 (which achieved ~4,000 tok/s with TP=8+EP).

Switch to SGLang: SGLang has more mature SM120 MoE kernel paths, including persistent grouped GEMM and adaptive grid sizing. Benchmarks show 1.3-2.2x improvement over vLLM for FP4 grouped GEMM on Blackwell. The message is careful to note that this is a hypothesis—Kimi-K2.5 INT4 uses Marlin kernels on vLLM, and SGLang might use a different (possibly better) path.

Tier 2: Medium Effort, High Potential (Days)

The second tier introduces kernel-level optimizations that require code changes but are well-scoped:

L2 Cache Pinning: SM120 has 128MB of monolithic L2 cache—enough to hold approximately 10 complete experts. Expert routing follows a Zipf distribution (a few experts handle most tokens), so pinning hot expert weights in L2 could dramatically reduce memory latency. The message estimates 20-50% improvement for decode. The implementation is relatively modest: a ~100-line patch to set cudaAccessPolicyWindow hints before GEMM calls.

Persistent Fused MoE Kernel: Currently, each MoE layer requires three separate kernel launches: gate_up GEMM, SiLU activation, and down GEMM. A fused kernel would combine all three operations, reading weights once and keeping intermediate results in registers. The message estimates 2-5x decode speedup for the MoE portion and notes that it would eliminate ~300MB of intermediate HBM traffic per forward pass. The engineering estimate of 1-2 weeks reflects the complexity of writing a custom CUTLASS/Triton kernel.

Column-Major GEMM Tile Scheduling: This optimization reorders tile processing within the grouped GEMM to maximize L2 cache reuse. The PyTorch team measured +60% L2 hit rate and 1.33x speedup from this alone on H100. Applying GROUP_SIZE_M parameter tuning to the Triton/CUTLASS grouped GEMM configs could yield similar benefits on SM120.

Tier 3: High Effort, Potentially Transformative (Weeks)

The third tier represents serious kernel engineering:

Custom SM120 W4A16 Grouped GEMM Kernel: This would be a purpose-built kernel for the exact workload—INT4 weights, BF16 activations, group_size=32 quantization, SM120's 100KB shared memory constraint, persistent design with TMA descriptors, L2-aware tile scheduling, and GDC/PDL for overlapping adjacent launches. The message notes that this kernel could also fuse dequantization into the GEMM itself, eliminating the dtype-cast overhead that consumed 69% of decode time in earlier profiling.

NVFP4 Re-quantization: Converting the INT4 model to NVFP4 would use native FP4 tensor cores, which offer 4x Hopper FP8 throughput. However, the message is cautious—the expert GEMMs are still memory-bound at these dimensions, and SM120 NVFP4 software support is known to be buggy.

Tier 4: Architectural Changes

The final tier acknowledges that some optimizations require architectural changes to how inference is structured:

Two Batch Overlap (TBO) on SGLang: Running prefill and decode concurrently on different CUDA streams could overlap communication with computation—similar to the user's original idea, but implemented within a single model instance rather than across two replicas.

Speculative Decoding: Using a small (1-7B) dense model as a draft model, verified by Kimi-K2.5, would convert many serial single-token decodes into fewer batched verification passes, increasing the effective M dimension of expert GEMMs during verification.

The Recommendation: A Prioritized Path Forward

The message concludes with a clear recommendation:

1. EP retry for Kimi-K2.5 on vLLM     [~1 hour]  → likely +20-50% at high C
2. Switch to SGLang for Kimi-K2.5      [~2 hours] → better kernel paths
3. L2 cache pinning                    [~1 day]   → +20-50% decode
4. Column-major tile scheduling        [~1 day]   → +30% from L2 reuse  
5. Persistent fused MoE kernel         [~1-2 wk]  → 2-5x decode MoE portion
6. Custom SM120 W4A16 grouped GEMM     [~2-3 wk]  → eliminate dtype-cast overhead

This is not a random list. It is a carefully sequenced investment plan where each step generates information that informs the next. The quick configuration changes (items 1-2) will reveal whether Expert Parallelism is viable for Kimi-K2.5—if it works, it may obviate the need for deeper kernel work. If EP fails (due to memory constraints or all-to-all latency), the kernel-level optimizations (items 3-6) become the primary path.

The message ends with a question to the user: "Where do you want to start?" This is not rhetorical. The assistant has laid out the options and is asking for strategic direction. The user's answer—"Plan profiling to truly understand the current bottlenecks and potential"—would redirect the conversation toward measurement before action, a decision that would shape the subsequent messages.

Assumptions and Their Implications

The optimization plan rests on several key assumptions, some explicit and some implicit:

Assumption 1: The profiling data from GLM-5 generalizes to Kimi-K2.5. The message repeatedly references "our previous GLM-5 experiments" as the evidence base for what works and what doesn't. Both models use similar MoE architectures, but they are not identical. Kimi-K2.5 has 384 experts versus GLM-5's 256, and uses INT4 quantization versus GLM-5's NVFP4. The message acknowledges this implicitly by noting that the EP retry is needed precisely because the GLM-5 EP attempt OOM'd.

Assumption 2: Expert parallelism will not be bottlenecked by all-to-all communication. The message estimates 20-50% improvement from EP, but this assumes that the all-to-all communication required for EP does not become the new bottleneck. On PCIe-only interconnects (no NVLink), all-to-all can be significantly slower than allreduce. The message's caveat about EP being "possibly slower at C=1 due to PCIe all-to-all latency" shows awareness of this risk.

Assumption 3: L2 cache pinning is feasible and effective on SM120. The cudaAccessPolicyWindow API exists, but its behavior on Blackwell GPUs is not well-documented. The message estimates 20-50% improvement, but this is based on theoretical analysis rather than empirical measurement on this specific hardware.

Assumption 4: The user is willing to invest significant engineering effort. Items 5 and 6 require 1-3 weeks of kernel engineering. The message implicitly assumes that the user values performance enough to justify this investment. The user's response—opting for profiling first—suggests a more measured approach.

Knowledge Flow: Input and Output

To fully understand &lt;msg id=2403&gt;, one must recognize the extensive knowledge required to produce it and the new knowledge it creates.

Input Knowledge Required

The message draws on at least five distinct knowledge domains:

  1. MoE Architecture Knowledge: Understanding how expert routing, gating, and parallel sharding work in models like DeepSeek V3 and Kimi-K2.5. The message assumes familiarity with concepts like TP (tensor parallelism), EP (expert parallelism), PP (pipeline parallelism), and their interactions with MLA (Multi-head Latent Attention).
  2. GPU Architecture Knowledge: Deep understanding of SM120 (Blackwell) microarchitecture—shared memory sizes, L2 cache hierarchy, tensor core capabilities, TMA descriptors, GDC/PDL mechanisms. The message also requires knowledge of how these compare to SM90 (Hopper) and SM100 (datacenter Blackwell).
  3. Kernel Engineering Knowledge: Familiarity with CUTLASS, Triton, cuBLASLt, and Marlin kernel implementations. The message discusses specific kernel paths, their strengths and weaknesses on SM120, and the engineering effort required to modify them.
  4. Prior Experimental Results: The message synthesizes results from at least six prior experiments (MSCCLPP, SBO, TP4+PP2, FlashInfer fusion, CuteDSL, cuBLASLt) conducted over the course of the conversation. Without this context, the optimization plan would appear speculative rather than evidence-based.
  5. vLLM and SGLang Architecture: Understanding the configuration options, kernel backends, and runtime behavior of both inference engines. The message references specific flags (--enable-expert-parallel, --dp-size, --enable-single-batch-overlap) and their interactions.

Output Knowledge Created

The message creates several forms of new knowledge:

  1. A Prioritized Optimization Map: Before this message, the team had scattered data points and intuitions. After it, they have a structured plan with ranked interventions, effort estimates, and expected impacts. This transforms raw data into actionable strategy.
  2. A Decision Framework: By organizing optimizations into tiers and providing effort/impact estimates, the message enables the user to make informed trade-offs. The question at the end forces a strategic choice: quick wins or deep engineering?
  3. A Root-Cause Narrative: The message provides a coherent explanation for why the GPUs are underperforming. The chain "384 experts → TP=8 → tiny K=256 GEMMs → <1% peak utilization" is a causal story that explains all the observed behavior. This narrative is itself a form of knowledge—it shapes how future problems will be diagnosed.
  4. Empirical Negative Results: The table of failed optimizations is valuable knowledge. It prevents future wasted effort and establishes that the problem is not communication-bound but compute-bound at the kernel level.
  5. Testable Hypotheses: Each recommendation in the plan is a hypothesis with a predicted outcome. "EP retry will yield 20-50% improvement at high concurrency" is a testable claim. The message sets up a framework where the next step is measurement and validation.

The Thinking Process: Synthesis Under Uncertainty

The reasoning visible in &lt;msg id=2403&gt; reveals a sophisticated cognitive process. The assistant is not simply reporting facts; it is constructing an argument from heterogeneous evidence.

The first move is reframing the problem. The user initially believed allreduce was the bottleneck. The assistant's research showed otherwise. Now, in this message, the assistant completes the reframing by providing a new, more accurate problem statement: the GEMMs are too small. This reframing is essential because it redirects attention to the interventions that might actually work.

The second move is organizing evidence by strength. The message separates what has been proven (failed optimizations, successful optimizations) from what is hypothesized (the five-tier plan). This creates a clear distinction between knowledge and speculation, allowing the user to trust the former while evaluating the latter.

The third move is ranking by expected value. The message does not simply list options; it orders them by a combination of effort, expected impact, and information value. The quick wins (EP retry, SGLang switch) come first not because they have the highest potential impact, but because they can be tested quickly and their results inform the deeper work.

The fourth move is managing uncertainty through caveats. Every recommendation includes qualifications: "possibly slower at C=1," "needs retry with tuned memory," "SM120 NVFP4 software support is buggy." These caveats protect against overconfidence and signal that the assistant understands the limits of its knowledge.

Conclusion: A Blueprint for Performance Engineering

&lt;msg id=2403&gt; is a masterclass in applied performance engineering. It demonstrates how to take a complex, multi-dimensional optimization problem—squeezing performance from a 1T-parameter MoE model on eight cutting-edge GPUs—and reduce it to a structured, prioritized plan of attack.

The message succeeds because it is grounded in empirical evidence, honest about failures, precise about mechanisms, and clear about trade-offs. It does not promise miracles; it promises a path. The expected improvements—20-50% from EP, 20-50% from L2 pinning, 2-5x from fused kernels—are ambitious but not fantastical. They are grounded in the physics of the hardware and the mathematics of the workload.

For the reader who has not followed the conversation, this message serves as both a summary of everything learned and a roadmap for everything to come. It captures a moment of strategic clarity—the point at which scattered data crystallizes into direction. In the broader narrative of the coding session, &lt;msg id=2403&gt; is the message that defines the next phase of work: a shift from configuration tuning to kernel engineering, from surface-level optimization to deep architectural understanding.

The user's response—choosing profiling over immediate action—would prove to be the right call. The subsequent profiling campaign would reveal that AllReduce actually accounted for 51.5% of decode time, not the 5-8% estimated from earlier experiments. This discovery would force yet another reframing of the problem, demonstrating that even the best synthesis is only as good as the data it's built on. But that is the nature of performance engineering: it is an iterative process of measurement, hypothesis, and refinement. &lt;msg id=2403&gt; represents the hypothesis stage of that cycle—comprehensive, well-reasoned, and ready to be tested against reality.