The Optimization Odyssey: How a 1T-Parameter MoE Deployment on 8x Blackwell GPUs Revealed the True Cost of PCIe Inference

Introduction

In the high-stakes world of large language model inference, few challenges are as frustrating as owning eight of the most powerful consumer-grade GPUs ever made—NVIDIA RTX PRO 6000 Blackwell Server Edition cards with 96GB of VRAM each—and watching them deliver less than one percent of their theoretical peak performance. This was the predicament facing the operator of a custom-built inference server running the 1-trillion-parameter Kimi-K2.5 INT4 model, a Mixture-of-Experts (MoE) architecture with 384 routed experts. After weeks of environment setup, driver installation, kernel debugging, and model deployment, the team had achieved a respectable 82 tokens per second for single-stream inference. But the user wanted more—they wanted to "squeeze out every last FLOP."

What followed was a remarkable optimization odyssey spanning dozens of conversation rounds, multiple hypothesis reversals, and a comprehensive three-phase profiling campaign that ultimately revealed a surprising truth: the dominant bottleneck was not the tiny expert GEMMs everyone assumed it was, nor the dtype-cast overhead that had plagued earlier models—it was AllReduce communication across the PCIe bus, consuming 51.5% of every decode step. This article traces that journey from initial assumption through systematic investigation to final discovery, examining the reasoning, methodology, and hard-won insights that emerged along the way.

The Spark: A User's Insightful Question

The conversation reached a critical inflection point when the user, digesting preliminary profiling data, asked a question that would launch an entire investigation. In message [msg 2398], the user wrote: "If allreduce is so slow it seems like it would make sense to run two inferences at the same time, one does allreduce while the other computes. Is that something vllm or sglang support?"

This question was deceptively sophisticated. The user had correctly identified that AllReduce—the collective communication operation where GPUs synchronize partial results after each tensor-parallel layer—was consuming a significant portion of each decode step. Their intuition was to hide this latency by overlapping communication with computation, a classic technique in high-performance computing. Run two independent inference executions, stagger their communication and compute phases, and the PCIe bottleneck effectively disappears.

The assistant recognized the quality of the question immediately. In [msg 2399], it validated the user's insight ("Great question"), reframed it in precise technical language ("You're describing pipeline parallelism across model replicas—overlapping the allreduce latency of one request batch with the compute of another"), and launched a research subagent to investigate what vLLM and SGLang actually supported. This pattern—validation, reframing, delegation—would prove characteristic of the assistant's approach throughout the session.

The First Reversal: Allreduce Wasn't the Bottleneck

The research results, delivered in [msg 2400], contained a bombshell that upended the user's premise. After investigating disaggregated prefill, data parallelism, SGLang's Single Batch Overlap (SBO) and Two Batch Overlap (TBO) features, and vLLM's --dp-size flag, the assistant concluded: "The uncomfortable truth from our earlier experiments: All communication optimizations we tested (MSCCLPP, SBO, allreduce fusion, even TP4+PP2 which halves allreduce data) yielded 0-2% improvement. The profiling showed allreduce is only 5-8% of total time."

The real bottleneck, the assistant revealed, was something far more fundamental. With 384 experts and tensor parallelism across 8 GPUs (TP=8), each expert's weight matrices were sharded into 8 slices. The down_proj GEMM—the matrix multiplication at the heart of each expert's computation—became a pathetically small M×7168×256 operation. At decode batch size 1, where M=1 per expert, this meant each GEMM performed a mere 3.7 million FLOPs on a GPU capable of 1,850 trillion FLOPs. The GPU was spending nanoseconds of useful compute wrapped in microseconds of kernel launch overhead and memory bandwidth stalls.

This revelation reframed the entire optimization problem. The user's response in [msg 2401] captured the mix of frustration and determination perfectly: "Aaah it's still the GEMMs! What would it take to properly optimize that for this GPU? Want to squeeze out every last FLOP." The word "still" is telling—it acknowledges that GEMMs had been a known concern earlier in the project but had been deprioritized relative to the allreduce narrative. The "Aaah" is the sound of cognitive dissonance resolving: the user had been thinking about overlapping communication with computation, but the real problem was that the computation itself was inefficient.

The Research Pivot: Investigating SM120 GEMM Optimization

The assistant's response in [msg 2402] launched three parallel research subagents to investigate the state of the art in optimizing tiny MoE expert GEMMs for NVIDIA's SM120 Blackwell architecture. The research covered CUTLASS 4.x support, Marlin kernel paths, persistent kernel approaches, L2 cache pinning strategies, and column-major tile scheduling—a comprehensive sweep of the Blackwell optimization landscape.

The synthesis of this research, delivered in [msg 2403], was a masterclass in performance engineering. The assistant produced a five-tier optimization plan ranging from quick configuration changes (Expert Parallelism retry, switching to SGLang) through medium-effort kernel optimizations (L2 cache pinning, column-major tile scheduling) to multi-week kernel engineering projects (persistent fused MoE kernels, custom SM120 W4A16 grouped GEMM kernels). Each tier included estimated effort, expected impact, and the evidence base supporting the estimate.

The plan was grounded in a precise root-cause analysis: "Kimi-K2.5 has 384 experts, TP=8 → each expert's down_proj GEMM is M×7168×256 where K=256 is pathetically small. At decode batch=1, M=1 per expert → 1×7168×256 = 3.7 MFLOP per GEMM. The GPU can do 1,850 TFLOPS. That's 2 nanoseconds of useful compute wrapped in microseconds of overhead." This framing—2 nanoseconds of compute in microseconds of overhead—captured the essence of the problem with devastating clarity.

The Pivot to Measurement: Profiling Before Optimizing

When the assistant asked where to start, the user made a decision that would prove crucial: "Plan profiling to truly understand the current bottlenecks and potential." Rather than diving into implementation, the user insisted on measurement first. The assistant embraced this choice in [msg 2404]: "Smart approach—measure first, optimize second."

This pivot was not trivial. The assistant had just spent considerable effort developing a detailed optimization plan with specific speedup estimates. Setting that plan aside to gather data required intellectual discipline. But the assistant recognized that the plan was built on extrapolation from earlier GLM-5 NVFP4 profiling—and the Kimi-K2.5 INT4 model used a completely different kernel path. The Marlin W4A16 kernels fused INT4 dequantization into the GEMM itself, potentially eliminating the 69% dtype-cast overhead that had dominated the GLM-5 profile. If that was true, the entire bottleneck landscape could be different.

The assistant designed a comprehensive three-phase profiling campaign ([msg 2407]). Phase 1 used vLLM's HTTP profiling API to capture a kernel-level breakdown of decode steps without restarting the server. Phase 2 ran standalone micro-benchmarks of the Marlin W4A16 grouped GEMM at exact Kimi-K2.5 dimensions, plus NCCL AllReduce burst measurements. Phase 3, if needed, would restart vLLM with NVIDIA's nsys profiler for deep GPU timeline analysis inside CUDAGraph replays.

The user's response in [msg 2408] was succinct and decisive: "proceed with all benchmarks, write down results into k25b6000bench1.md as you gather them." This single sentence authorized the entire campaign and established a documentation convention that would preserve every finding.

The Execution: From Planning to Discovery

The execution of the profiling campaign was itself a journey of discovery. The assistant began by checking the running vLLM service ([msg 2410]), verifying that the model was loaded and responsive. Initial macro-benchmarking ([msg 2420]) revealed a plateau at 1536 tokens—a puzzling throughput ceiling that hinted at deeper bottlenecks. The assistant then freed GPU resources by stopping the vLLM service ([msg 2422]) to run micro-benchmarks without interference.

The micro-benchmarks ([msg 2424]) were revealing. The Marlin W4A16 GEMM at Kimi-K2.5 dimensions showed that the dtype-cast overhead was indeed eliminated—the INT4→BF16 dequantization was fused directly into the GEMM, as the assistant had hypothesized. But this created a puzzle: if the GEMMs were efficient, why was throughput still limited?

The NCCL AllReduce burst benchmark ([msg 2426]) provided the first clue. At the model's message size (14KB per allreduce), each NCCL call took approximately 78 microseconds. With 122 allreduces per decode step (61 layers × 2 for attention + MoE), that added up to roughly 9.5ms of pure communication time per token. This was far higher than the 5-8% estimate from the GLM-5 experiments.

The definitive answer came from the full torch.profiler capture ([msg 2442]-[msg 2448]). After restarting vLLM with profiling enabled and waiting through the 30-minute model loading process, the assistant sent profiling requests and retrieved the output. The analysis script parsed the profiler summaries and produced a breakdown that would reshape the entire optimization strategy.

The Revelation: AllReduce at 51.5%

The profiler results, synthesized in [msg 2458], delivered a stunning finding:

| Category | Per-step (ms) | % of Total | |---|---|---| | AllReduce (NCCL + custom) | 11.17 | 51.5% | | Attention | 3.32 | 15.3% | | Other GEMMs (cuBLAS) | 2.90 | 13.4% | | MoE GEMMs (Marlin) | 2.40 | 11.1% | | Routing | 0.99 | 4.5% | | Everything else | 0.92 | 4.2% |

AllReduce was consuming over half of every decode step. The MoE GEMMs that everyone had assumed were the bottleneck accounted for only 11.1%. The Marlin kernels had done their job—they eliminated the dtype-cast overhead that plagued the earlier GLM-5 NVFP4 runs. But in doing so, they unmasked the true bottleneck: PCIe-bound AllReduce communication across 8 GPUs.

The assistant's analysis revealed the mechanism: each decode step involved 127 NCCL AllReduce calls at 78 microseconds each, totaling 9.88ms. An additional custom vllm::all_reduce operation (using IPC shared memory for a single large reduction, likely for the lm_head output) added another 1.16ms at 1125 microseconds per call—14 times slower than NCCL.

The message's key insight was profound: "The biggest surprise: Marlin W4A16 kernels have ZERO dtype-cast overhead. Remember how GLM-5 on SGLang spent 69% of decode time on unrolled_elementwise_kernel (dtype casting)? That's completely gone with Marlin. But eliminating the cast bottleneck revealed that AllReduce was always the real problem—it was just hidden behind the even-larger cast overhead."

This is a classic example of a masked bottleneck. When one bottleneck is so severe that it dominates all others (the 69% dtype-cast overhead in GLM-5), fixing it doesn't eliminate the performance problem; it simply reveals the next bottleneck in line. The team had experienced this phenomenon before, but they had assumed the next bottleneck would be compute-bound. Instead, it was communication-bound.

The Aftermath: Accepting the AllReduce Tax

The discovery of the AllReduce bottleneck forced a fundamental reassessment of the optimization strategy. The five-tier plan focused on GEMM optimization was no longer the right approach—even perfect GEMM kernels would leave 51.5% of the performance on the table.

The assistant identified Expert Parallelism (EP) as the single most impactful change. EP would replace 60 MoE AllReduces with 60 All-to-All operations (19 microseconds vs 78 microseconds at decode batch=1), saving approximately 3.54ms per token (29% of TPOT). Additionally, EP would make each expert GEMM 8× wider, better utilizing the GPU's tensor cores. The assistant noted that this was a single --enable-expert-parallel flag.

But Expert Parallelism faced its own challenges. The user correctly observed in [msg 2459] that without NVLink, all-to-all communication on PCIe would face serialization issues at higher batch sizes. Each all-to-all operation requires every GPU to send data to every other GPU—a pattern that scales quadratically with the number of GPUs. On PCIe without NVLink, this could become a new bottleneck.

The user also made a principled stand in [msg 2461]: they declined precision-compromising approaches. When the assistant explored options like FP4 re-quantization or reduced-precision communication, the user rejected them, prioritizing model quality over throughput. This decision established a "quality floor"—a line below which optimization would not be allowed to compromise the model's output.

The conversation ultimately arrived at a sobering conclusion: the AllReduce bottleneck was a physical constraint of the PCIe-only hardware topology. Without NVLink, inter-GPU communication would always be the limiting factor for tensor-parallel inference with large MoE models. The team could optimize around it—Expert Parallelism, speculative decoding, careful batch-size tuning—but they could not eliminate it.

The Final Pivot: Speculative Decoding

With the hardware constraints accepted, the conversation pivoted to speculative decoding as the most promising software-only optimization path. In [msg 2468], the assistant began researching draft model candidates—smaller, faster models that could generate candidate tokens for Kimi-K2.5 to verify in parallel. The idea was elegant: use a 1-7B parameter dense model (like Qwen2.5-7B or DeepSeek-R1-Distill) to generate multiple candidate tokens cheaply, then have Kimi-K2.5 verify them in a single batched forward pass. This would convert many serial single-token decodes into fewer batched verification passes, increasing the effective M dimension of expert GEMMs during verification and reducing the number of AllReduce operations per accepted token.

The assistant launched parallel research subagents to investigate draft model candidates, vLLM/SGLang support for speculative decoding, and the feasibility of training a custom distilled draft model. This pivot marked the transition from hardware-constrained optimization to algorithmic innovation—a recognition that the path forward lay not in fighting the PCIe bottleneck, but in working around it.

Lessons Learned

The optimization odyssey of Kimi-K2.5 INT4 on 8x Blackwell GPUs yielded several enduring lessons:

1. Measure before optimizing. The team's initial hypothesis (tiny GEMMs are the bottleneck) was plausible and grounded in real architectural knowledge, but it was wrong. Only by instrumenting the system and collecting empirical data could they discover the true bottleneck. The three-phase profiling campaign was an investment that paid for itself many times over by preventing wasted effort on the wrong optimizations.

2. Bottlenecks mask other bottlenecks. The 69% dtype-cast overhead in GLM-5 was so dominant that it hid the AllReduce bottleneck underneath. Fixing it didn't eliminate the performance problem—it revealed the next constraint. This is a fundamental principle of systems optimization: every fix reveals the next bottleneck.

3. Hardware constraints are not negotiable. The PCIe-only interconnect without NVLink was a hard physical constraint that no amount of software optimization could eliminate. The team could work around it (Expert Parallelism, speculative decoding) but could not remove it. Accepting this reality early would have saved considerable effort.

4. Model quality is a non-negotiable constraint. The user's principled refusal to compromise precision—even when it meant accepting lower throughput—established a clear optimization boundary. Not all optimizations are worth pursuing if they degrade the model's output.

5. Speculative decoding is the frontier. With hardware constraints accepted and quality floors established, speculative decoding emerged as the most promising path forward. It addresses the AllReduce bottleneck not by making communication faster, but by reducing the number of communication steps needed per accepted token.

Conclusion

The journey from "If allreduce is so slow it seems like it would make sense to run two inferences at the same time" to "AllReduce accounts for 51.5% of decode time" was not a straight line. It involved multiple hypothesis reversals, a comprehensive profiling campaign, and the gradual acceptance of hard physical constraints. But the methodology that produced this discovery—measure first, optimize second—is a model for performance engineering in any domain.

The final pivot to speculative decoding represents not defeat, but a strategic redirection. The team recognized that the PCIe AllReduce bottleneck was a fundamental constraint of their hardware topology, and that the path to higher throughput lay not in fighting physics but in changing the computational pattern. By generating multiple candidate tokens with a small draft model and verifying them in parallel with the large model, speculative decoding reduces the number of AllReduce operations per accepted token—addressing the bottleneck at the architectural level rather than the kernel level.

In the end, the optimization odyssey of Kimi-K2.5 INT4 on 8x Blackwell GPUs is a testament to the value of disciplined measurement, the humility of accepting wrong hypotheses, and the creativity required to work around hard physical constraints. The GPUs are still not achieving their theoretical peak performance—but the team now knows exactly why, and they have a clear path forward.