The Optimization That Wasn't: Diagnosing Fused MoE and Allreduce Fusion on Blackwell GPUs

In the relentless pursuit of higher inference throughput, every millisecond counts. When deploying large language models across multiple GPUs, the communication overhead from all-reduce operations and the computational cost of Mixture-of-Experts (MoE) kernel launches become prime targets for optimization. In message [msg 6448] of a long-running SGLang deployment session, the assistant confronts the sobering reality that not all optimizations deliver on their promise — and that the path to better performance often requires deeper investigation than simply toggling flags.

The Scene: Two Flags, One Hypothesis

The message opens with the assistant examining the aftermath of deploying two optimization flags to the Qwen3.5-122B-A10B BF16 model running on four NVIDIA RTX PRO 6000 Blackwell GPUs. The flags — --enable-fused-moe-sum-all-reduce and --enable-flashinfer-allreduce-fusion — were carefully researched in the preceding messages ([msg 6438] and [msg 6440]). The fused MoE flag promised to merge the expert output summation into the Triton MoE kernel, reducing kernel launch overhead. The flashinfer allreduce fusion flag aimed to fuse the all-reduce communication with the subsequent residual addition and RMSNorm operations, cutting synchronization points. Both seemed like straightforward, low-risk wins.

The assistant had good reason to be optimistic. The Qwen3.5-122B model uses 256 experts with num_experts_per_tok=8, well above the topk > 2 threshold that activates the fused MoE optimization. The flashinfer allreduce fusion was confirmed compatible with the Triton attention backend already in use. The stage was set for a measurable throughput improvement.

Three Observations, One Reality Check

As the server comes up with the new flags, the assistant reads the logs and spots three notable changes:

1. max_running_requests=26 — dropped from 48! This is likely because --enable-flashinfer-allreduce-fusion consumes extra GPU memory for the fusion workspace, reducing available KV cache. But wait — the speculative decoding auto-set was 48 before. Let me check if the fusion flag changed the auto-calculation.

>

2. Missing MoE kernel configs — "Performance might be sub-optimal!" — The Triton MoE kernels don't have tuned configs for this GPU. This is a major optimization opportunity — running the MoE kernel autotuning benchmark could significantly improve throughput.

>

3. disable_piecewise_cuda_graph=True is being auto-set by the flashinfer allreduce fusion.

The first observation is immediately concerning. The maximum number of concurrent requests the server can handle dropped from 48 to 26 — a 46% reduction. The assistant's initial hypothesis points to the flashinfer allreduce fusion consuming extra GPU memory for its fusion workspace, which would reduce the memory available for KV cache and thus lower the request capacity. The "But wait" is a moment of active reconsideration: the assistant realizes the auto-calculation logic itself might have changed, not just the available memory. This is a sharp piece of diagnostic thinking — distinguishing between a resource constraint and a configuration change.

The second observation is a tantalizing lead. The Triton MoE kernels are reporting untuned configurations for this GPU architecture (SM120 Blackwell). The "Performance might be sub-optimal!" warning suggests that running the MoE kernel autotuning benchmark could yield significant gains — potentially much larger than the two flags just deployed.

The third observation reveals an unintended side effect: the flashinfer allreduce fusion flag automatically disables piecewise CUDA graph capture. Piecewise CUDA graphs allow the server to incrementally capture and reuse portions of the computation graph, reducing launch overhead for variable-length sequences. Disabling this could partially offset any gains from the fusion.

The Benchmark: Silence from the Optimizations

Rather than speculating further, the assistant makes a pragmatic decision: benchmark first, investigate later.

Let me first benchmark this config to see if the fused MoE + allreduce fusion help despite the lower max_running_requests, then tackle the MoE kernel tuning.

The benchmark uses a fixed 1000-input-token, 1000-output-token workload across concurrency levels 1, 4, 16, 32, and 64. The results are unambiguous:

   C |  Agg tok/s |  Per-req tok/s |   Tokens |  Wall(s) |   Ok
-----------------------------------------------------------------
   1 |      121.9 |          123.3 |     4000 |     32.8 |    4
   4 |      410.8 |          118.4 |     7209 |     17.5 |    8
  16 |     1212.7 |           84.9 |    29423 |     24.3 |   32
  32 |     1509.2 |           62.5 |    51805 |     34.3 |   64
  64 |     1572.4 |           65.6 |    97000 |     61.7 |  128

Comparing with the baseline from [msg 6436]:

| Concurrency | Baseline (tok/s) | Optimized (tok/s) | Change | |-------------|------------------|-------------------|--------| | 1 | 121.6 | 121.9 | +0.2% | | 4 | 399.1 | 410.8 | +2.9% | | 16 | 1213.5 | 1212.7 | -0.1% | | 32 | 1515.1 | 1509.2 | -0.4% | | 64 | 1582.5 | 1572.4 | -0.6% |

The differences are well within measurement noise. The optimizations produced no measurable improvement — and at high concurrency, there's a whisper of regression. The fused MoE sum all-reduce and flashinfer allreduce fusion, despite their theoretical promise, delivered nothing.

Why Did the Optimizations Fail?

The benchmark results demand explanation. Several factors likely contributed:

The bottleneck wasn't where the optimizations targeted. The fused MoE sum all-reduce optimizes the expert output summation step within the MoE layer, while the flashinfer allreduce fusion optimizes the all-reduce + residual + normalization pipeline. If the dominant cost is elsewhere — for instance, the attention mechanism, the feed-forward network computations, or the CPU-side scheduling overhead — these optimizations would have little impact.

The lower max_running_requests may have capped throughput at high concurrency. At concurrency 64, the server can only handle 26 requests simultaneously. The remaining 38 requests must queue, introducing idle time that offsets any per-request speedup. The near-identical results at concurrency 1 (where queuing doesn't apply) suggest the per-request latency didn't improve either.

The piecewise CUDA graph disable may have offset gains. If the flashinfer allreduce fusion forced disable_piecewise_cuda_graph=True, the server lost the ability to incrementally capture CUDA graphs for variable-length sequences. This could increase kernel launch overhead, particularly for the diverse sequence lengths in a real workload.

The Blackwell GPU architecture (SM120) may not benefit from these specific fusions. The fused MoE kernel and flashinfer allreduce fusion were likely designed and tuned for earlier architectures (SM80/SM90). The Blackwell memory subsystem and compute capabilities might render the fusion unnecessary or even counterproductive.

The Thinking Process: A Model of Diagnostic Discipline

What makes this message particularly instructive is the assistant's disciplined approach to diagnosis. Rather than jumping to conclusions or immediately reverting the flags, the assistant:

  1. Reads the logs carefully, extracting three concrete observations from the startup output.
  2. Formulates hypotheses for each observation, including the "But wait" self-correction about the max_running_requests calculation.
  3. Prioritizes measurement over speculation, running the benchmark before deciding on next steps.
  4. Interprets results in context, recognizing that the near-identical numbers mean the optimizations didn't help — and that the MoE kernel tuning path is now the more promising lead. This is textbook systems optimization methodology: measure, hypothesize, change, measure again. The assistant resists the temptation to declare victory or defeat based on log messages alone.

Assumptions and Their Limits

The message reveals several implicit assumptions worth examining:

That the flags would compose cleanly. The assistant assumed both flags could be enabled simultaneously without negative interactions. In reality, the flashinfer allreduce fusion forced disable_piecewise_cuda_graph=True, an interaction that wasn't obvious from the individual flag documentation.

That the Blackwell GPU would benefit from the same optimizations as earlier architectures. The fused MoE and allreduce fusion kernels were developed and tested on Hopper (SM90) and Ada Lovelace (SM89) GPUs. The Blackwell SM120 architecture may have different performance characteristics that render these fusions unnecessary or even detrimental.

That the benchmark would reveal the effect. The fixed 1000-in/1000-out synthetic workload may not reflect the actual production traffic patterns. A workload with shorter sequences or different expert routing patterns might show different results.

That max_running_requests=26 would be a bottleneck. The assistant implicitly assumed that lower concurrency capacity would hurt throughput, but the benchmark showed no degradation at concurrency 64. This suggests either that the system was already saturated before reaching 26 concurrent requests, or that the queuing overhead was minimal.

Knowledge Created

This message creates several valuable pieces of knowledge for the deployment:

  1. The two optimization flags are ineffective for this model/GPU combination. Future optimization attempts should focus elsewhere.
  2. The MoE kernel autotuning is the next logical target. The "Performance might be sub-optimal!" warning points to a concrete, actionable optimization path.
  3. The flashinfer allreduce fusion has a hidden cost — it reduces max_running_requests and disables piecewise CUDA graphs. These side effects must be weighed against any benefits.
  4. The benchmark methodology is sound and reproducible. The assistant has a reliable way to measure the impact of future changes.

The Road Ahead

The message ends with the assistant preparing to tackle MoE kernel tuning — a potentially much larger optimization. The Triton MoE autotuning benchmark can search thousands of kernel configurations to find the optimal tile sizes, warp schedules, and pipeline depths for the Blackwell GPU. This is the kind of architecture-specific tuning that can unlock 20-50% throughput improvements on MoE-heavy models like Qwen3.5-122B.

The failed optimization flags are not a setback; they are a redirection. By ruling out two promising candidates, the assistant has narrowed the search space and identified the next most promising target. In the high-stakes world of LLM inference optimization, knowing what doesn't work is just as valuable as knowing what does.