The Persistent Optimizer: Iterating Toward Blackwell Performance with Expert Parallelism

Introduction

In the high-stakes world of large language model inference optimization, success rarely comes from a single breakthrough. More often, it emerges from a systematic process of hypothesis, experiment, failure, diagnosis, and iteration. Message [msg 5231] captures this iterative spirit perfectly: a single bash command that launches an SGLang server with Expert Parallelism (EP) using the flashinfer A2A backend, adjusting the memory fraction parameter to fix a crash from the previous attempt. On the surface, it is a mundane command-line invocation. Beneath the surface, it represents the culmination of a rigorous debugging session spanning multiple dead ends, careful diagnosis, and a calculated bet on a fundamentally different communication strategy.

The Message

Let us examine the message in full:

[assistant] [bash] ssh root@10.1.230.174 'nohup /root/ml-env/bin/python3 -m sglang.launch_server \
  --model /shared/kimi-k2.5-int4 \
  --port 30000 \
  --tp 8 \
  --trust-remote-code \
  --host 0.0.0.0 \
  --cuda-graph-max-bs 128 \
  --mem-fraction-static 0.92 \
  --disable-custom-all-reduce \
  --moe-a2a-backend flashinfer \
  --moe-runner-backend flashinfer_cutlass \
  > /data/eagle3/synth_100k/logs/ep_flashinfer_a2a_v2.log 2>&1 &'
echo "EP with flashinfer A2A launched (mem-fraction 0.92)"
EP with flashinfer A2A launched (mem-fraction 0.92)

The command launches the Kimi-K2.5 model (in INT4 quantization) across 8 GPUs using tensor parallelism (--tp 8), with Expert Parallelism enabled via the flashinfer all-to-all (A2A) backend. The critical parameter change from the previous attempt is --mem-fraction-static 0.92, raised from the auto-detected value of approximately 0.78. This single change is the assistant's hypothesis for fixing the out-of-memory (OOM) crash that killed the first EP attempt.

The Reasoning and Motivation

To understand why this message was written, we must trace the reasoning chain that led to it. The assistant had been engaged in a multi-session effort to optimize speculative decoding throughput for the Kimi-K2.5 model on a system with 8×RTX PRO 6000 Blackwell GPUs connected via PCIe (not NVLink). The core bottleneck was the verify pass in EAGLE-3 speculative decoding, which required 122 NCCL allreduce operations per forward pass—each taking approximately 200 microseconds, totaling roughly 24.4 milliseconds of communication overhead per verify step.

The assistant had systematically tested and eliminated several drop-in replacements for NCCL allreduce:

  1. FlashInfer allreduce fusion — Failed because its JIT compiler does not support SM120 (Blackwell architecture).
  2. Custom allreduce kernel — Achieved only 38 tok/s, more than 2× slower than NCCL, due to massive PCIe bus contention from the all-to-all communication pattern.
  3. NCCL Tree algorithm — Incompatible with CUDA graphs, which are essential for reducing kernel launch overhead.
  4. Torch symmetric memory — Failed because SM120 is not in PyTorch's architecture lookup table.
  5. MSCCL++ — Not installed, and installing it would require significant effort. These failures led the assistant to a crucial insight: the problem was not the allreduce algorithm itself (NCCL Ring is well-optimized for PCIe), but rather the per-call latency overhead multiplied by 122 separate allreduce operations. This realization motivated a shift from "make each allreduce faster" to "reduce the number of allreduces." Expert Parallelism (EP) offered a fundamentally different approach: instead of using allreduce to synchronize expert computations across all GPUs, EP uses all-to-all communication to route tokens to the specific GPU hosting the required expert. This replaces 61 MoE allreduces with a smaller number of all-to-all operations.

How Decisions Were Made

The decision process visible in this message is a textbook example of diagnostic iteration. The first EP attempt ([msg 5224]) used the same flags but without an explicit --mem-fraction-static, relying on SGLang's auto-detection. The server crashed during CUDA graph capture with a clear error message:

RuntimeError: Not enough memory. Please try to increase --mem-fraction-static. 
Current value: self.server_args.mem_fraction_static=0.778734296875

The assistant then diagnosed the root cause ([msg 5230]) by comparing memory usage between EP and non-EP configurations. With EP, each GPU had only 16.3 GB available after loading weights, compared to 21.7 GB without EP. The auto-detected mem_fraction_static=0.78 reserved 20.7 GB for static allocations (weights, KV cache), but only 16.3 GB was actually available—creating a negative remainder that triggered the OOM.

The decision to raise --mem-fraction-static to 0.92 was not arbitrary. It was a calculated response to the specific failure mode: EP's weight distribution uses more memory per GPU (likely because each GPU must hold additional expert parameters or dispatch buffers), so the fraction of GPU memory reserved for static allocations needed to increase. The assistant chose 0.92 as a substantial increase from 0.78, deliberately allocating more memory to the static pool at the expense of dynamic memory for request processing.

Assumptions Made

Several assumptions underpin this message. First, the assistant assumes that raising mem_fraction_static to 0.92 will resolve the OOM without causing other problems. This is a reasonable inference from the error message, but it carries risk: if the true memory requirement exceeds even 92%, the server will still crash. Conversely, if 92% is too high, the server might run but with insufficient dynamic memory for request processing, leading to poor throughput or request failures.

Second, the assistant assumes that Expert Parallelism with the flashinfer A2A backend will actually improve throughput once it runs without crashing. This is an unproven hypothesis at this point. The flashinfer A2A backend might have its own performance characteristics on Blackwell GPUs—it could be slower than NCCL allreduce for this specific model and hardware configuration. The assistant is investing significant time (each server launch takes 10+ minutes to load weights and capture CUDA graphs) to test this hypothesis.

Third, the assistant assumes that the combination of EP with the flashinfer_cutlass runner backend is compatible with the INT4-quantized Kimi-K2.5 model. The previous EP attempt showed that SGLang automatically set SGLANG_MOE_NVFP4_DISPATCH=True for the flashinfer A2A backend, suggesting some level of compatibility, but the actual runtime behavior remains untested.

Mistakes and Incorrect Assumptions

The most significant potential mistake in this message is the assumption that memory pressure is the only obstacle to EP working. The first EP attempt crashed with an OOM error, but the logs also showed logging errors and potential assertion failures that were obscured by the OOM. It is possible that even with sufficient memory, the flashinfer A2A backend has deeper compatibility issues with Blackwell GPUs (SM120) or with the INT4-quantized model format. The chunk summary for this segment notes that the flashinfer A2A backend "hit an assertion error and OOM, making it non-functional"—suggesting that the OOM was not the only problem.

Additionally, the assistant may be overestimating the benefit of EP for this specific hardware configuration. EP's all-to-all communication pattern can be more efficient than allreduce for MoE layers, but on PCIe-connected GPUs (without NVLink), all-to-all creates its own contention patterns. The custom allreduce kernel's catastrophic performance (38 tok/s) was attributed to PCIe bus contention from all-to-all communication—the very pattern that EP relies on. If the PCIe topology creates bottlenecks for all-to-all communication, EP could perform worse than the NCCL allreduce baseline, regardless of memory configuration.

Input Knowledge Required

To fully understand this message, one needs knowledge of several domains. The reader must understand the SGLang inference engine's architecture, including its server launch arguments and the distinction between tensor parallelism (--tp) and expert parallelism (--moe-a2a-backend). One must understand the flashinfer library's role as a CUDA kernel provider for attention and MoE operations, and the significance of the flashinfer_cutlass runner backend. Knowledge of GPU memory management in LLM inference is essential: the mem_fraction_static parameter controls what fraction of GPU memory is reserved for model weights and KV cache, with the remainder used for dynamic request processing. Finally, one must understand the Blackwell (SM120) architecture context—that many existing CUDA libraries lack SM120 support, forcing reliance on NCCL as the fallback communication backend.

Output Knowledge Created

This message creates several forms of knowledge. Most immediately, it produces a log file (ep_flashinfer_a2a_v2.log) that will reveal whether the higher memory fraction resolves the OOM and whether EP actually runs on this hardware. The subsequent benchmark results (which would appear in later messages) will either validate or refute the hypothesis that EP can improve throughput over NCCL allreduce. More broadly, this message contributes to the growing knowledge base about what works and what doesn't for Blackwell GPUs on PCIe—a niche but increasingly important configuration as Blackwell-based systems become more common.

The Thinking Process

The thinking process visible in the surrounding messages reveals a methodical, almost scientific approach. The assistant maintains a todo list tracking each optimization approach and its status (completed/dead end/in progress). When an experiment fails, the assistant diagnoses the failure by examining logs, comparing memory usage statistics, and formulating a specific fix. The progression from "try drop-in allreduce replacements" to "change the communication pattern entirely" represents a genuine shift in strategy driven by empirical evidence.

The assistant's reasoning in [msg 5230] is particularly revealing: "With EP: 16.3 GB available after weights (vs 21.7 GB without EP). The EP weight layout uses more memory per GPU — probably because with EP, each GPU needs to hold more expert parameters or additional dispatch buffers. The auto-detected mem_fraction_static=0.78 leaves 94 * (1-0.78) = 20.7 GB reserved, but only 16.3 GB is available → negative rest_memory." This shows the assistant performing a concrete quantitative analysis to understand the failure, rather than guessing blindly.

Conclusion

Message [msg 5231] is a small but significant step in a larger optimization journey. It represents the moment when the assistant, having exhausted simpler approaches, commits to testing a fundamentally different communication strategy—Expert Parallelism—and makes a targeted parameter adjustment based on careful diagnosis of the previous failure. Whether this particular attempt succeeds or fails, the iterative process of hypothesis, experiment, diagnosis, and refinement is the engine driving progress. In the world of ML inference optimization, where hardware and software ecosystems are in constant flux, this systematic approach is often the difference between stagnation and breakthrough.