The EP Gamble: A Pivot to Expert Parallelism on PCIe-Bound Blackwell GPUs

In the high-stakes world of large language model inference optimization, the difference between a breakthrough and a dead end can be measured in milliseconds. Message [msg 5227] captures a pivotal moment in a long-running optimization campaign for the Kimi-K2.5 model running on an 8× RTX PRO 6000 Blackwell GPU system connected via PCIe. The message is deceptively brief — a confirmation that Expert Parallelism (EP) has been enabled, followed by a bash command to wait ten minutes and check the logs. But this short message sits at the intersection of a systematic debugging effort, a series of failed optimizations, and a critical architectural pivot that ultimately ran into an unexpected memory constraint.

Context: The Road of Dead Ends

To understand why message [msg 5227] was written, one must first understand the optimization landscape that preceded it. The assistant had been engaged in a multi-day effort to improve speculative decoding throughput for the Kimi-K2.5 model, a Mixture-of-Experts (MoE) architecture with 61 attention layers and 61 MoE layers. The core bottleneck was the "verify step" — the process by which the target model validates tokens proposed by a draft model — which required 122 NCCL allreduce operations per forward pass, each taking approximately 200 microseconds, for a total of roughly 24.4 milliseconds of pure communication overhead per verify step.

The assistant had systematically tested and eliminated one optimization approach after another. FlashInfer allreduce fusion failed because its JIT compiler does not support the SM120 (Blackwell) architecture. A custom allreduce kernel, when forced to work over PCIe, produced only 38 tok/s — more than 2× slower than NCCL — because the all-to-all communication pattern created massive PCIe bus contention across 8 GPUs. Torch symmetric memory failed because SM120 is not in its architecture lookup table, producing a KeyError: 12. NCCL Tree algorithm was incompatible with CUDA graphs. Each approach was methodically tested, benchmarked, and marked as a dead end in the assistant's todo list.

Amidst these failures, one genuine improvement emerged: reducing --cuda-graph-max-bs from 512 to 128 improved the baseline throughput from 82 tok/s to 89.5 tok/s — a 9% gain achieved by freeing GPU memory for KV cache. This was documented in [msg 5222], where the assistant noted the improvement and prepared to pivot to the next approach.

The EP Hypothesis

Expert Parallelism represented a fundamentally different strategy from the "make each allreduce faster" approaches that had all failed. Instead of optimizing the communication primitive, EP changes the communication pattern itself. In a standard Tensor Parallelism (TP) configuration, every GPU holds a copy of all expert weights, and MoE layers require allreduce operations to aggregate results across GPUs. With EP, each GPU handles a subset of experts, and tokens are routed to the appropriate GPU via all-to-all communication. This replaces 61 MoE allreduces with a smaller number of all-to-all operations — a pattern that could potentially be more efficient, especially when using specialized backends like flashinfer's A2A (all-to-all) implementation.

The assistant had researched EP configuration in [msg 5215], discovering that SGLang automatically sets ep_size = tp_size when any A2A backend is enabled. Since DeepEP was not installed, the assistant opted for the flashinfer A2A backend, launching the server in [msg 5224] with the flags --moe-a2a-backend flashinfer and --moe-runner-backend flashinfer_cutlass, while retaining the proven optimizations from the baseline: --cuda-graph-max-bs 128 and --disable-custom-all-reduce.

What Message 5227 Reveals

The message itself consists of two parts. First, the assistant reads the server log and confirms that EP is functioning:

Good — EP is enabled (TP2 EP2, TP4 EP4, etc. in the logs). ep_size=8 auto-detected. Weights are loaded.

The TP2 EP2 and TP4 EP4 patterns in the log output confirm that SGLang has correctly configured the combined TP+EP topology, where each GPU is assigned both a tensor parallel rank and an expert parallel rank. The auto-detection of ep_size=8 confirms that the flashinfer A2A backend has correctly set the expert parallel size equal to the tensor parallel size, as the code in server_args.py dictates.

The second part of the message is a bash command that sleeps for 600 seconds (10 minutes) before tailing the log file:

Let me wait for CUDA graph capture and server ready: [bash] sleep 600 && ssh root@10.1.230.174 'tail -20 /data/eagle3/synth_100k/logs/ep_flashinfer_a2a.log'

This 600-second sleep reflects the assistant's understanding of the server initialization timeline. For a model of this size (Kimi-K2.5 with 8 GPUs), weight loading alone can take several minutes, and CUDA graph capture — where SGLang pre-compiles execution graphs for common token sequence lengths — can add additional minutes. The assistant is being patient, allowing the server to fully initialize before checking the result.

The Hidden Failure

The log output that eventually returned showed only logging infrastructure noise — handleError, launch_phase_sigquit_handler, and similar Python logging internals. This was the first sign of trouble, though the assistant didn't yet know it. In the very next message ([msg 5228]), the assistant would grep for the actual error and discover:

RuntimeError: Not enough memory. Please try to increase --mem-fraction-static. 
Current value: self.server_args.mem_fraction_static=0.778734296875

The EP configuration had crashed with an out-of-memory error. Further investigation in [msg 5229] and [msg 5230] revealed that EP used significantly more memory per GPU than TP alone: only 16.3 GB available after weight loading versus 21.7 GB without EP. The auto-detected mem_fraction_static of 0.78 left 20.7 GB reserved, but only 16.3 GB was actually available, creating a negative rest_memory condition.

Assumptions and Their Consequences

Message [msg 5227] reveals several implicit assumptions that the assistant was operating under:

Assumption 1: EP would be memory-neutral. The assistant assumed that switching from TP-only to TP+EP would not significantly change the per-GPU memory footprint. In reality, EP requires additional dispatch buffers and a different weight layout that increases memory consumption. This assumption was reasonable — the assistant had no prior data on EP memory usage for this specific model — but it proved incorrect.

Assumption 2: The flashinfer A2A backend would work with the INT4-quantized model. The --moe-runner-backend flashinfer_cutlass flag was chosen because the research in [msg 5215] indicated it was required for flashinfer A2A. However, the compatibility of this runner backend with NVFP4 (NVIDIA FP4) quantization was uncertain. The log did show SGLANG_MOE_NVFP4_DISPATCH is set to True for Flashinfer MoE A2A, suggesting some level of integration, but the OOM crash precluded any actual benchmarking.

Assumption 3: CUDA graph capture would complete within 600 seconds. The assistant allocated a generous 10-minute window for initialization. This was reasonable based on previous experience — the baseline server typically initialized in 5-7 minutes — but the EP server had actually crashed much earlier during weight loading or initial graph capture.

Input Knowledge Required

To fully understand this message, one needs knowledge of several domains. First, an understanding of distributed inference strategies for MoE models — specifically the difference between Tensor Parallelism (where each GPU holds all experts and communicates via allreduce) and Expert Parallelism (where each GPU holds a subset of experts and communicates via all-to-all). Second, familiarity with SGLang's server architecture, including its CUDA graph capture mechanism and the relationship between tp_size, ep_size, and the various A2A backends. Third, knowledge of the hardware constraints — 8× RTX PRO 6000 Blackwell GPUs connected via PCIe rather than NVLink, which makes allreduce optimization particularly challenging. Fourth, awareness of the preceding optimization attempts and their failures, which provides the context for why EP was the next logical step.

Output Knowledge Created

This message produced several pieces of knowledge, even though the EP experiment ultimately failed. It confirmed that SGLang's flashinfer A2A backend correctly initializes and auto-detects the EP configuration for the Kimi-K2.5 model. It revealed that EP with ep_size=8 increases per-GPU memory consumption by approximately 5.4 GB compared to TP-only (from 21.7 GB available to 16.3 GB available after weight loading). This is a significant finding — it means that for memory-constrained configurations, EP may not be viable without reducing other memory consumers like KV cache or batch size. The message also established that the --moe-runner-backend flashinfer_cutlass flag is compatible with NVFP4 quantization at the initialization level, even if the OOM prevented actual inference.

The Thinking Process

The message reveals a methodical, step-by-step approach to debugging. The assistant does not simply launch the server and hope for the best. It checks the logs incrementally — first confirming EP is enabled and weights are loaded, then waiting for CUDA graph capture. This staged verification is characteristic of the assistant's debugging style throughout the conversation: confirm each milestone before proceeding to the next.

The decision to use a 600-second sleep rather than a polling loop is pragmatic. The assistant could have written a script that periodically checks for the server ready signal, but a simple sleep-and-check is sufficient for a one-off experiment. The trade-off is that if the server crashes early, the assistant wastes 10 minutes waiting — which is exactly what happened here.

The parenthetical note about TP2 EP2, TP4 EP4 patterns is telling. The assistant is reading the raw log output and interpreting the distributed topology from the rank identifiers. This shows a deep familiarity with SGLang's internal logging conventions and an ability to diagnose configuration correctness from log patterns alone.

A Pivot That Changed Direction

Message [msg 5227] represents the last attempt at a "drop-in" optimization for the verify step before the assistant would pivot to a fundamentally different strategy. After the EP OOM failure, the assistant would go on to try increasing mem_fraction_static to 0.92 ([msg 5231]), which also failed. Eventually, the user would propose upgrading CUDA to version 13 — a system-level change that could unblock the SM120-native optimizations that had been consistently failing. This upgrade path, explored in subsequent messages, would offer a way to enable flashinfer fusion, torch symmetric memory, and other Blackwell-native optimizations that were previously unavailable.

In retrospect, message [msg 5227] marks the end of one optimization chapter and the beginning of another. The EP experiment failed, but it provided crucial data about memory constraints and confirmed that the flashinfer A2A infrastructure was functional. The path forward would require more drastic measures — a CUDA toolkit upgrade — rather than incremental configuration changes. The message stands as a testament to the systematic, hypothesis-driven approach that characterizes effective performance optimization: test each hypothesis, document the results, and pivot when the evidence points in a new direction.