The OOM That Killed Expert Parallelism: A Turning Point in the Blackwell Optimization Saga

In the high-stakes world of large language model serving, every millisecond counts. When you're running an 8×RTX PRO 6000 Blackwell system with PCIe interconnects, the path to performance is littered with dead ends. Message [msg 5228] captures one such moment — a crisp, almost laconic discovery that Expert Parallelism (EP), the last promising approach on the table, had crashed with an out-of-memory error. The message itself is brief: "Crashed. Let me find the error," followed by a grep command into a server log. But the weight it carries is enormous. It represents the systematic collapse of an entire optimization strategy and forces a fundamental pivot in the project's direction.

The Context: A Cascade of Dead Ends

To understand why this message matters, we must trace the path that led to it. The session's broader narrative is a desperate hunt for a way to make speculative decoding profitable on a PCIe-connected Blackwell system. The core problem is stark: the EAGLE-3 speculative decoding pipeline requires 122 NCCL allreduce operations per forward pass during the verification step. Each allreduce takes roughly 200 microseconds, totaling about 24.4 milliseconds of pure communication overhead per verify pass. With speculative decoding achieving only 54.1 tok/s against a baseline of 89.5 tok/s, the assistant had been systematically testing every conceivable approach to reduce this overhead.

The graveyard of failed attempts, meticulously documented in the preceding messages, is sobering:

The EP Hypothesis: A Fundamentally Different Approach

After all the "drop-in replacement for NCCL allreduce" approaches failed, the assistant articulated a crucial insight in [msg 5214]:

NCCL Ring allreduce is actually well-optimized for PCIe — the problem isn't the algorithm, it's the per-call latency overhead multiplied by 122 calls.

This reframing of the problem is the key intellectual move. Instead of trying to make each allreduce faster, the assistant proposed changing the communication pattern entirely. Expert Parallelism (EP) replaces the MoE allreduces with all-to-all communication. With EP, each GPU handles a subset of experts, and tokens are routed to the appropriate GPU. This replaces 61 MoE allreduces with a smaller number of all-to-all operations — a fundamentally different communication topology.

The assistant then launched a research subtask ([msg 5215]) to investigate EP configuration in SGLang, discovering that EP requires TP (tensor parallelism) and that the --moe-a2a-backend flashinfer flag automatically sets ep_size = tp_size. This seemed promising: the flashinfer A2A backend could potentially bypass the NCCL bottleneck entirely.

The Launch and the Crash

In [msg 5224], the assistant launched the EP server with the flags --moe-a2a-backend flashinfer --moe-runner-backend flashinfer_cutlass. The server began loading weights. In [msg 5226], the assistant checked the logs and found encouraging signs: "Flashinfer MoE A2A is enabled. The expert parallel size is adjusted to be the same as the tensor parallel size[8]." The weights loaded successfully. The assistant then waited 600 seconds (10 minutes) for CUDA graph capture and server initialization.

Then came message [msg 5227]: the tail of the log showed only logging infrastructure errors — no clear success signal. The assistant needed to dig deeper.

This brings us to the subject message, [msg 5228]. The assistant runs a targeted grep for errors:

[assistant] Crashed. Let me find the error:
[bash] ssh root@10.1.230.174 'grep "Traceback\|Error\|error\|Exception" /data/eagle3/synth_100k/logs/ep_flashinfer_a2a.log | grep -v "Ignore\|Logging\|handleError\|stderr\|launch_phase\|Message\|reentrant\|BufferedWriter" | head -15'

The result is unambiguous:

RuntimeError: Not enough memory. Please try to increase --mem-fraction-static. Current value: self.server_args.mem_fraction_static=0.778734296875

The EP server ran out of memory during scheduler initialization. The error message appears across multiple GPU ranks (TP1 EP1, TP3 EP3, etc.), confirming it's a systematic failure, not an isolated incident.

Why EP Ran Out of Memory

The subsequent messages reveal the root cause. In [msg 5230], the assistant checks the available memory after weight loading and discovers a critical difference:

With EP: 16.3 GB available after weights (vs 21.7 GB without EP). The EP weight layout uses more memory per GPU — probably because with EP, each GPU needs to hold more expert parameters or additional dispatch buffers.

The auto-detected mem_fraction_static=0.78 reserves about 20.7 GB of the 94 GB GPU memory for the KV cache and runtime buffers. But with EP consuming more weight memory, only 16.3 GB remains available — a shortfall of roughly 4.4 GB. The scheduler can't allocate its required buffers and crashes.

This is a subtle but important failure mode. Expert Parallelism doesn't just change the communication pattern; it also changes the weight distribution across GPUs. In standard tensor parallelism, each GPU holds a shard of every expert. In expert parallelism, each GPU holds a complete set of a subset of experts. For a model with 61 MoE layers and 256 experts (like Kimi-K2.5), this means each GPU in an 8-way EP configuration must store 32 complete experts plus their shared expert components — a significantly larger memory footprint per GPU than the TP-only layout.

The Assumptions and Their Failure

This message reveals several assumptions that turned out to be incorrect:

  1. EP would be memory-neutral: The assistant assumed that switching from TP to EP+TP would not significantly change the memory footprint. The research task ([msg 5215]) focused on communication patterns and configuration flags but didn't investigate memory implications. This blind spot led directly to the crash.
  2. Auto-detected mem_fraction would work: The assistant relied on SGLang's auto-detection of mem_fraction_static, which worked perfectly for the TP-only baseline. The assumption that the same auto-detection would work for EP was reasonable but wrong — the EP memory layout changed the available memory calculation.
  3. Flashinfer A2A would be a drop-in replacement: The research indicated that flashinfer A2A was a supported backend, but the memory implications of enabling it weren't clear from the code inspection.

The Thinking Process Visible in the Message

The message itself is terse, but it reveals a methodical debugging process. The assistant doesn't just check "did it crash?" — it runs a carefully filtered grep that excludes known noise sources (logging infrastructure errors, signal handlers, etc.) to find the actual failure. The head -15 limit suggests the assistant expects a manageable number of relevant errors, not a flood.

The structure "Crashed. Let me find the error" is telling. The assistant already knows the server crashed (from the previous message's log tail showing logging errors but no server ready message). The task now is diagnosis, not discovery. This is a mature engineering workflow: detect the failure, then pinpoint the cause.

The Knowledge Created

This message creates several pieces of critical knowledge:

The Broader Significance

In the arc of the session, this message is the moment when the assistant exhausts the "drop-in replacement" and "fundamentally different communication pattern" approaches simultaneously. The EP attempt failed not because of architecture incompatibility (like FlashInfer and torch symmetric memory) but because of memory pressure — a different class of problem that might be solvable with tuning, but one that adds complexity and risk.

The assistant's response in the following messages shows the pivot: first trying a higher mem_fraction_static ([msg 5231]), then discovering that even with memory fixed, the EP server hits assertion errors and crashes in the forward pass ([msg 5232]). The final verdict is that EP is non-functional on this stack.

This cascade of failures — allreduce fusion, custom kernels, symmetric memory, expert parallelism — sets the stage for the next major pivot: upgrading CUDA from version 12.8 to 13.1 to unlock Blackwell-native optimizations that were previously blocked. The OOM in message [msg 5228] is the last nail in the coffin of the "work with what we have" approach, forcing the assistant to consider a more radical system-level change. It's a quiet but pivotal moment in a deeply technical journey — the kind of moment where a single RuntimeError: Not enough memory redirects the entire trajectory of a multi-day optimization effort.