Pivoting Strategy: Researching Expert Parallelism After Allreduce Dead Ends on Blackwell PCIe

Introduction

In the long and winding journey to optimize speculative decoding throughput on an 8×RTX PRO 6000 Blackwell GPU system connected via PCIe, message [msg 5215] marks a critical inflection point. After systematically testing and eliminating four different allreduce optimization approaches—FlashInfer fusion, custom allreduce, NCCL Tree, and Torch symmetric memory—the assistant finds itself at a strategic crossroads. The baseline inference server is reloading (a 10-minute wait), and rather than idle, the assistant deploys a subagent task to research Expert Parallelism (EP) as the next candidate approach. This message is a study in strategic time management, systematic problem-solving, and the intellectual discipline of maintaining multiple parallel hypotheses in the face of repeated dead ends.

The Message in Full

The assistant writes:

While the baseline is loading (10 min), let me research EP configuration for Kimi-K2.5 and whether it can coexist with TP: [task] {"description":"Research EP configuration for SGLang","prompt":"Research how Expert Parallelism (EP) is configured in SGLang, specifically for DeepSeek-V2-style MoE models like Kimi-K2.5. I need to know:\n\n1. Can EP be used together with TP? (e.g., --tp 8 --ep-size 8)\n2. What --moe-a2a-backend opt...

The message then spawns a subagent via the task tool, which runs to completion and returns a detailed analysis of EP configuration in SGLang. The task result (partially visible in the message) confirms that EP requires TP and automatically sets ep_size = tp_size.

The Strategic Context: A Trail of Dead Ends

To understand why this message matters, one must appreciate the gauntlet of failures that preceded it. The assistant had been chasing a single goal: making EAGLE-3 speculative decoding profitable on this hardware. The verify step—where the draft model's predictions are checked against the target model—required 122 NCCL allreduce operations per forward pass, each taking roughly 200 microseconds, totaling about 24 milliseconds of pure communication overhead. If this verify cost could be reduced, speculative decoding might finally beat the baseline of ~89 tok/s.

The assistant had pursued four approaches, each eliminated in turn:

1. FlashInfer allreduce fusion (msg 5197 area): Dead because the FlashInfer JIT compiler does not support SM120 (Blackwell architecture). The kernel simply refused to compile.
2. Custom allreduce on PCIe (msg 5201–5202): The assistant forced SGLang's custom allreduce kernel to operate on PCIe topology, but the result was catastrophic: 38 tok/s, more than 2× slower than NCCL. The all-to-all communication pattern required every GPU to read from all 7 others simultaneously, saturating the PCIe switch with 56 simultaneous cross-GPU reads. NCCL Ring, by contrast, pipelines traffic through sequential neighbor-to-neighbor transfers, avoiding bus contention entirely.
3. NCCL Tree algorithm (earlier in segment): Incompatible with CUDA graphs, which SGLang uses for its optimized inference path.
4. Torch symmetric memory (msg 5210–5212): Failed with a KeyError: 12 because the PyTorch symmetric memory communicator does not recognize SM 12.0 (Blackwell's compute capability). The architecture lookup table simply had no entry for it. Each failure was cleanly documented, the server killed, and the approach marked as a dead end in the todo list. The assistant's systematic methodology—test, measure, document, move on—is a model of disciplined engineering research.

Why This Message Was Written

The immediate trigger was the 10-minute wait for the baseline server to load. The assistant had just killed the failed Torch symmetric memory experiment and relaunched a clean baseline with --disable-custom-all-reduce and --cuda-graph-max-bs 128. Rather than waste the loading time, the assistant proactively researched the next candidate.

But the deeper motivation was strategic. The assistant had identified two remaining paths forward:

• Path A: Upgrade CUDA to version 13. The user had proposed this in the previous segment, and initial research showed that CUDA 13 has native SM120 support, which could unblock FlashInfer fusion, Torch symmetric memory, and other Blackwell-native optimizations. This was a high-effort, high-upside path requiring a full toolkit upgrade.
• Path B: Expert Parallelism. This approach fundamentally changes the communication pattern for MoE layers from allreduce to all-to-all (A2A). Instead of every GPU holding all experts and reducing gradients across all GPUs, each GPU handles a subset of experts, and tokens are routed to the appropriate GPU. This replaces 61 MoE allreduces with a smaller number of all-to-all communications, potentially bypassing the allreduce bottleneck entirely. The assistant chose to research Path B first because it was faster to test—no installation required, just a command-line flag change. The task was designed to answer two critical questions: (1) Can EP coexist with TP (tensor parallelism)? and (2) Which A2A backend is appropriate for this hardware?

Assumptions and Knowledge

The assistant operated under several assumptions:

• That EP could be a drop-in optimization. The task prompt asks whether EP can be used "together with TP," implying the assistant hoped EP would layer on top of the existing TP=8 configuration without requiring architectural changes.
• That the flashinfer A2A backend would work on SM120. Given that FlashInfer's allreduce fusion had already failed on SM120, there was a risk that the A2A backend would face the same issue. The assistant's research prompt specifically asks about the --moe-a2a-backend options, suggesting awareness of this risk.
• That the 10-minute baseline load would complete without issues. The assistant was assuming the baseline would restore cleanly, providing a reference point against which to measure EP performance.
• That NCCL Ring was indeed optimal for PCIe. The evidence strongly supported this—the custom allreduce at 38 tok/s was definitive proof—but the assistant was implicitly assuming no further NCCL tuning could improve the baseline. The input knowledge required to understand this message includes: the architecture of DeepSeek-V2-style MoE models (which Kimi-K2.5 inherits), the concept of tensor parallelism vs. expert parallelism, the role of allreduce in distributed inference, and the SGLang server's command-line interface. The assistant draws on all of this to formulate a precise research question.

The Thinking Process Visible in the Message

The reasoning is compact but revealing. The assistant opens with "While the baseline is loading (10 min)"—a clear statement of opportunistic time management. The phrase "let me research EP configuration for Kimi-K2.5 and whether it can coexist with TP" reveals the core uncertainty: the assistant doesn't know if EP and TP are compatible, and this is the first question the subagent must answer.

The task prompt is structured as three numbered questions, reflecting methodical thinking:

1. Compatibility: Can EP and TP be combined? This is the gating question—if they can't, the approach is dead immediately.
2. Backend options: What A2A backends exist? This shows the assistant is thinking about implementation details, not just theory.
3. Hardware fit: Which backend works for this specific GPU and topology? This reflects the lesson learned from previous failures—FlashInfer didn't work on SM120, so the assistant is proactively checking compatibility. The task tool itself is a meta-cognitive choice: rather than running bash commands to grep through SGLang source code (which would be faster but require the assistant's own attention), the assistant delegates the research to a subagent. This frees the assistant to monitor the baseline server's startup and be ready to act immediately when it's ready. It's a deliberate division of cognitive labor.

Output Knowledge Created

The task result (partially visible in the message) produces several concrete findings:

• EP requires TP. When any A2A backend is enabled, SGLang automatically sets ep_size = tp_size. This means --tp 8 --ep-size 8 is redundant—EP always matches TP size.
• The flashinfer A2A backend exists as one of the options, alongside deepep and native-torch.
• The configuration is purely command-line driven, requiring no code changes—just adding --moe-a2a-backend flashinfer (or another backend) to the launch command. This knowledge directly shapes the next actions. The assistant will go on to test the flashinfer A2A backend (which will fail with an assertion error and OOM, as noted in the chunk summary), then pivot to the CUDA 13 upgrade path. The research wasn't wasted—it eliminated another approach efficiently.

Significance and Impact

This message represents the assistant's transition from "try drop-in allreduce replacements" to "fundamentally change the communication pattern." The allreduce approaches had all failed because they tried to make the existing 122-allreduce pipeline faster without changing its structure. EP represents a structural change: replace allreduces with all-to-all communication, which has fundamentally different scaling properties.

More broadly, this message exemplifies a key engineering virtue: using waiting time productively. The assistant could have simply waited for the baseline to load, but instead it advanced the investigation by researching the next candidate. In a debugging session spanning hundreds of messages and dozens of experiments, this kind of time optimization compounds significantly.

The message also demonstrates the value of systematic dead-end documentation. Because the assistant had clearly marked each failed approach in its todo list, it could confidently move to the next candidate without second-guessing or revisiting old ground. The research task was the natural next step in a methodical elimination process.

Conclusion

Message [msg 5215] is a quiet but pivotal moment in a complex optimization journey. Sandwiched between the failure of Torch symmetric memory and the eventual pivot to CUDA 13, it represents the assistant's disciplined approach to problem-solving: use every minute productively, research before acting, and systematically eliminate hypotheses until the viable path emerges. The EP research would ultimately lead to another dead end (the flashinfer A2A backend would fail on SM120), but that failure was clean, fast, and informative—exactly because the assistant had done the upfront research to know exactly how to test it. In engineering, knowing what not to pursue is often as valuable as knowing what to pursue, and this message captures that wisdom in action.