Reading the MoE Routing Code: A Precision Strike in the Optimization Campaign

Message Overview

The subject message (index 1085) is a single bash command executed by the assistant on a remote server:

ssh root@10.1.230.174 'sed -n "780,900p" /root/sglang/python/sglang/srt/layers/moe/topk.py'

The output reveals lines 780–900 of sglang's MoE routing implementation, specifically the fused_topk_deepseek function call and the _biased_grouped_topk_postprocess conditional logic. This seemingly mundane file-read operation is in fact a precision strike within a much larger optimization campaign: the assistant is systematically reverse-engineering sglang's Mixture-of-Experts (MoE) routing code to implement a novel technique called Opportunistic Expert Activation (OEA), which promises to significantly reduce decode latency for large MoE models like GLM-5-NVFP4.

Context: The Optimization Campaign

To understand why this message exists, one must appreciate the broader context. The assistant has been engaged in an intensive, multi-day effort to maximize inference throughput for the GLM-5-NVFP4 model — a 744B-parameter MoE model with 256 experts per layer, 8 active experts per token, and 80 layers — running on 8 NVIDIA RTX PRO 6000 Blackwell (SM120) GPUs. Earlier in the session, the assistant had already achieved remarkable results: throughput improved from ~880 tok/s to ~3,740 tok/s through FlashInfer CUTLASS MoE autotuning and server parameter tuning. But the core bottleneck remained: small per-expert GEMM operations on SM120 GPUs, which have only 100KB of shared memory per SM.

The assistant had commissioned three research agents (see [msg 1066]) to investigate the latest developments in sglang SM120 support, GLM-5 deployment strategies, and SM120 FP4 kernel optimization. One of the most promising leads came from a research agent that analyzed the "Opportunistic Expert Activation" technique (arXiv:2511.02237), which claims 39% reduction in MoE layer decode latency by reducing the number of unique experts loaded across a batch. The key insight of OEA is that during decode, the memory-bandwidth bottleneck comes from loading expert weights; if multiple tokens in a batch select different experts, the system must load many expert weight matrices. OEA modifies the routing to encourage tokens to share experts, reducing the memory footprint.

Why This Specific File Read?

The assistant is now in the implementation phase. Having received the research findings, it needs to modify sglang's MoE routing code to implement OEA. But before making any changes, it must thoroughly understand the existing routing pipeline. This is not a casual code review — it is a surgical exploration of a critical code path.

The file topk.py contains the top-k expert selection logic — the very heart of MoE routing. The assistant has been reading this file incrementally across multiple messages:

• In [msg 1081], it read lines 1–100 (file header, imports, initial setup).
• In [msg 1083], it read lines 520–640, which revealed the group-masked top-k selection using torch.topk.
• In [msg 1084], it read lines 640–780, showing the sigmoid score computation and parameter definitions.
• Now in message 1085, it reads lines 780–900 to see the actual fused_topk_deepseek function call and the post-processing logic. This incremental approach reveals a methodical thinking process. The assistant is building a mental model of the routing code layer by layer, starting from the high-level structure and drilling down into the specific function calls. Each read targets a specific section that fills a gap in understanding.

What the Message Reveals

The output of message 1085 shows two critical pieces of the routing pipeline:

1. The fused_topk_deepseek function call:

fused_topk_deepseek(
    gating_output.to(dtype=torch.float32),
    correction_bias,
    num_expert_group,
    topk_group,
    topk,
    scaling_factor,
    topk_weights,
    topk_ids,
    True,
)

This is the core routing function, specific to DeepSeek-style MoE architectures (which GLM-5 also uses). It takes the gating output (converted to float32), a correction bias, grouping parameters, and outputs the selected expert IDs and weights. The final True parameter likely indicates whether to return sorted results.

2. The conditional post-processing:

if (expert_location_dispatch_info is not None) or (
    num_token_non_padded is not None
):
    topk_ids = _biased_grouped_topk_postprocess(
        topk_ids, expert_location_dispatch_info, num_token_non_padded
    )

This reveals that after the fused top-k selection, there is an optional post-processing step that adjusts the selected expert IDs based on expert location dispatch information or non-padded token counts. This is likely used for expert load balancing or for handling padding tokens in variable-length batches.

Input Knowledge Required

To fully understand this message, one needs:

1. MoE architecture knowledge: Understanding that Mixture-of-Experts layers route each token to a subset of experts (typically via top-k selection), and that the routing is a critical performance bottleneck.
2. DeepSeek MoE routing specifics: Knowledge of the grouped top-k approach used by DeepSeek models, where experts are divided into groups and the top-k selection happens within groups with a correction bias.
3. SGLang codebase familiarity: Understanding that topk.py is the central file for expert routing, and that the fused_topk_deepseek function is a fused CUDA kernel that performs the entire routing operation efficiently.
4. OEA technique understanding: Knowing that OEA modifies the routing to reduce unique expert count, and that implementing it requires intercepting the routing at the right point.
5. GLM-5 model architecture: Knowing that GLM-5 has 256 experts, 8 active per token, 80 layers, and uses DeepSeek-style routing.

Assumptions and Decisions

The assistant makes several implicit assumptions in this message:

Assumption 1: The routing code is the right place to implement OEA. This is a sound assumption — OEA is fundamentally a routing modification, so modifying the top-k selection logic is the natural approach. However, the assistant could have chosen to implement OEA at a higher level (in the model forward pass) or at a lower level (in the expert dispatch logic). The choice to target topk.py shows a preference for modifying the routing itself rather than the surrounding infrastructure.

Assumption 2: The fused_topk_deepseek function handles the core routing. The assistant is verifying this by reading the code. The presence of topk_weights and topk_ids as output parameters confirms that this function is where expert selection happens.

Assumption 3: The post-processing step is relevant for OEA. The _biased_grouped_topk_postprocess function can modify the selected experts, which means OEA could potentially be implemented either before fused_topk_deepseek (by modifying the gating scores) or after (by modifying the selected IDs in the post-processing step).

Assumption 4: The existing routing uses sigmoid scores. This was confirmed in [msg 1084] where the assistant saw scores = gating_output.sigmoid(). This is important because OEA uses sigmoid scores for its expert selection modification.

Output Knowledge Created

This message produces several valuable pieces of knowledge:

1. Complete routing pipeline map: The assistant now knows the full flow: gating output → sigmoid → grouped top-k via fused_topk_deepseek → optional _biased_grouped_topk_postprocess. This is the foundation for implementing OEA.
2. Function interface documentation: The exact parameter list of fused_topk_deepseek is now known, including the correction_bias parameter which could be relevant for OEA modifications.
3. Post-processing conditions: The assistant now knows that post-processing only activates when expert_location_dispatch_info or num_token_non_padded are provided, which helps understand when routing modifications would be applied.
4. Implementation entry points: The assistant now has two clear entry points for OEA: (a) modify the scores before fused_topk_deepseek, or (b) modify the selected IDs after the function call. Each approach has different trade-offs.

The Thinking Process

The assistant's thinking process, visible through the sequence of messages, reveals a systematic, evidence-based approach:

1. Research phase ([msg 1066]): Commission three research agents to gather intelligence on sglang updates, GLM-5 deployment, and SM120 kernel optimization.
2. Prioritization ([msg 1067]): Synthesize findings and create a prioritized todo list, with OEA implementation as a high-priority item.
3. Environment assessment (<msg id=1068-1074>): Check current server state, sglang version, and available patches. Update sglang to the latest commit.
4. Code exploration (<msg id=1081-1084>): Systematically read the MoE routing code, starting from the top of the file and working downward.
5. Precision read (message 1085): Read the specific section containing the fused top-k function call — the critical piece needed to complete the routing pipeline understanding. This is not the work of someone guessing or hacking. It is the work of an engineer methodically building a complete understanding before making a single line of modification. The assistant knows that implementing OEA correctly requires understanding every detail of the existing routing logic, including edge cases like the post-processing step.

Mistakes and Considerations

One potential concern is that the assistant is reading a static snapshot of the code. The fused_topk_deepseek function itself might be defined elsewhere (likely in a C++ CUDA extension or a Triton kernel), and its internal implementation details are not visible from this Python file. The assistant may need to examine the fused kernel implementation to understand exactly how the routing works at the GPU level.

Additionally, the assistant assumes that OEA can be implemented purely by modifying the routing logic in topk.py. However, OEA may also require changes to the expert dispatch mechanism (how experts are loaded and executed) to fully realize its benefits. The routing modification is necessary but may not be sufficient.

Conclusion

Message 1085 is a small but critical step in a sophisticated optimization campaign. It represents the transition from research to implementation — the moment when the assistant gathers the final piece of information needed to begin coding. By reading lines 780–900 of topk.py, the assistant completes its mental model of sglang's MoE routing pipeline and identifies the precise insertion points for Opportunistic Expert Activation. This message exemplifies the methodical, evidence-based approach that characterizes the entire session: research first, understand completely, then implement with precision.