Reading the MoE Routing Code: A Pivotal Information-Gathering Step in the GLM-5 Optimization Campaign

The Message

In message 1084 of this extensive optimization session, the assistant executed a single, focused command:

ssh root@[REMOTE_SERVER] 'sed -n "640,780p" /root/sglang/python/sglang/srt/layers/moe/topk.py'

This command connects via SSH to a remote machine (the 8× RTX PRO 6000 Blackwell GPU server) and uses sed to print lines 640 through 780 of the file topk.py from the sglang codebase. On its surface, this is a mundane file-reading operation — a developer peeking at a specific section of source code. But in the context of the broader optimization campaign, this message represents a critical juncture where months of research, experimentation, and benchmarking converge into a concrete implementation effort.

The Strategic Context: Why This Message Was Written

To understand why this particular file read matters, we must examine the arc of the entire session. The assistant and user have been engaged in a methodical campaign to maximize inference throughput for the GLM-5-NVFP4 model — a 744B-parameter Mixture-of-Experts (MoE) language model — on a cluster of 8× NVIDIA RTX PRO 6000 Blackwell GPUs (SM120 architecture). The campaign had already achieved remarkable results, pushing throughput from an initial ~880 tokens/second to approximately 3,740 tok/s through a series of optimizations including FlashInfer CUTLASS MoE autotuning for SM120, server parameter tuning, and various engineering improvements.

However, the team had identified the core remaining bottleneck: small per-expert GEMMs on SM120. The Blackwell RTX PRO 6000 GPUs have 100KB of shared memory per SM, which constrains the tile sizes that CUTLASS grouped GEMM kernels can use for the MoE expert computations. Each token in the batch activates only 8 out of 256 experts per layer, and at decode time, the batch sizes are small enough that the expert GEMMs are memory-bandwidth-bound rather than compute-bound. This means the GPU's compute units are starved while waiting for data to arrive from HBM.

The assistant had just completed a round of research that included reading improvement documents, investigating BTankut's SM120 MoE configuration work, and — most critically — researching Opportunistic Expert Activation (OEA), a technique from arXiv:2511.02237 that claims 39% reduction in MoE layer decode latency. OEA works by modifying the decode-time routing to reduce the number of unique experts loaded across a batch. Instead of each token independently selecting its top-k experts (which can result in many distinct experts being loaded), OEA opportunistically steers tokens toward experts that are already loaded for other tokens in the same batch, reducing memory traffic at the cost of slightly suboptimal expert selection.

The research agent had already explored the sglang MoE routing code and identified that the top-k expert selection happens in the topk.py file. The assistant began reading this file in message 1083 (lines 520–640) and now continues with lines 640–780 in message 1084. This is not random browsing — it is a targeted information-gathering operation with a clear objective: understand the exact routing code structure to implement OEA.

Input Knowledge Required

To fully grasp the significance of this message, one needs several layers of contextual knowledge:

  1. The GLM-5-NVFP4 model architecture: A 744B MoE model with 256 experts per layer, 8 activated per token, 80 layers, and 40B active parameters. The model uses NVFP4 (4-bit floating point) quantization for expert weights, which is critical because FP4 GEMMs have different performance characteristics than standard FP16 or FP8 operations.
  2. The SM120 (Blackwell) GPU architecture: Specifically, the 100KB shared memory limit per SM, the use of mma.sync instructions (not the newer tcgen05 found in SM100/B200), and how these constraints affect CUTLASS tile selection for grouped GEMM kernels.
  3. The sglang inference engine architecture: How MoE layers are implemented, where routing happens, how expert parallelism works, and the relationship between MoEGate.forward() (router logits computation) and topk_selftype() (top-k selection).
  4. The OEA technique: The paper's approach of reducing unique expert count per batch by modifying the routing to prefer already-loaded experts, and how this interacts with the top-k selection process.
  5. The remote server setup: The machine at [REMOTE_SERVER] running Ubuntu with 8× RTX PRO 6000 GPUs, the sglang codebase at /root/sglang, and the Python virtual environment at /root/ml-env.
  6. The session history: The assistant had already read lines 520–640 of the same file in the previous message (msg 1083), which showed the tail end of the topk_selftype() function including the actual torch.topk() call that selects the top-k experts. Lines 640–780 would reveal the function signature and the beginning of the main routing logic.

Output Knowledge Created

This message produces a specific piece of output: the content of lines 640–780 of topk.py. While we don't see the actual output in this message (it would appear in the next round), the assistant is gathering this information to:

  1. Identify the exact function signature of the top-k routing function, including all parameters and their types.
  2. Understand the data flow from router logits → sigmoid scores → top-k selection → expert dispatch.
  3. Locate the precise insertion point for OEA logic — where token-level top-k selections could be modified to reduce unique expert count.
  4. Determine what data structures are available at the routing point (e.g., topk_ids, topk_weights, scores, num_token) that OEA would need to manipulate.
  5. Assess the complexity of modifying the routing code — whether it's a simple patch or requires significant refactoring. This knowledge is immediately actionable: once the assistant understands the routing code structure, it can write the OEA implementation, deploy it, benchmark it, and determine whether the technique yields meaningful improvements on this specific hardware/model combination.

The Thinking Process Visible in the Reasoning

Although this message contains no explicit reasoning text (it is a bare bash command), the reasoning is visible in the sequence of actions leading up to it:

  1. Research phase (messages 1066–1069): The assistant launched three parallel research agents to investigate sglang SM120 updates, GLM-5 deployment strategies, and SM120 FP4 kernel optimization. It also specifically researched BTankut's SM120 MoE configs and the OEA paper.
  2. Prioritization (message 1069): The assistant updated its todo list, marking research tasks as completed and preparing for implementation. The decision to pursue OEA reflects a judgment that this technique offers the best potential impact among available options.
  3. Environment preparation (messages 1076–1080): The assistant updated sglang to the latest commit (9 commits ahead), reinstalled it, and fixed a transformers version downgrade issue. These steps ensure the codebase is current and the environment is stable before making modifications.
  4. Code exploration (messages 1081–1083): The assistant began reading topk.py from the beginning, then jumped to lines 520–640 to see the core top-k selection logic. Message 1084 continues this exploration at lines 640–780. The deliberate, incremental reading pattern (start → middle section → next section) reveals a systematic approach: the assistant is building a mental model of the routing code before attempting any modifications. It is not skimming randomly but reading in contiguous blocks to understand the full function structure.

Assumptions and Potential Pitfalls

Several assumptions underpin this message:

  1. The OEA technique is applicable to GLM-5: The assistant assumes that the OEA paper's findings generalize to the GLM-5 model architecture. However, OEA was likely tested on different model families (e.g., Mixtral, DeepSeek), and GLM-5's specific expert routing patterns may respond differently.
  2. The routing code in topk.py is the correct insertion point: The assistant assumes that modifying the top-k selection function is sufficient to implement OEA, rather than needing changes elsewhere in the MoE pipeline (e.g., in the grouped GEMM dispatch or the attention layer).
  3. The sglang codebase is stable enough to modify: The assistant had just updated sglang and merged its existing patches cleanly, but further modifications could introduce conflicts or unexpected interactions with other components.
  4. Performance gains will be measurable: The assistant assumes that OEA's benefits (reduced memory bandwidth from fewer unique experts) will outweigh the costs (suboptimal expert selection). On random input data, where expert routing is inherently uniform, OEA may show minimal gains — a possibility the assistant would later confirm through benchmarking.
  5. The remote server is accessible and the file path is correct: This is a practical assumption that the SSH connection will succeed and the file exists at the expected location.

The Broader Significance

Message 1084 is a microcosm of the entire optimization campaign: methodical, evidence-driven, and relentlessly focused on understanding the system before modifying it. The assistant does not blindly implement OEA based on a paper abstract; it first reads the actual code, understands the data structures, and identifies the precise modification point. This approach — research → understand → implement → benchmark → iterate — is the hallmark of serious systems optimization work.

The message also illustrates a key principle of AI-assisted coding sessions: the assistant acts as both researcher and engineer, seamlessly transitioning from reading academic papers to reading source code to writing patches. The SSH command is the bridge between understanding and action — the moment when theoretical knowledge meets practical implementation.

In the larger narrative of the GLM-5 optimization campaign, this message marks the beginning of the OEA implementation effort. The assistant would go on to implement OEA as an environment-variable-gated feature (SGLANG_OEA_K0), benchmark it against the baseline, and discover that — on random data — it yields near-zero average throughput improvement (though peak throughput improves ~5% at high concurrency). This negative result is itself valuable: it confirms that OEA's benefit depends on non-uniform expert routing patterns, which may not be present in typical benchmark workloads. The careful, systematic approach visible in message 1084 — reading the code before writing it — is what makes such clear conclusions possible.