The Reconnaissance Grep: How a Single Bash Command Unlocked MoE Routing Secrets for OEA Implementation

In the midst of an intensive optimization campaign targeting the GLM-5-NVFP4 Mixture-of-Experts (MoE) model running on NVIDIA RTX PRO 6000 Blackwell GPUs (SM120), the assistant issued a seemingly mundane command — a grep across a single Python file. Yet this message, [msg 1089], represents a critical moment of discovery: the bridge between theoretical research and practical implementation. It is the moment the assistant stopped reading about the Opportunistic Expert Activation (OEA) technique and began the work of actually coding it, by first understanding exactly how the target model configures its expert routing.

The Message in Full

The assistant executed the following command on the remote server:

ssh root@10.1.230.174 'grep -n "topk\|correction_bias\|num_expert_group\|topk_group\|num_experts_per_tok\|TopKConfig\|MoEGate\|top_k" /root/sglang/python/sglang/srt/models/glm4_moe.py | head -30'

The output returned:

65:from sglang.srt.layers.moe.topk import TopK
326:        self.e_score_correction_bias = nn.Parameter(
345:        self.top_k = config.num_experts_per_tok
377:            top_k=self.top_k + self.num_fused_shared_experts,
387:        self.topk = TopK(
388:            top_k=self.top_k + self.num_fused_shared_experts,
390:            renormalize=config.norm_topk_prob,
391:            use_grouped_topk=True,
392:            num_expert_group=config.n_group,
393:            topk_group=config.topk_grou...

This is the entirety of the subject message — a single bash invocation and its truncated output. But its significance extends far beyond the 30 lines it displays.

Why This Message Was Written: The OEA Implementation Context

To understand why this simple grep matters, one must look at the broader narrative. The assistant had been engaged in a weeks-long optimization campaign to maximize inference throughput for the GLM-5-NVFP4 model on eight RTX PRO 6000 Blackwell GPUs. The core bottleneck had been identified: small per-expert GEMM operations on SM120 (Blackwell's compute architecture) were memory-bandwidth-bound, with the GPU's shared memory limit of 101,376 bytes constraining the CUTLASS kernel tile sizes that could be used.

After exploring and ruling out several approaches — piecewise CUDA graphs, MSCCLPP allreduce, single-batch overlap — the assistant turned to a promising technique from recent literature: Opportunistic Expert Activation (OEA). Described in arXiv:2511.02237, OEA is a decode-time routing modification that reduces the number of unique experts loaded across a batch, directly targeting the memory-bandwidth bottleneck in MoE decoding. The paper claimed a 39% reduction in MoE layer decode latency.

In the messages immediately preceding [msg 1089], the assistant had dispatched research agents to study the OEA paper and the sglang MoE routing code. The research task (see [msg 1078]) returned a detailed analysis of where router logits are computed and top-k experts selected. But there was a problem: the research agent had analyzed the generic MoE routing infrastructure in sglang/srt/layers/moe/topk.py, but the actual GLM-5 model file could not be found. In [msg 1087], the assistant tried to grep glm_moe_dsa.py — which returned nothing. The file didn't exist. In [msg 1088], a find command revealed the available model files, including glm4_moe.py.

So [msg 1089] is the resolution of that discovery: the assistant now knows the correct file is glm4_moe.py, and this grep is the first deep look at how that specific model configures its routing. Without this information, any OEA implementation would be guesswork — the assistant needed to know the exact routing parameters used by this model.## What the Grep Output Revealed

The truncated output — ending mid-line at config.topk_grou... — nonetheless reveals a wealth of critical information. Each line is a piece of the puzzle:

  1. Line 65: from sglang.srt.layers.moe.topk import TopK — The model uses the standard TopK class from sglang's MoE layer infrastructure. This confirms that OEA can be implemented by modifying or wrapping this existing routing mechanism, rather than requiring a completely new routing path.
  2. Line 326: self.e_score_correction_bias = nn.Parameter( — The model uses a score correction bias, a learned per-expert bias that adjusts router logits before top-k selection. This is a DeepSeek-style routing enhancement that improves expert load balancing. Any OEA implementation must account for this correction bias — it cannot simply override the raw router logits.
  3. Line 345: self.top_k = config.num_experts_per_tok — The number of experts activated per token is read from the model configuration. For GLM-5, this is typically 8 experts out of a total expert pool (likely 256 or more).
  4. Lines 377-378: top_k=self.top_k + self.num_fused_shared_experts — The model has fused shared experts, a technique where certain experts are always activated alongside the routed ones. This adds complexity: the OEA mechanism must only constrain the routed experts, not the shared ones.
  5. Lines 387-393: The TopK instantiation reveals the routing configuration: - renormalize=config.norm_topk_prob — Whether to renormalize the routing weights after top-k selection - use_grouped_topk=True — The model uses grouped top-k routing, where experts are organized into groups and a minimum number of groups must be selected - num_expert_group=config.n_group — The number of expert groups - topk_group=config.topk_grou... — The number of groups to select (truncated, but critical) This grouped routing structure is the key insight for OEA. The OEA technique works by opportunistically reusing experts already loaded for other tokens in the batch, reducing the number of unique experts that must be fetched from HBM. But with grouped routing, the assistant cannot simply reduce top_k — it must respect the group constraints. The OEA implementation would need to work within or modify the grouped top-k framework.

The Thinking Process Visible in This Message

The assistant's reasoning is evident not just in what this message says, but in its position within the broader conversation. The sequence tells a story:

  1. Discovery ([msg 1087]): The assistant tried to grep glm_moe_dsa.py and found nothing — the model file didn't exist under that name.
  2. Exploration ([msg 1088]): A find command located the actual model files, revealing glm4_moe.py as the correct target.
  3. Reconnaissance ([msg 1089]): With the correct filename identified, the assistant immediately greps for routing-related patterns to understand the model's specific configuration. This is classic debugging and reverse-engineering methodology: when the expected file doesn't exist, search for the actual file, then probe its structure. The assistant could have simply read the entire glm4_moe.py file, but instead chose a targeted grep — a more efficient approach that extracts only the routing-relevant lines from what is likely a large model file. The choice of grep patterns is itself revealing. The assistant searched for: - topk — the core routing mechanism - correction_bias — the DeepSeek-style routing enhancement - num_expert_group, topk_group — grouped routing parameters - num_experts_per_tok — the model configuration parameter - TopKConfig, MoEGate, top_k — additional routing infrastructure This pattern selection shows that the assistant already knew, from the research agents, what the relevant routing concepts were. The grep was not exploratory in the sense of discovering what exists — it was confirmatory, verifying that the expected patterns exist in this specific model file and extracting their exact parameter names and values.

Assumptions and Their Implications

The message makes several assumptions, some more justified than others:

Assumption 1: glm4_moe.py is the correct model file. This was a reasonable inference from the find output in [msg 1088], which showed glm4_moe.py as the only GLM-related model file (the earlier attempt at glm_moe_dsa.py having failed). However, the assistant did not verify that the GLM-5-NVFP4 model actually uses glm4_moe.py at runtime. If the model had been updated to use a different routing configuration (e.g., a hypothetical glm5_moe.py), this grep would have been misleading. The assistant implicitly assumed backward compatibility between GLM-4 and GLM-5 routing.

Assumption 2: The grep pattern captures all relevant routing configuration. The pattern was comprehensive but not exhaustive. For example, it did not search for shared_experts or fused_shared_experts, which later turned out to be important (line 377 shows num_fused_shared_experts being added to top_k). The assistant also did not search for n_group or topk_group as standalone config attributes, instead relying on the pattern topk_group which appears in the TopK constructor call.

Assumption 3: The routing configuration is static and defined at model load time. The assistant assumed that the routing parameters visible in the model file are the ones used at runtime. This is generally true for model architecture, but OEA might need to modify routing dynamically per batch — the static configuration only tells part of the story.

Assumption 4: The truncated output is sufficient. The head -30 truncation cut off the topk_group value mid-line. The assistant accepted this and moved on rather than requesting the full line. This implies the assistant either already knew the value (from the research agent's analysis of the generic routing code) or planned to retrieve it later if needed.

Input Knowledge Required

To fully understand this message, one needs:

  1. MoE routing architecture: Knowledge of how Mixture-of-Experts models select which experts to activate per token, including top-k routing, grouped top-k, score correction bias, and fused shared experts.
  2. SGLang codebase structure: Understanding that sglang/srt/layers/moe/topk.py contains the generic routing logic, while model-specific files like glm4_moe.py configure and instantiate it.
  3. The OEA technique: Familiarity with Opportunistic Expert Activation and how it modifies routing to reduce unique expert count per batch.
  4. The GLM-5 model family: Knowledge that GLM-5 uses grouped top-k routing with a large expert count, and that it inherits routing architecture from GLM-4.
  5. The optimization context: Understanding that the assistant is in the middle of a systematic optimization campaign, having already tried and ruled out several other approaches.

Output Knowledge Created

This message produced several concrete outputs:

  1. Confirmed file location: The assistant now knows that glm4_moe.py is the correct file for the GLM-5 model's routing configuration.
  2. Identified key routing parameters: The grep revealed the exact parameter names used in the model: num_experts_per_tok, n_group, topk_group, norm_topk_prob, num_fused_shared_experts.
  3. Confirmed routing infrastructure: The model uses TopK from sglang.srt.layers.moe.topk with use_grouped_topk=True, meaning any OEA implementation must work within the grouped top-k framework.
  4. Identified the correction bias: The presence of e_score_correction_bias means OEA must handle score-adjusted routing, not just raw logits.
  5. Revealed the fused shared expert pattern: The top_k parameter includes num_fused_shared_experts, meaning some experts are always activated and should not be constrained by OEA. This knowledge directly enabled the subsequent OEA implementation. In the messages following [msg 1089], the assistant would go on to implement OEA as a decode-time routing optimization, gated by the SGLANG_OEA_K0 environment variable, and benchmark it against the baseline. The grep output provided the precise configuration parameters needed to correctly integrate OEA with the GLM-5 model's existing routing logic.

The Broader Significance

This message exemplifies a pattern that recurs throughout the optimization campaign: methodical, evidence-based decision-making. The assistant does not guess at the model's routing configuration — it reads the actual source code. It does not assume the file name — it searches for it. It does not read the entire file — it extracts only the relevant information.

This grep also represents the transition from research to implementation. The research agents had analyzed the generic routing code and the OEA paper. But implementing OEA for the specific GLM-5 model required knowing the exact routing parameters used by that model. This message bridges that gap.

In the context of the overall optimization campaign, this message is a small but essential step. The OEA implementation that followed would ultimately show near-zero average throughput improvement on random data (since random routing patterns don't benefit from expert reuse), but it would demonstrate a ~5% peak throughput improvement at high concurrency. The grep in [msg 1089] made that implementation possible by providing the routing configuration details that the implementation needed to target.

The message also illustrates a key principle of AI-assisted coding: the assistant can read and understand source code on the remote machine, but it must be told what to read. The user or system does not automatically provide model architecture details — the assistant must actively probe for them. This grep is an act of autonomous exploration, driven by the assistant's understanding of what information it still lacks.