The Sigmoid Problem: Catching a Subtle Bug in Opportunistic Expert Activation

Introduction

In the high-stakes world of large language model inference optimization, even a single misplaced function call can silently degrade performance. Message 1123 of this opencode session captures a moment of precise debugging: the assistant, having just implemented an optimization called Opportunistic Expert Activation (OEA) for the GLM-5-NVFP4 model running on 8 RTX PRO 6000 Blackwell GPUs, realizes that its implementation contains a subtle mathematical error. The router_logits tensor being passed to the OEA function contains raw logits (pre-sigmoid), but the function uses these raw values for weight gathering and renormalization as if they were sigmoid scores. While the monotonic nature of sigmoid means ranking is preserved, the actual weight values used in renormalization are incorrect. This message documents the moment of discovery and the first corrective step.

The Message in Full

The assistant writes:

OEA is loaded on all 8 TP ranks + the main process. The key question is whether router_logits passed to OEA is pre-sigmoid (raw logits) or post-sigmoid (scores). Looking at the code: select_experts receives router_logits which is raw logits, and the sigmoid is applied inside biased_grouped_topk_impl or the fused kernels. My OEA function uses router_logits.float() for scoring — but these are raw logits, not sigmoid scores. For sorting/ranking this should still work (sigmoid is monotonic, so ordering is preserved), but the renormalization needs to use sigmoid scores instead of raw logits.

>

Let me fix this issue: [bash command to find the line]

The assistant then executes a bash command to locate the exact line in the source code where scores = router_logits.float() is defined, confirming the bug location at line 942 of topk.py.

Why This Message Was Written: The Reasoning and Motivation

This message emerges from a specific investigative workflow. In the preceding messages (1104–1122), the assistant had:

  1. Identified the OEA optimization opportunity: Based on the paper "arXiv:2511.02237" (Oncescu et al.), OEA reduces the number of unique experts loaded per decode batch by "piggybacking" on experts already activated by other tokens in the batch. This is particularly valuable for MoE (Mixture-of-Experts) models like GLM-5, where loading expert weights from HBM to SRAM is a major bottleneck.
  2. Implemented the OEA patch: The assistant wrote a Python script that patched /root/sglang/python/sglang/srt/layers/moe/topk.py to insert the OEA function and its call site. The function takes router_logits, topk_weights, and topk_ids as inputs, and attempts to replace some of the lower-ranked experts with already-loaded ones.
  3. Run initial benchmarks: After launching the OEA-enabled server (with SGLANG_OEA_K0=5), the assistant ran benchmarks at concurrency 10, 1, 256, and 1024. The results showed a modest 5.7% improvement in output throughput at concurrency 1024, with a 25% improvement in peak output throughput (4,012 vs 3,210 tok/s).
  4. Became suspicious: The assistant noticed that OEA was loaded on all 8 TP ranks plus the main process, and began questioning whether the router_logits tensor was in the correct format. This suspicion is the direct trigger for message 1123. The motivation for writing this message is the classic engineer's instinct: when an optimization produces results that are plausible but not dramatic, you check your assumptions. The assistant had implemented OEA based on a mental model of the data flow, but needed to verify that model against the actual code. The key question — "is router_logits pre-sigmoid or post-sigmoid?" — is the kind of precise, data-type-level question that distinguishes a careful implementation from a sloppy one.

How Decisions Were Made

This message contains one clear decision: the assistant decides to verify and fix the OEA implementation's handling of router logits vs. sigmoid scores. The decision process is visible in the reasoning:

  1. Observation: OEA is loaded on all 8 TP ranks + main process (confirmed by log inspection).
  2. Hypothesis: The router_logits tensor might be raw logits, not sigmoid scores.
  3. Code analysis: The assistant mentally traces the data flow — select_experts receives router_logits (raw logits), and sigmoid is applied later inside biased_grouped_topk_impl or fused kernels.
  4. Impact assessment: For ranking/sorting, using raw logits vs. sigmoid scores doesn't matter because sigmoid is monotonic — the order of experts by score is preserved. However, for renormalization (computing the final weights that determine how much each expert contributes), raw logits produce incorrect weight values.
  5. Action: Run a bash command to find the exact line where scores = router_logits.float() is defined, confirming the bug location. The decision to fix this issue rather than ignore it reflects a commitment to correctness. The assistant could have accepted the 5.7% improvement and moved on, but instead chose to investigate a potential source of inaccuracy. This is particularly important because the renormalization step directly affects the model's output quality — incorrect weights could produce degraded generations, even if throughput appears improved.

Assumptions Made by the User or Agent

Several assumptions underpin this message:

Assumption 1: Sigmoid monotonicity preserves ranking. The assistant correctly notes that "sigmoid is monotonic, so ordering is preserved." This is mathematically sound — the sigmoid function σ(x) = 1/(1+e^(-x)) is strictly increasing, so ranking by raw logits produces the same order as ranking by sigmoid scores. This assumption is valid.

Assumption 2: The renormalization step uses incorrect values. The assistant assumes that using raw logits instead of sigmoid scores for renormalization produces incorrect weights. This is correct — raw logits can range from negative to positive unbounded values, while sigmoid scores are bounded between 0 and 1. Using raw logits for the softmax-like renormalization would produce different (and incorrect) weight distributions.

Assumption 3: The fix is straightforward. The assistant assumes that applying sigmoid to the logits before using them for weight gathering and renormalization is the correct fix. This is likely correct for the MoE routing in GLM-5, which uses sigmoid (not softmax) for expert selection — a characteristic of the GLM architecture's "biased" routing mechanism.

Assumption 4: The bug affects benchmark results. The assistant implicitly assumes that the incorrect weight values might be affecting the benchmark results. However, it's worth noting that the OEA function's primary mechanism is expert selection (which experts to route to), not weight computation. If the weight renormalization is only used for the final weighted sum of expert outputs, and the expert selection (which experts are chosen) is based on ranking (which is preserved), then the bug might only affect the quality of generations, not the throughput. The throughput improvement from OEA comes from reducing unique experts loaded, which is purely a function of which experts are selected, not the exact weight values.

Mistakes or Incorrect Assumptions

The message itself is a correction, so the "mistake" is the original implementation's use of raw logits for renormalization. Let me examine this more carefully.

The original mistake: In the OEA function (inserted at line ~942), the code reads:

scores = router_logits.float()  # (B, N)

This treats router_logits as if they are already sigmoid scores. In reality, router_logits contains raw logits (pre-activation values), and the sigmoid is applied later in the fused kernel or biased_grouped_topk_impl.

Why this is a mistake: The OEA function uses scores for two purposes:

  1. Ranking experts: full_ranking = scores.sort(dim=-1, descending=True) — This is fine because sigmoid is monotonic.
  2. Gathering weights for renormalization: new_topk_weights = scores.gather(1, new_topk_ids.to(torch.int64)) — This is incorrect because raw logits are not probabilities/scores. Potential impact: The renormalization step divides each weight by the sum of all selected weights. If raw logits are used instead of sigmoid scores, the relative weight distribution could be distorted. For example, if one expert has a logit of 10 and another has a logit of 1, the sigmoid scores would be σ(10) ≈ 0.99995 and σ(1) ≈ 0.731, giving a ratio of about 1.37:1. But raw logits would give a ratio of 10:1, massively over-weighting the high-logit expert. But is this actually a problem for throughput? The interesting subtlety is that OEA's throughput benefit comes from reducing the number of unique experts loaded per batch. This is determined by which experts are selected (the topk_ids), not the exact weight values. Since expert selection is based on ranking (which is preserved), the bug might not affect throughput at all — it would only affect the quality of the model's output. The assistant's focus on fixing this for "renormalization" suggests awareness that output quality matters, not just raw throughput.

Input Knowledge Required

To fully understand this message, a reader needs:

  1. MoE (Mixture-of-Experts) architecture: Knowledge that MoE models have multiple "expert" sub-networks, and a router network selects which experts to use for each token. GLM-5 uses a variant with sigmoid-based routing (not softmax), where each expert can be independently activated.
  2. The sigmoid function: Understanding that σ(x) = 1/(1+e^(-x)) is monotonic (order-preserving) but maps from (-∞, +∞) to (0, 1), so raw logits and sigmoid scores have different numerical values.
  3. SGLang's codebase structure: Familiarity with the select_experts function in sglang/srt/layers/moe/topk.py, which is the routing logic for MoE models. The function receives router_logits (raw) and passes them to kernel implementations that apply the activation function internally.
  4. Tensor parallelism (TP): Understanding that the model is split across 8 GPUs (TP8), and each rank runs its own copy of the routing logic. The assistant notes OEA is loaded on "all 8 TP ranks + the main process," indicating awareness of the distributed execution model.
  5. The OEA algorithm: Knowledge of the Opportunistic Expert Activation technique from the cited paper, which reduces unique expert loading by replacing some experts with already-loaded ones from the same batch.
  6. The GLM-5 model specifics: Understanding that GLM-5 uses "biased" grouping for expert selection (biased_grouped_topk_impl), which applies sigmoid to logits before selection, and that num_expert_group=1 means all experts are in a single group.

Output Knowledge Created

This message produces several important pieces of knowledge:

  1. Bug identification: The OEA implementation uses raw logits where sigmoid scores are needed for renormalization. This is documented for future reference.
  2. Code location: The buggy line is confirmed at line 942 of topk.py: scores = router_logits.float().
  3. Correctness analysis: The assistant establishes that the ranking/sorting is unaffected (due to sigmoid monotonicity), but the weight renormalization is incorrect.
  4. Debugging methodology: The message demonstrates a pattern of verifying assumptions by tracing data flow through the code. The assistant doesn't just guess — it reasons about the code structure and then confirms with a targeted grep.
  5. Performance context: The benchmarks run before this message establish that OEA provides a 5.7% throughput improvement at concurrency 1024, with a 25% peak improvement. This creates a baseline for comparison after the fix is applied.

The Thinking Process Visible in Reasoning

The assistant's reasoning in this message is a beautiful example of systematic debugging. Let me trace the thought process:

Step 1: Observation. "OEA is loaded on all 8 TP ranks + the main process." This is a neutral observation from log inspection.

Step 2: Formulate the key question. "The key question is whether router_logits passed to OEA is pre-sigmoid (raw logits) or post-sigmoid (scores)." This is the critical insight — the assistant realizes that the meaning of the input tensor depends on where in the data pipeline it's intercepted.

Step 3: Trace the data flow. "Looking at the code: select_experts receives router_logits which is raw logits, and the sigmoid is applied inside biased_grouped_topk_impl or the fused kernels." This is a mental trace of the function call chain, showing understanding of the codebase architecture.

Step 4: Compare with implementation. "My OEA function uses router_logits.float() for scoring — but these are raw logits, not sigmoid scores." The assistant identifies the mismatch between the actual data and the assumed data.

Step 5: Analyze the impact. "For sorting/ranking this should still work (sigmoid is monotonic, so ordering is preserved), but the renormalization needs to use sigmoid scores instead of raw logits." This is a nuanced assessment — the bug is real but partial. Some parts of the algorithm work correctly (ranking), while others don't (weight computation).

Step 6: Take corrective action. "Let me fix this issue" followed by a bash command to locate the exact line. The assistant doesn't fix it immediately in this message — this message is the diagnosis, and the fix will come in subsequent messages.

What's particularly impressive is the precision of the reasoning. The assistant doesn't just say "there's a bug" — it identifies exactly which part of the algorithm is affected (renormalization) and which part is unaffected (ranking), and it provides the mathematical justification (sigmoid monotonicity). This level of analysis requires both deep knowledge of the algorithm and careful attention to the code's data types.

Broader Context: The Optimization Campaign

This message sits within a larger narrative of systematic optimization. The assistant has been iterating through a series of improvements:

Conclusion

Message 1123 captures a moment of clarity in a complex optimization campaign. The assistant, having implemented a sophisticated routing optimization, steps back to verify a fundamental assumption about data types. The discovery that router_logits contains raw logits rather than sigmoid scores is subtle — it requires understanding both the mathematical properties of the sigmoid function and the architecture of the SGLang codebase. The assistant's analysis correctly identifies that ranking is preserved (so the expert selection logic works correctly) but renormalization is incorrect (potentially affecting output quality). This message exemplifies the kind of careful, evidence-based reasoning that separates robust optimization from superficial tuning. The fix, while not yet applied in this message, is clearly scoped and ready for implementation in the next step.