The Sigmoid Fix: Diagnosing Why Opportunistic Expert Activation Underperformed at High Concurrency

In the long arc of an optimization campaign targeting the GLM-5-NVFP4 model on eight NVIDIA RTX PRO 6000 Blackwell GPUs, message [msg 1125] represents a moment of reflective diagnosis. The assistant had just implemented Opportunistic Expert Activation (OEA), a decode-time routing optimization inspired by the paper "Oncescu, Wu, Chung, Wu, Gopal, Wang, Tri Dao, Athiwaratkun" (arXiv:2511.02237), benchmarked it, discovered a subtle bug in the implementation, fixed it, and then paused to interpret the results before deciding the next experimental direction. This message captures the transition from "does it work?" to "why does it work this way, and where should we look next?" — a crucial inflection point in any systematic optimization effort.

The Context: A Methodical Optimization Campaign

By the time the assistant wrote message [msg 1125], it had already spent many rounds building and refining the GLM-5-NVFP4 inference stack. The model is a Mixture-of-Experts (MoE) architecture with 256 experts, running with tensor parallelism across 8 GPUs (TP8). The assistant had previously updated sglang to the latest commit — which alone yielded a 2× throughput improvement — and had implemented OEA as a new optimization gated by the environment variable SGLANG_OEA_K0 (see [msg 1107] and [msg 1108] for the implementation).

OEA works by opportunistically reducing the number of unique experts loaded per decode batch. In standard MoE decoding, each token in a batch independently selects its top-k experts, potentially activating many distinct experts across the batch. OEA attempts to "piggyback" some tokens onto already-loaded experts when the routing scores are close, reducing the expert computation footprint. The implementation was careful: it handled unsorted top-k output, used proper sigmoid scores for weight gathering, and was gated by a minimum batch size (_OEA_MIN_BATCH=2) to avoid interfering with single-token decoding.

The Benchmark Results That Prompted Reflection

The assistant had run benchmarks comparing baseline performance against OEA with k0=5 (meaning it would keep the top-5 baseline experts and attempt piggyback routing for the rest). At concurrency 1024, the results showed:

• Baseline: 1,520.55 tok/s output throughput, 3,028.87 total tok/s, peak 3,210 tok/s
• OEA k0=5: 1,607.61 tok/s output throughput, 3,202.29 total tok/s, peak 4,012 tok/s This was a 5.7% improvement in output throughput and a 25% improvement in peak throughput — non-trivial, but far below what one might hope for from a routing optimization. The assistant's immediate reaction in [msg 1122] was enthusiastic ("The OEA is working!"), but by [msg 1123] a more critical question emerged: was the implementation even correct?

Discovering the Sigmoid Bug

In [msg 1123], the assistant identified a subtle but important issue. The OEA function received router_logits — raw logits from the gating network, before the sigmoid activation had been applied. The code was using router_logits.float() directly to compute scores for weight gathering:

scores = router_logits.float()  # (B, N)

But the actual routing code in biased_grouped_topk_impl applies a sigmoid to these logits before using them as routing probabilities. The assistant realized that while using raw logits for ranking experts is fine (sigmoid is monotonic, so the order is preserved), using raw logits for weight gathering (renormalization) is incorrect. The weights need to be proper sigmoid probabilities to match what the original routing code produces.

The fix was applied in [msg 1124] via a sed command that changed the line to:

scores = router_logits.float().sigmoid()  # (B, N) — sigmoid scores for proper weight gathering

Message 1125: The Reflective Analysis

Message [msg 1125] opens with the assistant confirming the fix is correct: "Good. Now the weights gathered from scores will be proper sigmoid probabilities, matching what the original routing code produces. The ranking is the same either way (sigmoid is monotonic), so the expert selection logic is unchanged." This is a precise technical observation — the assistant correctly notes that the monotonicity of sigmoid means the fix only affects the magnitude of the gathered weights, not the selection of which experts to keep or piggyback.

But then the assistant pivots to a deeper question. Even with the fix, the 5.7% improvement at concurrency 1024 is modest. Why? The assistant performs a back-of-the-envelope calculation:

"The reason is clear from our analysis: at 1024 concurrency, expert saturation is already nearly 100% (1024 × 8/256 ≈ 32 tokens/expert, ~251/256 experts active). OEA can't reduce unique experts when nearly all are already needed."

This is the key insight. The GLM-5 model has 256 experts. With 8 GPUs in tensor-parallel configuration, each GPU handles 32 experts (256/8). At 1024 concurrent requests, assuming roughly uniform routing across experts, each expert receives approximately 32 tokens per batch (1024 × 8/256 = 32). With top-2 routing (each token selects 2 experts), that's roughly 64 tokens per expert, but the key point is that at this concurrency level, nearly all 256 experts are activated in every decode step. There is almost no "slack" for OEA to exploit — you cannot reduce the number of unique experts when every expert is already needed.

The Sweet Spot Hypothesis

The assistant then formulates a clear hypothesis: "The sweet spot for OEA should be at medium concurrency (32-128) where expert overlap is partial." This is a well-reasoned prediction. At low concurrency (e.g., 1-16 tokens), each token selects its top-2 experts, and with 256 experts available, there's virtually no overlap — every token activates a nearly unique set of experts. OEA has nothing to piggyback onto. At very high concurrency (256+), expert saturation is nearly complete — all experts are active anyway, so OEA can't reduce the count. The opportunity lies in the middle range where some experts are activated by multiple tokens (creating overlap that OEA can exploit) but not all experts are yet saturated.

This hypothesis directly informs the next experimental step: "Let me restart with the fix and test at those levels." The assistant issues a bash command to kill the running server and prepare for a new round of benchmarks focused on the 32-128 concurrency range.

Assumptions and Their Validity

The message rests on several assumptions, most of which are well-grounded:

1. Sigmoid monotonicity preserves ranking: This is mathematically correct — the sigmoid function is strictly monotonic, so argmax(logits) == argmax(sigmoid(logits)). The expert selection logic is indeed unchanged.
2. Expert saturation is the limiting factor at high concurrency: This is a reasonable inference from the model architecture (256 experts, top-2 routing) and the concurrency level (1024). The calculation of ~251/256 experts active is an approximation but directionally correct.
3. The sweet spot is medium concurrency: This follows logically from the saturation analysis, but it's a hypothesis to be tested, not a proven fact. Other factors could affect OEA's effectiveness at medium concurrency — for example, the batch size might be too small for the piggyback mechanism to find enough "close calls" where a token's second-best expert is close enough to the best expert to justify piggybacking.
4. The sigmoid fix will meaningfully change results: The assistant doesn't explicitly quantify how much the fix will change the weight gathering, but the implication is that it matters. In practice, the difference between raw logits and sigmoid(logits) for weight gathering could be significant — raw logits can have arbitrary magnitude, while sigmoid normalizes them to (0,1), which affects the renormalization step. One potential blind spot in the analysis: the assistant assumes that the 5.7% improvement at concurrency 1024 was entirely due to whatever OEA was doing with raw logits, and that the sigmoid fix might improve or change this. But it's also possible that at 1024 concurrency, even with correct sigmoid scores, the improvement would remain modest because the fundamental limitation is expert saturation, not the weight gathering formula. The assistant seems to recognize this implicitly by deciding to test at lower concurrency levels where the saturation bottleneck doesn't apply.

Input Knowledge Required

To fully understand this message, one needs:

• MoE routing mechanics: How tokens select experts via a gating network, what router logits are, how sigmoid/softmax converts them to probabilities, and how weight gathering works.
• The GLM-5 model architecture: Specifically that it has 256 experts with top-2 routing, and that it uses biased_grouped_topk_impl with num_expert_group=1.
• The hardware configuration: 8 GPUs with TP8, meaning each GPU handles 32 experts.
• OEA algorithm: The paper's approach of opportunistically routing tokens to already-loaded experts when the score difference is small.
• Sigmoid properties: Monotonicity, range (0,1), and how it differs from softmax.
• Benchmarking methodology: How bench_serving works, what output throughput vs total throughput means, and how concurrency levels are controlled via num-prompts and request-rate.

Output Knowledge Created

This message creates several pieces of knowledge:

1. A validated bug fix: The sigmoid correction is now applied and understood.
2. A saturation model for OEA: The insight that OEA's benefit is bounded by expert saturation, with a formula for estimating active experts at a given concurrency.
3. A testable hypothesis: The sweet spot is medium concurrency (32-128).
4. A documented decision: The next experimental step is to restart with the fix and benchmark at those levels.
5. An analytical framework: The assistant demonstrates a pattern of interpreting benchmark results, forming hypotheses about why they occurred, and designing targeted experiments to test those hypotheses.

The Thinking Process

The assistant's reasoning in this message follows a clear arc:

1. Confirm correctness of the fix (sigmoid preserves ranking, improves weight gathering).
2. Acknowledge the quantitative result (5.7% improvement is decent but not huge).
3. Diagnose the root cause using a simple mathematical model (expert saturation at high concurrency).
4. Formulate a hypothesis about where OEA would be most effective (medium concurrency).
5. Design the next experiment (restart server, benchmark at 32-128 concurrency).
6. Execute (issue the bash command to kill the server). This is textbook scientific method applied to systems optimization: observe, hypothesize, predict, test. The assistant doesn't just apply the fix and move on — it takes the time to understand why the results were what they were, and uses that understanding to guide the next experiment. The message also reveals the assistant's awareness of the broader optimization campaign. The phrase "let me document what we have and think about next steps" shows a meta-cognitive awareness of the need to synthesize findings before charging ahead. This is the mark of a mature optimization process — one that values understanding over brute-force experimentation.

Conclusion

Message [msg 1125] is a small but revealing window into the assistant's optimization methodology. It demonstrates how a seemingly simple bug fix (adding .sigmoid()) leads to a deeper analysis of why an optimization technique behaves the way it does. The assistant correctly identifies that OEA's modest improvement at high concurrency is not a bug in the implementation but a fundamental limitation of the technique in the saturated regime. By formulating the sweet spot hypothesis and designing targeted experiments to test it, the assistant turns a disappointing result into a productive research direction. This is the essence of systematic optimization: every result, even a modest one, contains information that can guide the next step.