The Moment of Truth: Committing a Null Result in the GLM-5 Optimization Campaign

The Message

[assistant] [edit] /home/theuser/glm-kimi-sm120-rtx6000bw/glm5findings.md
Edit applied successfully.

At first glance, message [msg 1158] appears to be nothing more than a mundane status notification: a file was edited, and the edit succeeded. There are no benchmark numbers here, no triumphant throughput gains, no clever algorithmic insight. Just a tool confirming it performed a write operation. Yet this single line represents the culmination of one of the most intellectually rigorous episodes in the entire opencode session: the careful, methodical evaluation and documentation of the Opportunistic Expert Activation (OEA) optimization for the GLM-5-NVFP4 model running on 8× NVIDIA RTX PRO 6000 Blackwell GPUs.

To understand why this message matters, one must trace the thread of reasoning that led to it—a thread spanning dozens of messages, multiple server restarts, a subtle correctness bug, and a clean A/B benchmark that ultimately delivered a null result.

The OEA Hypothesis: A Promising Idea

The OEA concept emerged from a fundamental observation about Mixture-of-Experts (MoE) routing. In the GLM-5 model, each token is routed to 8 experts out of 256 available. At high concurrency (1024 simultaneous requests), the model was activating nearly all 256 experts per layer because each token independently chose its top-8. The OEA idea was elegantly simple: if a batch of tokens already activates a set of experts (the "baseline" set), why not steer the remaining routing slots toward experts already in that set? This would reduce the total number of unique experts activated per layer, improving the efficiency of grouped GEMM operations and potentially increasing throughput.

The assistant implemented OEA as a decode-time routing optimization, gated by the SGLANG_OEA_K0 environment variable. The algorithm worked in phases: first, identify each token's top-k0 experts (the "baseline"); second, build a batch-wide mask of which experts are in any token's baseline; third, for each token's remaining k-k0 slots, "piggyback" onto experts already in the baseline set rather than selecting new ones.

The Two Bugs That Nearly Invalidated the Benchmark

Before the clean A/B comparison could happen, the assistant discovered and fixed two subtle bugs in the OEA implementation. The first was a sigmoid score issue ([msg 1123]): the OEA function was using raw router logits for weight gathering instead of sigmoid-transformed scores. While sigmoid is monotonic (so ranking is preserved), the renormalization step required proper probability scores. The fix was a single line change: scores = router_logits.float().sigmoid().

The second bug was more insidious ([msg 1137]). The assistant realized that torch.topk was called with sorted=False in the MoE routing code. This meant that topk_ids[:, :k0] was NOT guaranteed to contain the k0 highest-scored experts—it was an arbitrary subset of the top-k. The OEA algorithm's Phase 1 depended on taking the highest-scored experts as the baseline, and if those weren't actually the highest-scored, the entire piggybacking logic could be built on a flawed foundation.

The fix ([msg 1140]) was a careful rewrite: explicitly sort the top-k experts by their sigmoid scores using argsort, then take the first k0 from the sorted list. This guaranteed that the baseline set truly contained each token's most preferred experts.

The Clean A/B Benchmark

With the fixes applied, the assistant executed a rigorous A/B comparison ([msg 1147], [msg 1156]). The methodology was exemplary: run the same benchmark at four concurrency levels (10, 64, 256, 1024) with the OEA server, then kill it, start a baseline server with identical configuration except OEA disabled, and run the same benchmarks again. The results, compiled in [msg 1157], were unambiguous:

| Concurrency | Baseline Output (tok/s) | OEA Output (tok/s) | Delta | |---|---|---|---| | 10 | 38.46 | 38.40 | -0.2% | | 64 | 218.74 | 218.36 | -0.2% | | 256 | 718.83 | 716.66 | -0.3% | | 1024 | 1,603.04 | 1,599.13 | -0.2% |

The average throughput numbers were essentially identical—within noise. The peak throughput showed a modest improvement at 256 (+3.6%) and 1024 (+5.9%), but the average told a clear story: OEA provided no measurable benefit for this workload.

Why the Null Result Was Inevitable

The assistant's analysis of why OEA failed to improve throughput was as insightful as the implementation itself. The key insight: with random input data (as used in the benchmark), expert routing is approximately uniform. Each expert receives roughly the same number of tokens. OEA's benefit depends on natural clustering in expert routing patterns—some experts being "hotter" than others. With perfectly uniform random routing, all experts are equally loaded, and there is minimal piggybacking opportunity.

At 1024 concurrency, the situation is even worse: expert saturation is already nearly 100%. With 1024 tokens × 8 experts/token = 8192 expert slots distributed across 256 experts, each expert gets roughly 32 tokens. Nearly all 256 experts are already active. OEA cannot reduce the unique expert count when nearly every expert is already needed.

The sweet spot for OEA would be at medium concurrency (32–128) where expert overlap is partial—but even there, the random data eliminated any routing pattern that OEA could exploit. The null result was not a failure of the algorithm; it was a confirmation that OEA's applicability depends on real-world routing distributions that random benchmarks cannot capture.

The Significance of Message 1158

This brings us back to message [msg 1158]. The edit being applied to glm5findings.md was the permanent recording of these results. The assistant was doing something that distinguishes rigorous engineering from mere tinkering: documenting a null result with the same care as a positive one.

The input knowledge required to understand this message is substantial. One must grasp the MoE routing architecture of GLM-5 (256 experts, top-8 routing per token), the concept of expert saturation at high concurrency, the mechanics of grouped GEMM kernels and why reducing unique experts could improve efficiency, the difference between sigmoid scores and raw logits, and the behavior of torch.topk with sorted=False. Without this background, the edit appears trivial.

The output knowledge created by this message is equally significant. The glm5findings.md document now contains a definitive answer: OEA, as implemented, provides no measurable average throughput improvement on random data for this model and hardware combination. This is not a speculative conclusion—it is backed by clean A/B data at four concurrency levels, with the implementation verified for correctness (including the sort fix) and the overhead measured as near-zero.

Assumptions and Their Consequences

The OEA implementation rested on several assumptions, some of which proved incorrect. The assumption that topk_ids[:, :k0] contains the highest-scored experts was wrong because torch.topk was called with sorted=False. The assumption that raw logits could substitute for sigmoid scores in weight gathering was technically incorrect (though it preserved ranking). The broader assumption—that reducing unique experts per layer would improve throughput on SM120 GPUs—was not disproven by this experiment; rather, the experiment showed that random data does not create the conditions where OEA can reduce unique experts.

A more subtle assumption was that the benchmark methodology (random input/output lengths, random content) would reveal OEA's potential. In hindsight, a benchmark with non-uniform routing patterns—perhaps using real conversational data or synthetic data with topic clustering—might have shown a different result. The assistant acknowledged this limitation explicitly in the analysis.

The Documentation Philosophy

Message [msg 1158] also reveals something about the assistant's working style. Throughout the session, every optimization attempt—whether successful or not—was documented in the glb5improvement-*.md files and the comprehensive glm5findings.md. This is not merely record-keeping; it is a form of intellectual integrity. By recording null results alongside positive ones, the assistant ensures that future researchers (or the same assistant in a later session) do not waste time re-exploring dead ends.

The glm5findings.md document, which grew to over 500 lines, became the canonical record of everything learned during the optimization campaign. It covers the environment setup, the baseline benchmarks, the OEA results, the EP8 attempt and its CUTLASS tile failure, the sglang update that yielded a 2× throughput improvement, and the beginning of theoretical maximum performance analysis. Each section is grounded in real measurements rather than speculation.

Conclusion

Message [msg 1158] is a quiet moment in a loud optimization campaign. It contains no breakthrough, no dramatic speedup, no clever hack. It is simply the act of writing down what was learned. But in the context of the entire session, this message represents something essential: the discipline to test hypotheses rigorously, the honesty to accept null results, and the foresight to preserve knowledge for future use. The edit was applied successfully—and with it, the OEA chapter of the GLM-5 optimization story was closed, not with a bang, but with a well-documented footnote.