The Server Restart That Sealed a Bug Hunt: Message 1145 in the GLM-5 Optimization Campaign

The Message

[assistant] [bash] ssh root@10.1.230.174 'nohup bash /root/run_tp8_oea.sh > /root/sglang-server-oea3.log 2>&1 &' && echo "OEA v3 server starting" OEA v3 server starting

At first glance, message [msg 1145] is unremarkable: a single bash command that kills the old sglang server process and launches a new one, then echoes a confirmation string. It is the kind of mechanical step that appears dozens of times in any extended ML engineering session — a restart to pick up code changes. But this particular restart is the culmination of a focused debugging chain that spanned the preceding eight messages, and it represents something far more significant than a routine bounce. This restart is the moment when three distinct bugs in a novel optimization technique called Opportunistic Expert Activation (OEA) were corrected, verified, and deployed into production. The message is a punctuation mark at the end of a detective story.

The Context: What Led Here

To understand why message [msg 1145] matters, one must understand what OEA is and why it was being implemented. The assistant was in the midst of an intensive optimization campaign for the GLM-5-NVFP4 model running on 8× NVIDIA RTX PRO 6000 Blackwell GPUs (SM120 architecture). The core bottleneck had been identified as small per-expert GEMM operations in the Mixture-of-Experts (MoE) layers — each token activates only 8 out of 256 experts, and at low-to-medium batch sizes, the GPU's tensor cores are underutilized because too few tokens map to the same expert.

OEA was designed to address this by exploiting a property of MoE routing: when the top-8 experts are selected per token, many tokens in a batch will have overlapping expert selections. OEA's insight was to "piggyback" lower-scored tokens onto experts already activated by higher-scored tokens in the same batch, reducing the total number of unique experts that need to be computed. The implementation was gated by the environment variable SGLANG_OEA_K0 and lived as a patch to sglang's topk.py routing logic.

Initial benchmarks had shown a modest 5.7% throughput improvement at high concurrency (1024), but the assistant suspected the implementation had correctness issues that were limiting its effectiveness. This suspicion launched the debugging chain that message [msg 1145] concludes.

The Two Bugs Discovered

The debugging process unfolded across messages [msg 1137] through [msg 1144], and it revealed two distinct bugs in the OEA implementation.

Bug 1: Unsorted top-k indices. The OEA function assumed that topk_ids[:, :k0] — taking the first k0 of the top-8 expert indices — would yield the highest-scored experts. This is a natural assumption: when you ask for the top-8 elements of a tensor, you expect them in descending order. However, the assistant discovered that sglang's biased_grouped_topk_impl calls torch.topk with sorted=False when num_fused_shared_experts=0 (the GLM-5 case). The comment in the code confirmed it: sorted=(True if num_fused_shared_experts > 0 else False). This meant topk_ids[:, :k0] was returning an arbitrary subset of the top-8 experts, not the highest-scored ones. The entire OEA algorithm was built on a false premise — it was selecting a "baseline" set of experts that might not actually be the most important ones for each token.

Bug 2: Cloning from the wrong tensor. Even after the assistant fixed the sorting issue (in message [msg 1140] by explicitly sorting topk_ids by score using argsort), a second bug remained. The line new_topk_ids = topk_ids.clone() was using the original unsorted topk_ids tensor as the template for the output, rather than the newly sorted topk_ids_sorted. This meant the first k0 slots of the output would still come from the unsorted original, silently corrupting the expert ordering. The assistant caught this in message [msg 1143] and fixed it with a sed substitution.

Why This Restart Is Different

Message [msg 1145] launches the server with the label "OEA v3" — the third iteration of the OEA server. The "v3" designation is earned: v1 was the initial implementation, v2 added sigmoid scores for proper weight gathering (fixing a separate issue where raw logits were used instead of sigmoid probabilities), and v3 incorporates both the sorting fix and the clone fix. Each version represents a cycle of hypothesis, implementation, testing, and debugging.

The restart command itself is structurally identical to earlier restarts — the same nohup bash /root/run_tp8_oea.sh script, the same log file pattern (sglang-server-oea3.log), the same pkill preceding it (in message [msg 1144]). But the log file name increments from oea2 to oea3, and that increment encodes the entire debugging journey. The assistant could have simply restarted without a version bump, but the explicit "v3" in the echo message signals awareness that this is a materially different deployment.

Input Knowledge Required

Understanding this message requires knowledge of several layers of context. First, one must know what OEA is and why it was invented — the MoE expert utilization problem on SM120 GPUs. Second, one must understand the sglang routing architecture: that topk_ids are produced by a fused kernel, that torch.topk can return unsorted results, and that the OEA patch lives in /root/sglang/python/sglang/srt/layers/moe/topk.py. Third, one must be familiar with the deployment topology: the server runs on a remote machine at 10.1.230.174, uses the run_tp8_oea.sh startup script, and logs to /root/sglang-server-oea*.log. Finally, one must appreciate the iterative methodology at work — each server restart is an experiment, and the version number tracks the experimental condition.

Output Knowledge Created

Message [msg 1145] produces two kinds of output. The immediate output is a running sglang server with the corrected OEA implementation, ready to serve inference requests and be benchmarked. The downstream output — visible in the following messages — is a new set of benchmark results that will determine whether the bug fixes translate into improved throughput. More broadly, the message produces a documented state transition: the OEA implementation is now at v3, with all known correctness issues resolved. Any future benchmarking against this version can be interpreted cleanly, without the caveat "but the expert selection might be wrong."

The Thinking Process Visible in the Chain

The reasoning that leads to message [msg 1145] is a textbook example of systematic debugging. It begins with a discrepancy: OEA shows only 5.7% improvement at high concurrency, but the theoretical analysis suggests it should help more at medium concurrency where expert overlap is partial. Rather than accepting the result, the assistant questions its own implementation. The key insight comes in message [msg 1137]: "I realize the OEA function has a subtle issue. The topk_ids from the fused kernel may NOT be sorted by score." This is followed by a grep to verify the hypothesis, a Python script to apply the fix, a verification step (reading the patched lines), and then a second bug discovery — the clone issue — caught during the verification read.

What is striking is the assistant's refusal to assume correctness. The original code had a comment saying "topk_ids are sorted by score" — a comment that was simply wrong. The assistant trusted the evidence (the sorted=False flag in the kernel) over the documentation, and was rewarded by finding the root cause. The second bug was found not by testing but by careful reading of the patched code: seeing new_topk_ids = topk_ids.clone() and realizing it should be topk_ids_sorted.clone(). This is the kind of bug that static analysis or code review catches, not runtime testing — and the assistant was effectively doing code review on its own patch.

The Broader Significance

Message [msg 1145] is, in one sense, just a server restart. But in the context of the GLM-5 optimization campaign, it represents the closure of a debugging loop and the beginning of a new experimental cycle. The assistant had identified two bugs, fixed them, verified the fixes, and was now deploying the corrected code for evaluation. The "v3" label signals that this is not the same experiment repeated — it is a materially different implementation that deserves its own benchmarks and its own entry in the findings document.

The message also illustrates a deeper truth about ML engineering optimization work: the most impactful improvements often come not from clever new algorithms but from finding and fixing bugs in existing ones. The OEA concept was sound — exploiting expert overlap to reduce computation — but its implementation was subtly broken in ways that would have been invisible to anyone who didn't dig into the kernel source code. The 5.7% improvement seen with the buggy version might have been accepted as "good enough" by a less rigorous engineer. The assistant's insistence on understanding why the improvement wasn't larger led directly to the discovery of the unsorted indices bug.

Conclusion

Message [msg 1145] is a single bash command that restarts an inference server. But it is also the culmination of a methodical debugging process that uncovered two subtle bugs in a novel optimization technique, the deployment of a corrected v3 implementation, and the continuation of a broader campaign to maximize GLM-5 inference throughput on Blackwell GPUs. It demonstrates that in ML engineering, the line between "implementation" and "debugging" is blurry — every optimization is a hypothesis, every benchmark is a test, and every server restart is an experiment waiting to be measured.