The Critical A/B Test: Methodological Rigor in ML Optimization

A Pivotal Moment in the GLM-5 Optimization Campaign

In the course of an intensive optimization campaign for deploying the GLM-5-NVFP4 model on 8× NVIDIA RTX PRO 6000 Blackwell GPUs, a single message captures the essence of scientific rigor in ML engineering. Message [msg 1148] reads:

Now let me stop the OEA server and run the same benchmarks on baseline for a clean A/B comparison: ``bash ssh root@10.1.230.174 'pkill -f "sglang.launch_server" 2>/dev/null; sleep 8; nohup bash /root/run_tp8_cds16.sh > /root/sglang-server-baseline3.log 2>&1 &' `` Baseline server starting...

On its surface, this is a simple server restart command. The assistant stops one running process and launches another. But this seemingly mundane action represents a critical inflection point in a weeks-long optimization journey — the moment when a promising but unproven optimization technique is put to the test against a clean baseline under rigorously controlled conditions.

The Context: An Optimization Odyssey

To understand why this message matters, we must trace the arc of the optimization campaign that preceded it. The assistant had been systematically working through a prioritized list of optimization ideas for the GLM-5-NVFP4 model — a large Mixture-of-Experts (MoE) language model with 256 experts, running on cutting-edge Blackwell GPUs (SM120 architecture). Each optimization was documented, implemented, benchmarked, and either adopted or discarded based on empirical evidence.

One of the most intriguing ideas was Opportunistic Expert Activation (OEA) — a decode-time routing optimization that exploits the observation that at any given batch size, not all 256 experts are equally utilized. OEA works by identifying which experts are already "active" in the batch (selected by any token) and preferentially routing additional tokens to those experts rather than activating new ones. The hypothesis was that by reducing the number of unique experts activated per batch, OEA could improve GPU utilization through better cache locality and reduced expert fragmentation.

The assistant had implemented OEA as a gated feature (controlled by the SGLANG_OEA_K0 environment variable), debugged it, fixed a subtle sorting bug where topk_ids were not guaranteed to be ordered by score, and run initial benchmarks. The results were interesting but inconclusive: at 1024 concurrency, OEA showed a 5.7% improvement in output throughput (1,607 vs 1,520 tok/s) and a 25% improvement in peak output (4,012 vs 3,210). However, these numbers were compared against a baseline that had been measured at a different time, under potentially different conditions.

The Methodological Insight

This brings us to the critical insight captured in message [msg 1148]. The assistant recognized that the existing baseline comparisons were not truly controlled. The baseline numbers had been collected from earlier server runs, possibly with different server configurations, different GPU thermal states, different memory fragmentation patterns, or even different versions of the sglang runtime. For a rigorous evaluation, the assistant needed to run the exact same benchmark suite on a freshly started baseline server — ideally interleaved with the OEA runs to minimize temporal confounding factors.

The message reveals this recognition explicitly: "run the same benchmarks on baseline for a clean A/B comparison." The word "clean" is telling — it signals an awareness that the previous comparisons were contaminated by uncontrolled variables.

The Mechanics of the Switch

The command itself is instructive. It consists of two parts:

pkill -f "sglang.launch_server" — This forcefully terminates any running sglang server process. The -f flag matches the full command line, ensuring that all server instances are killed. The 2>/dev/null suppresses error messages if no matching process is found.
nohup bash /root/run_tp8_cds16.sh > /root/sglang-server-baseline3.log 2>&1 & — This launches a new server using the run_tp8_cds16.sh script, which presumably configures the server without OEA enabled. The nohup ensures the process survives shell termination, output is redirected to a log file, and the & backgrounds the process. The sleep 8 between kill and launch gives the GPUs time to cool down and the CUDA driver time to release resources. This is a small but important detail — it prevents GPU memory conflicts and ensures a clean initialization. The log file name sglang-server-baseline3.log is also revealing. The "3" suffix indicates this is at least the third baseline server instance, suggesting that the assistant has been iterating on server configurations and restarting as needed. Each restart is an opportunity to reset state and eliminate accumulated issues.

What This Message Reveals About the Optimization Process

This single message illuminates several important aspects of the assistant's working methodology:

1. Evidence-Based Decision Making

The assistant does not assume OEA works based on the initial promising results. Instead, it insists on a proper controlled comparison before drawing conclusions. This is the hallmark of rigorous empirical science — the willingness to test one's own hypotheses under the most stringent conditions.

2. Awareness of Confounding Variables

By restarting the server for the baseline measurement, the assistant eliminates several potential confounders: GPU thermal throttling that may have accumulated during the OEA runs, memory fragmentation from the previous workload, and any stateful behavior in the sglang runtime. This demonstrates a sophisticated understanding of the experimental pitfalls in ML systems benchmarking.

3. Systematic Iteration

The message is part of a larger pattern of systematic iteration. Throughout the conversation, the assistant cycles through: implement → test → analyze → document → compare. Each cycle produces new knowledge that informs the next decision. The OEA implementation itself went through multiple iterations (v1, v2, v3) as bugs were discovered and fixed.

4. The Cost of Rigor

There is an implicit cost to this approach: time. Restarting the server takes at least 8 seconds plus model loading time (which for a 256-expert MoE model on 8 GPUs could be several minutes). Running the full benchmark suite at multiple concurrency levels takes additional time. The assistant is willing to pay this cost because the alternative — drawing incorrect conclusions from contaminated data — would be more costly in the long run.

The Broader Implications

Message [msg 1148] is not just about OEA. It represents a template for how optimization work should be conducted in production ML systems. The temptation in such campaigns is to chase promising numbers and declare victory early. The assistant's insistence on clean A/B testing is a defense against this temptation.

This methodological rigor is especially important in the context of MoE models on Blackwell GPUs, where the optimization landscape is poorly understood. The SM120 architecture is new, the FP4 compute path is novel, and the interaction between expert routing, GPU kernel scheduling, and memory bandwidth is complex. In such an environment, the only reliable guide is careful measurement under controlled conditions.

The Outcome

While the message itself does not contain the benchmark results, the subsequent conversation reveals that the clean A/B comparison confirmed OEA's modest improvement at high concurrency but showed near-zero average gain on random data. This led to the conclusion that OEA's benefit depends on non-uniform expert routing patterns — a finding that could only be established through the kind of rigorous testing that message [msg 1148] represents.

The lesson is clear: in ML optimization, the most important tool is not a faster kernel or a clever algorithm, but the discipline to test one's ideas properly. Message [msg 1148] is a testament to that discipline.