The Null Result That Advanced Understanding: A/B Testing Opportunistic Expert Activation on GLM-5-NVFP4

In the long arc of optimizing inference throughput for a massive Mixture-of-Experts (MoE) language model on Blackwell GPUs, most messages in the conversation are about doing: installing drivers, patching kernels, launching servers, running benchmarks. But message [msg 1157] is about thinking. It is the moment when a carefully crafted optimization hypothesis meets the cold reality of empirical data, and the result is a near-perfect null. The assistant's response to that null — neither despair nor denial, but clear-eyed analysis and documentation — reveals the scientific mindset that defines this entire optimization campaign.

The Message

The assistant writes:

Excellent — now I have clean A/B comparison data. Let me compile the results:

>

| Concurrency | Baseline Output | OEA Output | Delta | Baseline Peak | OEA Peak | Delta | |---|---|---|---|---|---|---| | 10 | 38.46 | 38.40 | -0.2% | 73 | 73 | 0% | | 64 | 218.74 | 218.36 | -0.2% | 433 | 433 | 0% | | 256 | 718.83 | 716.66 | -0.3% | 1,428 | 1,480 | +3.6% | | 1024 | 1,603.04 | 1,599.13 | -0.2% | 4,081 | 4,320 | +5.9% |

>

The throughput numbers are essentially identical — within noise. The peak output shows improvement at 256 and 1024, suggesting OEA helps during peak-throughput phases but doesn't change the average significantly. The OEA overhead is near-zero (good), but the benefit is also near-zero for average throughput.

>

This makes sense: with random input data, expert routing is approximately uniform. OEA's benefit depends on expert routing patterns having natural clustering (some experts being "hotter" than others). With perfectly uniform random routing, all experts are equally loaded and there's minimal piggybacking opportunity.

>

Let me update the findings document with these clean results.

This short message — barely 150 words — is the culmination of hours of implementation, debugging, and benchmarking. It is a masterclass in how to respond to a negative experimental result.

The Context: What Led Here

To understand why this message matters, we need to understand the optimization landscape the assistant was navigating. The GLM-5-NVFP4 model is a massive MoE architecture running on 8× NVIDIA RTX PRO 6000 Blackwell (SM120) GPUs. The core bottleneck, as established over many prior messages, is the small per-expert GEMM (general matrix multiply) operations that cannot fully utilize the GPU's computational capacity. Each expert in the MoE layer is relatively small, so dispatching tokens to many experts results in many small, inefficient matrix multiplications that leave the GPU underutilized.

The assistant had already explored numerous optimization strategies documented in files like glb5improvement-01-piecewise-cuda-graphs.md through glb5improvement-11.md. These included piecewise CUDA graphs (blocked by API limitations), MSCCLPP allreduce (minimal gains), expert parallelism (crashed under load), and many others. Each was implemented, tested, and either adopted or ruled out.

The Opportunistic Expert Activation (OEA) idea emerged from a specific insight: at decode time, each token selects its top-k experts independently. If many tokens in the same batch select the same "popular" experts, the GPU ends up computing the same expert weights multiple times. OEA's approach was to "piggyback" less popular expert selections onto already-activated experts within the same batch. If token A selects experts [1, 2, 3] and token B selects [1, 4, 5], OEA would try to reroute token B's selection to [1, 2, 3] if experts 2 and 3 are already being computed for token A. This reduces the total number of unique experts activated per batch, increasing the batch size per expert and improving GEMM efficiency.

The Implementation Journey

The OEA implementation was not trivial. The assistant implemented it as a post-processing step on the router logits, gated by the SGLANG_OEA_K0 environment variable. The function intercepted the topk_ids output from the MoE routing layer and attempted to replace some expert selections with others already active in the batch.

But the implementation required several correctness fixes before it could be benchmarked. In messages [msg 1137] through [msg 1143], the assistant discovered two critical issues:

  1. Unsorted top-k: The fused MoE kernel uses torch.topk with sorted=False in most paths. This means topk_ids[:, :k0] — the first k0 experts in the returned tensor — are NOT guaranteed to be the k0 highest-scored experts. The OEA function assumed they were, which meant it was selecting a potentially arbitrary subset of experts as the "baseline" set rather than the truly highest-scored ones.
  2. Raw logits vs. sigmoid scores: The initial implementation used raw router_logits for scoring, but the actual routing uses sigmoid-transformed scores. Since sigmoid is monotonic, the ranking is preserved, but the weight gathering for renormalization was using incorrect values. The assistant fixed both issues: first by explicitly sorting the topk_ids by score before taking the baseline, and second by applying .sigmoid() to the scores used for weight gathering. These fixes demonstrate a deep understanding of the underlying code and a commitment to correctness — even though, as it turned out, the optimization itself didn't improve throughput.

The Benchmark Methodology

The A/B comparison in message [msg 1157] is notable for its rigor. The assistant:

  1. Ran the OEA server and benchmarked at four concurrency levels: 10, 64, 256, and 1024 (see [msg 1147]).
  2. Stopped the OEA server and started a fresh baseline server using the same configuration (see [msg 1148]).
  3. Ran identical benchmarks on the baseline server at the same four concurrency levels (see [msg 1156]).
  4. Compiled the results into a clean comparison table. This is proper experimental design. The same model, the same tokenizer, the same input/output lengths (128/128), the same request rate (999, effectively infinite — saturating the server), the same number of prompts at each concurrency level. The only variable is the OEA optimization. The assistant even ensured the baseline server was a fresh process, not contaminated by any state from the OEA run.

Interpreting the Results

The data tells a clear story. At every concurrency level, the average output throughput is within 0.3% of baseline — essentially measurement noise. The peak throughput shows a modest improvement at 256 (+3.6%) and 1024 (+5.9%), but the average is unchanged.

The assistant's interpretation is spot-on: "This makes sense: with random input data, expert routing is approximately uniform. OEA's benefit depends on expert routing patterns having natural clustering (some experts being 'hotter' than others)."

This is the key insight. The benchmark uses random input data (--dataset-name random), which means the tokens are random and their expert routing patterns are approximately uniform across all experts. In this scenario, every expert is equally likely to be selected by any token, so there is no "hot" expert to piggyback onto. The baseline set of experts for each batch already contains a near-random sample of the full expert set, and trying to reroute tokens to already-active experts doesn't help because the already-active set is itself a random subset.

In a real deployment with natural language, expert routing would likely show non-uniform patterns — some experts specialize in certain types of tokens and are activated more frequently. OEA would exploit this clustering to reduce the unique expert count per batch. But on random data, the clustering doesn't exist, so OEA can't help.

The Value of a Clean Null Result

This message is valuable precisely because it is a null result. In optimization work, negative results are often underreported, leading to a literature of only successful techniques and a biased understanding of what works. The assistant's decision to document this result in glm5findings.md — a comprehensive findings document — ensures that future researchers and engineers working on this model or similar architectures will know that OEA was tried and found ineffective on random data.

The null result also provides indirect evidence about the model's behavior. The fact that OEA doesn't help on random data but might help on real data tells us something about the expert routing patterns: they are approximately uniform for random tokens. This is consistent with the model's design, where experts are meant to specialize in different linguistic phenomena, and random tokens don't exhibit any linguistic structure.

Furthermore, the near-zero overhead of OEA (the OEA throughput is never more than 0.3% below baseline) confirms that the implementation is efficient. The sorting and gathering operations add negligible latency. This means OEA could potentially be left enabled as a "free" optimization that occasionally helps on real data without ever hurting performance.

Assumptions and Their Validation

The message reveals several assumptions the assistant made:

  1. Assumption: OEA would improve throughput by reducing unique expert count. This was the core hypothesis, and it was tested and found to be false on random data. The assumption was reasonable — reducing unique experts should increase per-expert batch size and improve GEMM efficiency — but it failed because the reduction opportunity doesn't exist on random data.
  2. Assumption: The benchmark methodology would reveal the effect. The assistant assumed that a clean A/B test at multiple concurrency levels would be sufficient to measure OEA's impact. This assumption was validated: the methodology produced clear, interpretable results.
  3. Assumption: Random data is a reasonable proxy for real data in this context. This assumption is more nuanced. The assistant explicitly acknowledges that OEA's benefit depends on non-uniform routing, which random data doesn't provide. This suggests the assistant understands the limitation of the benchmark and is careful not to overgeneralize.
  4. Assumption: The OEA implementation is correct. The assistant invested significant effort in verifying correctness (the sorting fix, the sigmoid fix). The null result is not due to a bug — the implementation is correct but the optimization simply doesn't help on this data.

The Thinking Process

The assistant's reasoning in this message is exemplary. It follows a clear pattern:

  1. Present the data in a clean, comparable format.
  2. Compute deltas to quantify the difference.
  3. Interpret the pattern: average throughput unchanged, peak throughput slightly improved.
  4. Explain the mechanism: uniform routing on random data eliminates the piggybacking opportunity.
  5. Acknowledge the limitation: the benchmark data doesn't exercise the conditions where OEA would help.
  6. Take action: document the results for future reference. This is textbook scientific reasoning. The assistant doesn't try to spin the null result as a success or make excuses for it. It accepts the data as it is and uses it to refine understanding.

Knowledge Inputs and Outputs

The message consumes several pieces of knowledge:

Conclusion

Message [msg 1157] is a turning point in the optimization campaign. After implementing a clever optimization idea, debugging it, fixing correctness issues, and running a rigorous A/B test, the assistant confronts a null result and responds with intellectual honesty and scientific maturity. The message demonstrates that the true value of an optimization campaign lies not just in the successes, but in the disciplined rejection of ideas that don't work — and in the understanding gained from each experiment, whether positive or negative.

The assistant's ability to extract insight from a null result — understanding why OEA didn't help and what conditions would be necessary for it to work — is a skill that separates effective optimization from random tinkering. This message, for all its brevity, captures the essence of what makes this conversation remarkable: a methodical, evidence-based approach to pushing the boundaries of what's possible on novel hardware.