The Art of Negative Results: Documenting a Failed Optimization in the GLM-5 Benchmarking Campaign

Message Overview

The subject message, <msg id=1159>, is deceptively simple:

[assistant] Now update the baseline benchmark results section: [edit] /home/theuser/glm-kimi-sm120-rtx6000bw/glm5findings.md Edit applied successfully.

This brief exchange — a single tool invocation and its success confirmation — represents the culmination of an intensive, multi-hour optimization cycle. The assistant is updating a comprehensive findings document with the results of a clean A/B benchmark comparison between the baseline SGLang inference server and a custom optimization called Opportunistic Expert Activation (OEA). The message itself contains no reasoning, no analysis, no dramatic revelations. Yet it is the quiet closing of a loop that began with a hypothesis, proceeded through careful implementation and debugging, and ended with a definitive — and negative — result.

The Context: A Systematic Optimization Campaign

To understand why this message matters, one must understand the broader campaign in which it sits. The assistant has been engaged in an exhaustive effort to maximize inference throughput for the GLM-5-NVFP4 model running on eight NVIDIA RTX PRO 6000 Blackwell (SM120) GPUs. This is not a casual tuning exercise; it is a methodical, evidence-driven optimization campaign where every idea is implemented, benchmarked against a clean baseline, and either adopted or discarded based on real measurements rather than speculation.

The campaign is documented in a repository at /home/theuser/glm-kimi-sm120-rtx6000bw/, which contains a series of improvement documents (glb5improvement-01.md through glb5improvement-12.md) and a master findings document (glm5findings.md). The subject message is an edit to that master document, updating it with the latest benchmark data.

The OEA Hypothesis and Its Implementation

Opportunistic Expert Activation was the assistant's twelfth optimization attempt. The idea was elegant: in Mixture-of-Experts (MoE) models, each token is routed to a subset of experts (typically the top-k with highest router scores). During batch inference, different tokens may be routed to different experts, causing the total number of active experts to grow with batch size. OEA proposed a decode-time routing optimization: for each token, after selecting its top-k experts, the algorithm would check whether any of those experts were already activated by other tokens in the same batch. If so, it would "piggyback" on those already-loaded experts, filling remaining expert slots with experts that were already in the batch-wide working set rather than introducing new ones. The goal was to reduce the total number of unique experts activated per batch, improving GPU utilization by increasing the density of computation per loaded expert.

The implementation was non-trivial. The assistant had to carefully handle several subtleties:

  1. Unsorted top-k output: The fused top-k kernel in SGLang uses sorted=False for most routing paths, meaning topk_ids[:, :k0] does NOT guarantee the k0 highest-scored experts. The assistant discovered this bug during code review and fixed it by explicitly sorting topk_ids by score before taking the baseline subset.
  2. Sigmoid scoring: The router produces raw logits, not probabilities. The assistant initially used raw logits for scoring, then corrected this to use sigmoid-transformed scores for proper weight gathering. Since sigmoid is monotonic, the ranking is preserved, but the renormalization step requires actual probabilities.
  3. Batch-wide expert deduplication: The algorithm needed to build a batch-wide mask of which experts were already in the baseline set, then for each token's remaining slots, find experts that were both in the batch-wide set AND not already selected by that token. These fixes were applied across multiple iterations (OEA v1, v2, v3), each requiring a server restart and re-benchmarking.

The A/B Benchmarking Protocol

The assistant executed a rigorous benchmarking protocol. First, the OEA server was started with the SGLANG_OEA_K0=5 environment variable (meaning 5 of the 8 expert slots per token were reserved for the token's own top experts, and the remaining 3 were available for piggybacking). Benchmarks were run at four concurrency levels — 10, 64, 256, and 1024 — using SGLang's built-in bench_serving tool with random input/output lengths of 128 tokens each.

Then the OEA server was killed, a baseline server was started with identical configuration except without OEA, and the same four benchmarks were repeated. This produced a clean paired comparison at each concurrency level, eliminating confounding factors like server warmup state or GPU thermal conditions.

The Results: A Definitive Negative

The comparison data, compiled in the immediately preceding message <msg id=1157>, was unambiguous:

| Concurrency | Baseline Output (tok/s) | OEA Output (tok/s) | Delta | |---|---|---|---| | 10 | 38.46 | 38.40 | -0.2% | | 64 | 218.74 | 218.36 | -0.2% | | 256 | 718.83 | 716.66 | -0.3% | | 1024 | 1,603.04 | 1,599.13 | -0.2% |

Average throughput was essentially identical across all concurrency levels — the differences were within measurement noise. Peak throughput showed a modest improvement at high concurrency (+3.6% at 256, +5.9% at 1024), but this did not translate to sustained gains.

The assistant's analysis was precise: "This makes sense: with random input data, expert routing is approximately uniform. OEA's benefit depends on expert routing patterns having natural clustering (some experts being 'hotter' than others). With perfectly uniform random routing, all experts are equally loaded and there's minimal piggybacking opportunity."

This is a crucial insight. The OEA optimization was not flawed in implementation — it was fundamentally mismatched to the benchmark workload. Random input data produces uniform expert activation, leaving no clustering to exploit. The optimization might still provide value with real-world data where certain tokens or patterns consistently activate the same experts, but the benchmark methodology (random inputs) could not demonstrate it.

Why This Message Matters

The subject message is the act of recording this negative result. The assistant is updating the "baseline benchmark results section" of glm5findings.md with the comparison table and the analysis. This is not glamorous work, but it is essential to the scientific method that underpins the entire optimization campaign.

There are several reasons why this documentation step is critical:

First, it prevents wasted effort. Without a written record, future optimization attempts might revisit OEA, not knowing it was already tested and found ineffective for this workload. The findings document serves as a institutional memory that accumulates knowledge across the campaign.

Second, it captures negative results. In optimization work, negative results are as valuable as positive ones. Knowing that OEA provides no benefit on random data — and understanding why — informs future decisions about which optimizations to pursue. The assistant explicitly notes that OEA might still help with non-uniform routing patterns, preserving that knowledge for future work with real datasets.

Third, it closes the loop. Each optimization in the campaign follows a lifecycle: hypothesis → implementation → testing → documentation → either adoption or rejection. The subject message represents the final "documentation" step for OEA, formally closing that loop and allowing the assistant to move on to the next optimization.

Fourth, it demonstrates intellectual honesty. The assistant invested significant effort in OEA — writing the implementation, fixing bugs across multiple iterations, running benchmarks, and even writing a dedicated improvement document (glb5improvement-12-oea.md at <msg id=1129>). Despite this investment, when the data showed no benefit, the assistant recorded the result faithfully rather than searching for post-hoc justifications or moving the goalposts.

The Thinking Process Behind the Message

The subject message is a tool call, so it does not contain explicit reasoning. But the reasoning is visible in the surrounding messages. In <msg id=1157>, immediately before the subject message, the assistant:

  1. Compiled the comparison table from the raw benchmark outputs
  2. Computed deltas and identified the pattern (near-zero average improvement, modest peak improvement)
  3. Analyzed why the result occurred (uniform random routing leaves no clustering to exploit)
  4. Read the existing findings document to understand its structure
  5. Decided to update it with the new data The subject message then executes that decision. The follow-up messages (<msg id=1160> through <msg id=1163>) continue the documentation process, updating the OEA section, the outstanding work section, and the lessons learned section in separate edits. This sequential editing approach — one section at a time rather than a single monolithic update — reflects a deliberate strategy. Each edit is focused and verifiable. If an edit fails or produces unexpected results, the assistant can identify and fix the specific section without redoing the entire document. It also allows the assistant to maintain coherence across the document by reading the relevant section before editing it.

Assumptions and Their Validity

The assistant made several assumptions in this work:

Assumption: OEA would reduce unique expert count. This is the core hypothesis of the optimization. It is theoretically sound — if multiple tokens in a batch select overlapping experts, OEA can reduce the total number of distinct experts that need to be loaded. The assumption was validated by the implementation (the code correctly identifies and exploits overlap), but the benefit depends on the data distribution.

Assumption: Random input data is a reasonable benchmark. This assumption is more questionable. The assistant explicitly acknowledges that OEA's benefit depends on non-uniform routing, which random data does not provide. However, random data is a standard benchmark workload because it is reproducible and avoids data-dependent confounding. The assistant correctly interprets the negative result as "OEA doesn't help on random data" rather than "OEA doesn't work."

Assumption: The benchmark methodology produces stable, comparable results. The assistant runs multiple concurrency levels, uses the same server configuration (except for the OEA flag), and measures both average and peak throughput. This is a sound methodology. The near-identical results at low concurrency (38.46 vs 38.40 tok/s at N=10) confirm that the baseline and OEA configurations are otherwise identical.

Input Knowledge Required

To fully understand this message, one needs:

Output Knowledge Created

The subject message produces an updated glm5findings.md document that now includes:

Conclusion

Message <msg id=1159> is a quiet moment in a noisy optimization campaign. It is the act of writing down a negative result — the documentation of an optimization that did not work as hoped. But in the context of rigorous, evidence-driven engineering, this documentation is as valuable as any breakthrough. It closes a loop, preserves knowledge, prevents future wasted effort, and demonstrates a commitment to truth over narrative. The assistant invested hours in OEA — implementing, debugging, fixing, benchmarking — and when the data came back negative, the response was not denial or rationalization, but a simple edit to the findings document. That is the mark of a mature engineering practice.