The Art of Documenting Negative Results: How a Single Edit Captured the Culmination of an Optimization Campaign

The Message

The subject message, at first glance, appears almost trivial:

Now update the OEA section with the clean comparison: [edit] /home/theuser/glm-kimi-sm120-rtx6000bw/glm5findings.md Edit applied successfully.

A single edit to a markdown file. No code was written, no server was restarted, no benchmark was run. Yet this message represents the quiet culmination of a sophisticated optimization campaign — the moment when a carefully engineered idea was tested, measured, and conclusively evaluated, with its results permanently recorded in the project's institutional memory. This is the story of how Opportunistic Expert Activation (OEA) went from hypothesis to implementation to evidence-based conclusion, and why the act of documenting that journey matters as much as the technical work itself.

The Context: An Optimization Campaign on Blackwell GPUs

To understand this message, one must understand the broader context. The assistant had been engaged in a multi-week effort to maximize inference throughput for the GLM-5-NVFP4 model — a large Mixture-of-Experts (MoE) language model — on a system with 8 NVIDIA RTX PRO 6000 Blackwell (SM120) GPUs. This was not a straightforward deployment. The Blackwell architecture (SM120) introduced new challenges: a reduced shared memory limit (100KB vs 228KB on Hopper), unsupported FlashInfer allreduce fusion, and CUTLASS tile configuration constraints that made standard optimization recipes inapplicable.

The assistant had already explored and documented eleven distinct optimization approaches (see [msg 1129]), from piecewise CUDA graphs to MSCCLPP allreduce to expert parallelism. Each was implemented, benchmarked, and either adopted or ruled out based on real measurements. The twelfth idea — Opportunistic Expert Activation (OEA) — was born from a simple observation: at any given decode step, many tokens in a batch route to the same experts. If the system could "piggyback" tokens onto already-activated experts rather than activating new ones, it might reduce the total number of unique experts per layer, improving computational efficiency.

The OEA Experiment: From Hypothesis to Implementation

The OEA implementation was non-trivial. The assistant wrote a custom routing function that intercepts the MoE top-k selection process (see [msg 1123]-[msg 1143]). The algorithm works in phases: first, it identifies the top-k0 highest-scored experts per token from the already-computed top-k selection; second, it builds a batch-wide baseline set of unique experts from these top-k0 selections; third, for the remaining k-k0 slots per token, it preferentially selects experts already in the baseline set rather than activating new ones. The result is that fewer unique experts need to be computed per layer, at the cost of potentially suboptimal routing for individual tokens.

The implementation required careful engineering. The assistant discovered that the fused top-k kernel returns results with sorted=False, meaning the first k0 entries in topk_ids are NOT guaranteed to be the highest-scored experts — they are an arbitrary subset of the top-k. This was a correctness bug that needed fixing (see [msg 1137]-[msg 1143]). The fix involved explicitly sorting the top-k entries by their sigmoid scores before selecting the top-k0 baseline, adding a non-trivial argsort operation that itself carries computational cost.

The A/B Benchmark: A Model of Scientific Rigor

What makes this message significant is not the OEA idea itself, but the rigor with which it was tested. The assistant executed a textbook A/B comparison:

  1. Start the OEA server with the custom routing enabled (see [msg 1145]-[msg 1146])
  2. Benchmark at four concurrency levels: 10, 64, 256, and 1024 (see [msg 1147])
  3. Stop the OEA server and start a baseline server with standard routing (see [msg 1148])
  4. Benchmark the baseline at the same four concurrency levels (see [msg 1156])
  5. Compile and analyze the results (see [msg 1157]) This methodology eliminates confounding variables. The same hardware, the same model, the same benchmark script, the same input parameters — only the routing algorithm differs. The results were unambiguous: | Concurrency | Baseline Output (tok/s) | OEA Output (tok/s) | Delta | |---|---|---|---| | 10 | 38.46 | 38.40 | -0.2% | | 64 | 218.74 | 218.36 | -0.2% | | 256 | 718.83 | 716.66 | -0.3% | | 1024 | 1,603.04 | 1,599.13 | -0.2% | Average throughput was essentially identical — within measurement noise. The peak throughput showed a modest improvement at higher concurrency (3.6% at 256, 5.9% at 1024), suggesting OEA helps during transient peak-load phases but doesn't change steady-state behavior.

The Analysis: Why OEA Failed to Deliver

The assistant's analysis of why OEA showed near-zero benefit is itself a valuable piece of knowledge (see [msg 1157]). With random input data — the benchmark's default — expert routing is approximately uniform across all 256 experts. When every expert is roughly equally loaded, there is no "piggybacking" opportunity: the baseline set of unique experts is already nearly the full set. OEA's overhead (the argsort operation, the mask construction, the score gathering) cancels out any marginal benefit.

The assistant correctly identified the condition under which OEA would help: non-uniform expert routing patterns, where some experts are systematically "hotter" than others. This occurs in real-world deployments with natural language distributions — certain tokens (common words, punctuation, syntactic patterns) route to the same experts repeatedly. But the benchmark used random token sequences, which lack this structure. The OEA experiment was therefore a valid test of the mechanism but an invalid test of the use case.

The Significance of Documenting Negative Results

This brings us to the subject message itself. The assistant is not simply making an edit — it is performing an act of intellectual integrity. The OEA section in glm5findings.md will now contain the clean comparison data, the analysis, and the conclusion: OEA provides near-zero average throughput improvement on random data, though it may benefit non-uniform routing patterns.

This documentation serves multiple purposes:

  1. Institutional memory: Future readers (including the same engineer months later) will know exactly what was tried and what the results were, avoiding repeated effort.
  2. Evidence-based decision making: The conclusion is grounded in data, not speculation. The "5.9% peak improvement" is not dismissed but contextualized.
  3. Boundary conditions: The document records when OEA might help (non-uniform routing) and when it doesn't (uniform routing), providing guidance for future deployment scenarios.
  4. Scientific integrity: Negative results are as valuable as positive ones. They prevent the optimization campaign from chasing dead ends and redirect effort toward more promising approaches.

The Broader Philosophy: A Methodical Optimization Campaign

This message exemplifies the assistant's broader methodology. Throughout the session, every optimization idea was subjected to the same cycle: hypothesize, implement, benchmark against baseline, document, and either adopt or discard. The glm5findings.md document (see [msg 1135]) grew to over 500 lines, becoming a comprehensive record of what was tried, what worked, what didn't, and why.

The assistant's approach contrasts sharply with ad-hoc optimization, where changes are made based on intuition and the results are evaluated subjectively. Here, every claim is backed by numbers, every comparison is controlled, and every conclusion is caveated with its boundary conditions. The OEA edit is not the most dramatic moment in the campaign — no 2x throughput improvement, no breakthrough discovery — but it is perhaps the most characteristic. It shows what it means to do optimization the right way: not just making things faster, but understanding why they are faster, and recording that understanding for posterity.

Conclusion

The message "Now update the OEA section with the clean comparison" is a single edit to a markdown file. But it represents the culmination of a complete scientific cycle: hypothesis formation, implementation, debugging, controlled experimentation, data analysis, and permanent documentation. In the context of a high-stakes optimization campaign on cutting-edge hardware, this message is a testament to the value of rigor, the importance of negative results, and the discipline of writing things down. It is, in its own quiet way, the most important kind of message in any engineering project: the one that says "we tested this, we measured it, and here is what we learned."