The Baseline That Tells the Truth: A Clean A/B Benchmark in the GLM-5 Optimization Campaign

Introduction

In the middle of an intensive optimization campaign for deploying the GLM-5-NVFP4 mixture-of-experts (MoE) model on 8× NVIDIA RTX PRO 6000 Blackwell GPUs, the assistant sends a single message that embodies the entire philosophy of the project: a bash command that runs the same benchmark suite on the baseline (unoptimized) server that was just run on the OEA (Opportunistic Expert Activation) server. This message, <msg id=1156>, is deceptively simple — it is a single SSH command executing a shell loop — but it represents the culmination of hours of implementation work, the critical moment of truth in a scientific experiment, and the disciplined commitment to evidence-based decision-making that defines the entire session.

Context: The OEA Hypothesis

To understand why this message matters, we must understand what came before it. The assistant had been engaged in a weeks-long campaign to maximize inference throughput for GLM-5-NVFP4, a large MoE model with 256 experts and top-8 routing. The core bottleneck had been identified as small per-expert GEMM (general matrix multiply) operations on SM120 (Blackwell) GPUs, where the CUTLASS tile scheduler struggles with the tiny matrix dimensions produced when each expert processes only a handful of tokens.

The Opportunistic Expert Activation (OEA) optimization was the assistant's own invention — a decode-time routing optimization that reduces the number of unique experts activated per batch by "piggybacking" tokens onto experts already selected by other tokens in the same batch. The idea was elegant: if token A selects experts {1, 3, 5, 7} and token B selects experts {2, 3, 5, 9}, OEA would notice that experts 3 and 5 are already in the baseline set and preferentially route token B's remaining slots to those shared experts, reducing the total number of distinct experts that need to be computed. In theory, this would increase expert utilization (more tokens per expert → larger GEMMs → better GPU utilization) without significantly changing the model's output distribution.

The assistant had implemented OEA carefully, debugging issues with unsorted top-k indices (the fused kernel returns topk_ids in arbitrary order, not sorted by score) and fixing the sigmoid scoring for proper weight normalization. After multiple iterations — OEA v1, v2, v3 — the implementation was correct. The OEA server was running with SGLANG_OEA_K0=5 (keeping the top-5 experts per token as the immutable baseline, and applying piggybacking only to the remaining 3 slots).

The Message: A Clean Baseline Measurement

The message itself is a single bash command executed over SSH on the remote server:

ssh root@10.1.230.174 'source /root/ml-env/bin/activate && for N in 10 64 256 1024; do echo ""; echo "=== Baseline, N=$N ==="; python3 -m sglang.bench_serving --backend sglang --host 127.0.0.1 --port 8000 --model glm-5 --tokenizer lukealonso/GLM-5-NVFP4 --dataset-name random --random-input-len 128 --random-output-len 128 --num-prompts $N --request-rate 999 2>&1 | grep -E "Output token throughput|Total token throughput|Peak output|Mean TPOT|Mean ITL"; done'

This runs the bench_serving benchmark at four concurrency levels (10, 64, 256, 1024) using random input/output lengths of 128 tokens, with a very high request rate (999) to ensure maximum server saturation. The grep filters extract only the key metrics: output token throughput, total token throughput, peak output, mean time per output token (TPOT), and mean inter-token latency (ITL).

The output shows the first two data points — N=10 and N=64 — before the message is truncated (the bash output was cut off, likely due to the 15-second timeout on the bash tool). The complete results would be revealed in the next message (<msg id=1157>), where the assistant compiles the full comparison table.

Why This Message Was Written: The Scientific Imperative

This message exists because the assistant recognized a fundamental requirement of rigorous experimental science: you cannot compare an optimization against a moving baseline. Earlier in the conversation, the assistant had been comparing OEA results against baseline numbers from days earlier — numbers that were collected with a different sglang commit, different server parameters, and potentially different system states. When the assistant updated sglang to the latest commit (which alone yielded a 2× throughput improvement), all previous baselines became invalid.

The decision to stop the OEA server, start a fresh baseline server, and run the exact same benchmark suite was a conscious choice to prioritize clean data over convenience. It would have been easier to simply compare against the numbers already in the findings document. But the assistant understood that confounded variables — the sglang update, changes in GPU thermal state, differences in server configuration — could mask or exaggerate the true effect of OEA.

This is the mark of a mature engineering approach: the willingness to spend additional time (server restarts take 60-90 seconds each, and the full benchmark suite takes several minutes) to produce trustworthy results.

The Thinking Process Visible in the Message

While the message itself does not contain explicit reasoning (it is a single tool call), the surrounding messages reveal the assistant's thought process. In <msg id=1148>, immediately after receiving the OEA benchmark results, the assistant says: "Now let me stop the OEA server and run the same benchmarks on baseline for a clean A/B comparison." This is followed by a careful server shutdown sequence: pkill -f "sglang.launch_server", a sleep to allow GPU memory to drain, verification with nvidia-smi that memory usage is at 0 MiB, and then starting the baseline server.

The assistant then waits for the server to become ready by polling for the "ready to roll" string in the log file, with a timeout of 60 iterations at 10-second intervals. This patience — waiting up to 10 minutes for the server to load the model — demonstrates an understanding that benchmarking is only valid when the system is in a known, stable state.

Input Knowledge Required

To understand this message, one needs to know:

1. The GLM-5-NVFP4 model: A Mixture-of-Experts model with 256 experts, top-8 routing, using NVFP4 (NVIDIA FP4) quantization. It is deployed using SGLang, a serving framework for large language models.
2. The OEA optimization: A custom routing algorithm that opportunistically reuses experts already activated by other tokens in the batch, implemented as a patch to SGLang's MoE routing code.
3. The hardware: 8× NVIDIA RTX PRO 6000 Blackwell GPUs (SM120 architecture), connected via NVLink with P2P (peer-to-peer) access enabled.
4. The benchmark methodology: Using SGLang's bench_serving tool with random datasets, measuring throughput at multiple concurrency levels to understand both low-load latency and high-load saturation behavior.
5. The concept of A/B testing: The understanding that optimizations must be compared against a baseline measured under identical conditions, not against historical data that may be confounded by other changes.

Output Knowledge Created

This message produces the baseline throughput numbers that enable the critical comparison. The full results (as revealed in <msg id=1157>) show:

| Concurrency | Baseline Output | OEA Output | Delta | |---|---|---|---| | 10 | 38.46 | 38.40 | -0.2% | | 64 | 218.74 | 218.36 | -0.2% | | 256 | 718.83 | 716.66 | -0.3% | | 1024 | 1,603.04 | 1,599.13 | -0.2% |

The conclusion is stark: OEA produces no measurable improvement in average throughput on random data. The peak output shows a modest improvement (3.6% at 256, 5.9% at 1024), but the averages are identical within noise.

This is a negative result — and a valuable one. The assistant had invested significant effort in designing, implementing, debugging, and testing OEA. The clean A/B comparison allows the assistant to confidently conclude that OEA's benefit depends on non-uniform expert routing patterns (which do not occur with random input data) and that for this workload, the optimization is not worth the complexity.

Assumptions and Potential Pitfalls

The assistant made several assumptions in this benchmark:

1. That random input data is representative: The benchmarks use random input/output sequences with uniform token distributions. Real-world traffic may have different characteristics — conversational patterns, document processing, or code generation could produce non-uniform expert routing. The assistant acknowledges this in <msg id=1157>: "with random input data, expert routing is approximately uniform. OEA's benefit depends on expert routing patterns having natural clustering."
2. That the baseline server is identical to the OEA server: The baseline server uses run_tp8_cds16.sh while the OEA server uses run_tp8_oea.sh. These scripts may differ in parameters other than the OEA flag (e.g., --mem-fraction-static, --max-running-requests). The assistant does not verify that the only difference between the two servers is the OEA environment variable.
3. That the benchmark is stable: Running a single benchmark at each concurrency level (rather than multiple runs with statistical analysis) assumes that the numbers are reproducible. The assistant does not report variance or run multiple trials.
4. That the truncation of the bash output doesn't affect the results: The message is cut off mid-output, showing only N=10 and N=64. The assistant relies on the full output being captured and processed correctly, which it is — the next message shows the complete data.

The Deeper Significance

This message is a microcosm of the entire optimization campaign. It demonstrates:

• Discipline: The willingness to restart servers and rerun benchmarks rather than take shortcuts
• Skepticism: The refusal to trust one's own optimization without rigorous testing
• Transparency: The clean presentation of results, both positive and negative
• Iteration: The cycle of hypothesize → implement → test → analyze → document The assistant could have declared OEA a success based on the earlier comparison against stale baselines (which showed a 5.7% improvement). Instead, they insisted on a proper A/B test. That test revealed the truth: OEA, as implemented, does not improve average throughput on this workload. This negative result is arguably more valuable than a positive one — it saves future effort chasing a dead end and deepens the team's understanding of where the real bottlenecks lie (small per-expert GEMMs on SM120, not expert diversity).

Conclusion

Message <msg id=1156> is a single bash command, but it represents the scientific method applied to systems optimization: form a hypothesis, implement it, then test it against a clean baseline under controlled conditions. The assistant's commitment to this process — even when it means extra work and the possibility of disappointing results — is what makes the overall campaign trustworthy. The baseline numbers produced here allowed the team to confidently rule out OEA as a significant optimization for their workload, redirecting effort toward more promising approaches like EP8 (expert parallelism) and kernel-level tuning of the FP4 GEMM operations on SM120.