The Moment of Truth: Benchmarking Opportunistic Expert Activation After Three Iterations of Bug Fixing

In the high-stakes world of large language model inference optimization, there is a moment every engineer dreads: the moment when you've implemented a clever optimization, fixed two subtle correctness bugs, restarted the server for the third time, and now you must run the benchmark that will tell you whether all that effort was worth it. Message [msg 1147] captures exactly this moment in the context of deploying the GLM-5-NVFP4 model on 8× NVIDIA RTX PRO 6000 Blackwell GPUs. The assistant issues a single bash command that loops over four concurrency levels (10, 64, 256, 1024) and benchmarks the Opportunistic Expert Activation (OEA) optimization — a decode-time routing technique designed to reduce the number of unique experts activated per batch, thereby improving throughput on the memory-bandwidth-constrained SM120 architecture.

The Long Road to This Benchmark

To understand why this message matters, one must appreciate the journey that led to it. The OEA optimization was born from a fundamental observation about Mixture-of-Experts (MoE) inference: at any given decode step, a batch of tokens collectively activates a set of experts. The GLM-5 model uses 256 experts with top-8 routing per token, meaning each token independently selects its 8 most relevant experts. When the batch is large, the union of all selected experts can approach all 256 — but if many tokens agree on which experts are most useful, the effective expert count could be much lower. OEA exploits this by "piggybacking" tokens onto experts already activated by other tokens in the batch, reducing the total number of expert GEMM kernels that must be launched.

The implementation had been through two significant bug fixes before this benchmark. First, in [msg 1123], the assistant discovered that the OEA scoring function was using raw router_logits (pre-activation values) instead of sigmoid scores. While sigmoid is monotonic and preserves ranking, the weight gathering step needed proper probability values for correct renormalization. The fix was straightforward — adding .sigmoid() to the scores computation.

The second bug was more subtle and was uncovered in [msg 1137]. The assistant was examining the sglang source code and noticed that torch.topk was called with sorted=False in most routing paths. This meant that topk_ids[:, :k0] — the slice of the top-k expert IDs that OEA used as its "baseline" set — was not guaranteed to contain the k0 highest-scored experts. It was an arbitrary subset of the top-k, determined by whatever order the kernel happened to produce. This was a correctness issue that could cause OEA to select suboptimal experts for the baseline, potentially degrading quality. The fix in <msg id=1140-1144> explicitly sorted the topk_ids by score before taking the top-k0 slice, ensuring the baseline always contained the highest-scored experts.

The Architecture of the Benchmark

The bash command in [msg 1147] is a model of systematic evaluation. It iterates over four concurrency levels — 10, 64, 256, 1024 — chosen to probe different regimes of expert utilization. At N=10, with 8 experts per token, the expected unique expert count is roughly 80 (10 × 8), far below the 256 available, so OEA has little room to improve. At N=64, the expected unique expert count is around 512, but capped at 256 — meaning significant overlap is possible, making this the sweet spot for OEA. At N=256 and N=1024, expert saturation approaches 100% (251/256 experts active), so OEA's benefit should diminish. The assistant's earlier analysis in [msg 1125] had correctly predicted this: "The sweet spot for OEA should be at medium concurrency (32-128) where expert overlap is partial."

The benchmark uses sglang.bench_serving with random input/output lengths of 128 tokens, a request rate of 999 (effectively infinite, saturating the server), and extracts five key metrics: Output token throughput, Peak output token throughput, Total token throughput, Mean TPOT (time per output token), and Mean ITL (inter-token latency). These metrics collectively capture both throughput and latency characteristics.

The Results: A Story in Two Halves

The message shows results for N=10 and N=64, with N=256 and N=1024 truncated (the output was cut off by the tool's display). At N=10, the OEA server achieves 38.40 tok/s output throughput with a Mean TPOT of 109.92ms — virtually identical to the baseline of 109.66ms measured earlier in [msg 1119]. This is expected: at batch size 10, OEA is skipped entirely (the _OEA_MIN_BATCH threshold), so the results should match baseline exactly.

At N=64, the output throughput is 218.36 tok/s with a peak of 433.00 tok/s. These numbers need comparison with baseline to be meaningful — which is precisely what the assistant plans to do next, as evidenced by the very next message ([msg 1148]) where the OEA server is stopped and the baseline server is started for a clean A/B comparison.

The Thinking Process Visible in the Message

What makes this message fascinating is what it reveals about the assistant's methodology. The assistant does not simply run one benchmark and declare victory or defeat. Instead, it carefully constructs a multi-point benchmark sweep, choosing concurrency levels that will illuminate the optimization's behavior across different operating regimes. The assistant also understands the importance of clean A/B comparisons — the OEA results are meaningless without corresponding baseline numbers at the same concurrency levels.

The assistant's thinking is also visible in what it doesn't do. It doesn't cherry-pick the most favorable concurrency level and report only that. It doesn't stop after the first benchmark to check results. It runs all four levels in a single command, ensuring consistent conditions. This systematic approach is characteristic of the entire optimization campaign documented in this session.

Input Knowledge Required

To fully understand this message, one needs to know several things. First, the GLM-5 model architecture: it's a Mixture-of-Experts model with 256 experts, top-8 routing per token, and grouped GEMM routing. Second, the hardware: 8× NVIDIA RTX PRO 6000 Blackwell GPUs (SM120 architecture) connected via NVLink, with the model deployed using tensor parallelism (TP8). Third, the OEA algorithm itself: it intercepts the expert routing step at decode time, identifies which experts are already activated by other tokens in the batch, and redirects tokens to use those experts instead of activating new ones. Fourth, the sglang serving framework and its benchmark tools.

Output Knowledge Created

This message produces concrete benchmark data at four concurrency levels for the OEA-optimized server. This data becomes the basis for the A/B comparison that follows in subsequent messages. The results feed into the comprehensive glm5findings.md document written in [msg 1135], which captures all discoveries, benchmarks, and lessons learned from the optimization campaign. The benchmark also validates (or refutes) the assistant's hypothesis about OEA's sweet spot at medium concurrency.

Assumptions and Their Validity

The assistant makes several assumptions in this message. It assumes that the OEA implementation is now correct after the two bug fixes — that the sigmoid scores and sorted topk_ids produce proper routing decisions. It assumes that the benchmark methodology (random inputs/outputs of 128 tokens, request rate 999) provides a representative evaluation. It assumes that the four chosen concurrency levels adequately probe the optimization's behavior. These assumptions are reasonable and grounded in the assistant's deep understanding of both the model architecture and the optimization technique.

The Broader Context

This message sits within a larger narrative of methodical optimization work. The session has already explored piecewise CUDA graphs (blocked by framework limitations), MSCCLPP allreduce (minimal gains), expert parallelism (crashed under load due to CUTLASS tile failures on SM120), and various server parameter tunings. Each optimization was implemented, benchmarked, documented, and either adopted or discarded based on real measurements. The OEA optimization represents one of the more promising avenues — a purely algorithmic change that requires no hardware or framework modifications.

The message also exemplifies the assistant's disciplined approach to research: form a hypothesis, implement a solution, identify and fix bugs, benchmark systematically, compare against baseline, and document everything. This scientific methodology is what separates a genuine optimization campaign from random tweaking.