The Pivot Point: Benchmarking Opportunistic Expert Activation at Medium Concurrency

In the long and methodical optimization campaign for the GLM-5-NVFP4 model on 8× NVIDIA RTX PRO 6000 Blackwell (SM120) GPUs, message [msg 1132] stands as a quiet but pivotal moment. It is the instant where two parallel threads of work — a bug fix and a theoretical insight — converge into a single, deliberate action. The assistant, having just corrected a subtle error in its Opportunistic Expert Activation (OEA) implementation and restarted the inference server, launches a benchmark at concurrency 64 and simultaneously begins composing the comprehensive glm5findings.md document. This message is not flashy; it contains no breakthrough result, no dramatic chart. But it encapsulates the entire philosophy of the optimization effort: hypothesis-driven, evidence-based, and relentlessly methodical.

The Message Itself

The message is deceptively simple:

Good, server is ready. Let me kick off the medium-concurrency benchmark in the background while I write the findings doc: ``bash ssh root@10.1.230.174 'source /root/ml-env/bin/activate && echo "=== OEA k0=5, Concurrency ~64 ===" && python3 -m sglang.bench_serving --backend sglang --host 127.0.0.1 --port 8000 --model glm-5 --tokenizer lukealonso/GLM-5-NVFP4 --dataset-name random --random-input-len 128 --random-output-len 128 --num-prompts 64 --request-rate 999 2>&1 | grep -E "throughput|TPOT|ITL|TTFT"' & ``

The assistant confirms the server is ready (the grep -c "ready to roll" from the previous message returned 1), then launches a benchmark at concurrency 64 — the hypothesized "sweet spot" for OEA — as a background process, freeing itself to simultaneously write the findings document. The & at the end of the command is a small but telling detail: it reveals an engineer who values time efficiency, who understands that model loading takes minutes and benchmarks take seconds, and who refuses to sit idle.

Why This Message Matters: The Reasoning and Motivation

To understand why this message was written, we must trace the chain of reasoning that led to it.

The story begins with the OEA implementation itself. Opportunistic Expert Activation, inspired by the paper "Onescu, Wu, Chung, Wu, Gopal, Wang, Tri Dao, Athiwaratkun" (arXiv:2511.02237), is a decode-time routing optimization. The core idea is elegant: during autoregressive decoding, when multiple tokens in a batch are routed to the same expert, OEA opportunistically "piggybacks" additional tokens onto already-loaded experts rather than activating new ones. This reduces the number of unique experts loaded per batch, which in turn reduces memory traffic and improves throughput — at the cost of slightly suboptimal routing decisions.

The assistant had implemented OEA in the SGLang server's MoE routing code (topk.py), controlled by the environment variable SGLANG_OEA_K0. The initial implementation used router_logits.float() directly as scores for weight gathering. But as the assistant realized in <msg id=1123-1124>, this was incorrect: the raw router logits are pre-sigmoid, while the actual routing weights used by the model are sigmoid probabilities. Because sigmoid is a monotonic function, the ranking of experts is preserved either way — but the magnitudes of the gathered weights would be wrong, potentially distorting the output distribution.

This bug fix — changing scores = router_logits.float() to scores = router_logits.float().sigmoid() — required a server restart. The assistant killed the first OEA server and launched a second instance (the "OEA server v2"), then waited for it to load the 8-shard model across all GPUs. Message [msg 1132] is the moment that wait ends.

But there is a deeper motivation. The initial OEA benchmarks at high concurrency (1024) had shown only a modest 5.7% improvement in output throughput (1,607 vs 1,520 tok/s), though the peak output rate jumped an impressive 25% (4,012 vs 3,210). The assistant's analysis in [msg 1125] was clear-eyed: "at 1024 concurrency, expert saturation is already nearly 100% (1024 × 8/256 ≈ 32 tokens/expert, ~251/256 experts active). OEA can't reduce unique experts when nearly all are already needed." This is the key insight. OEA's benefit comes from reducing the number of unique experts loaded. If all experts are already active, there is nothing to reduce.

The sweet spot, the assistant reasoned, should be at medium concurrency (32-128) where expert overlap is partial — enough tokens per batch that some experts serve multiple tokens, but not so many that all experts are saturated. Concurrency 64 sits squarely in this hypothesized optimal range. Message [msg 1132] is the test of that hypothesis.

How Decisions Were Made

The decision process visible in this message and its immediate context reveals a sophisticated experimental methodology:

1. Hypothesis formation: Based on the theoretical understanding of OEA's mechanism (reducing unique expert count) and the empirical observation of near-total expert saturation at high concurrency, the assistant predicted that medium concurrency would yield the largest relative improvement.
2. Controlled comparison: The benchmark uses identical parameters to the baseline runs — same input/output lengths (128/128), same model, same server configuration (TP8, flashinfer_cutlass MoE backend, 16 continuous decode steps). The only variable is the OEA environment variable.
3. Background execution: By launching the benchmark with &, the assistant avoids blocking on the ~30-second benchmark run and can simultaneously write the findings document. This is not just convenience — it reflects a workflow where documentation is treated as a parallel, not sequential, activity.
4. Targeted metric extraction: The grep -E "throughput|TPOT|ITL|TTFT" pipeline extracts only the key performance indicators, filtering out the verbose benchmark configuration output. This shows a mature understanding of which metrics matter.

Assumptions Embedded in This Message

Every experiment rests on assumptions, and this message is no exception:

• That concurrency 64 is indeed the sweet spot: This is the central hypothesis, but it is an educated guess based on the expert count (256) and the average tokens-per-expert calculation. The actual optimal concurrency could be different.
• That the sigmoid fix matters: The assistant fixed the raw-logits bug but acknowledged that "the ranking is the same either way (sigmoid is monotonic)." The fix ensures correct weight magnitudes, but whether this materially affects throughput or quality is untested.
• That random input/output distribution is representative: The benchmark uses random data with 128-token input and 128-token output. Real-world traffic patterns (conversational, code generation, etc.) might have different routing distributions that affect OEA's effectiveness.
• That the server is fully stable: The OEA server v2 had just started. The assistant checked for the "ready to roll" message, but subtle issues like CUDA graph compilation or kernel warmup might not manifest immediately.

Mistakes and Incorrect Assumptions

The most significant mistake visible in the surrounding context is the original OEA implementation using raw logits instead of sigmoid scores ([msg 1123]). While the assistant correctly identified and fixed this, it's worth noting that the initial benchmarks at concurrency 256 and 1024 were run with the buggy version. Those results — the 5.7% improvement at 1024 — may have been affected by incorrect weight magnitudes. The message [msg 1132] launches the first clean benchmark with the corrected implementation.

Another subtle issue: the assistant's earlier analysis at concurrency 256 ([msg 1121]) showed anomalously high throughput numbers (706.50 tok/s output) that didn't match the baseline. The assistant noted this discrepancy but attributed it to different request rates rather than a potential measurement artifact. The concurrency 64 benchmark in [msg 1132] will provide a cleaner comparison point.

Input Knowledge Required

To fully understand this message, one needs:

• The OEA algorithm: Knowledge that OEA reduces unique expert count during decode by routing additional tokens to already-active experts, trading routing optimality for memory bandwidth efficiency.
• The GLM-5-NVFP4 model architecture: Specifically that it uses 256 experts with top-8 routing per token, and that the biased_grouped_topk_impl with num_expert_group=1 means all experts are in a single group.
• The hardware topology: 8× RTX PRO 6000 Blackwell GPUs with SM120 architecture, connected via NVLink with P2P access, but with known limitations on FlashInfer allreduce fusion for SM120.
• The SGLang server configuration: TP8 (tensor parallelism across 8 GPUs), flashinfer_cutlass MoE backend, 16 continuous decode steps, no CUDA graph, no radix cache.
• The benchmarking methodology: sglang.bench_serving with --request-rate 999 creates a burst-mode benchmark where all prompts are submitted simultaneously, measuring peak throughput under maximum concurrency.

Output Knowledge Created

This message produces several forms of knowledge:

1. The concurrency 64 benchmark result: This is the first clean measurement of OEA with the corrected sigmoid implementation at the hypothesized sweet spot. The result (which appears in subsequent messages) will either confirm or refute the medium-concurrency hypothesis.
2. The glm5findings.md document: The assistant begins writing this comprehensive document in parallel with the benchmark. This becomes the canonical record of all discoveries, benchmarks, and lessons learned across the entire optimization campaign.
3. A reproducible experimental protocol: The exact command and parameters serve as a template for future benchmarks. Any researcher can replicate this exact test.

The Thinking Process

The reasoning visible in and around this message is a model of systematic experimental science. The assistant:

1. Observes a phenomenon (OEA shows modest gain at high concurrency)
2. Forms a hypothesis (the gain should be larger at medium concurrency where expert overlap is partial)
3. Identifies a confound (the raw-logits bug)
4. Fixes the confound (adds .sigmoid())
5. Designs a clean test (concurrency 64 with identical parameters to baseline)
6. Executes the test efficiently (background process while writing documentation)
7. Documents everything (the findings doc captures the entire chain) This is not the work of someone chasing random optimizations. It is the work of an engineer who treats the inference server as an experimental apparatus, each benchmark as a controlled experiment, and each result as data to inform the next hypothesis.

Conclusion

Message [msg 1132] is the quiet hinge on which the OEA investigation turns. Before it, the assistant had a buggy implementation and inconclusive high-concurrency results. After it, the clean benchmark at concurrency 64 would reveal whether OEA is worth pursuing further or whether the team should pivot to other optimization strategies. The message itself is just a command — a benchmark launched in the background, a document begun. But the context, the reasoning, and the methodology behind it tell the story of a deeply systematic optimization campaign, one where every idea is tested, every bug is fixed, and every hypothesis is put to the fire of real measurement.