The First Measurement: Testing Opportunistic Expert Activation on GLM-5-NVFP4

In the relentless pursuit of inference throughput on an 8-GPU Blackwell cluster running the massive GLM-5-NVFP4 mixture-of-experts model, every optimization idea must eventually face the cold reality of a benchmark. Message [msg 1118] captures one such moment: the very first test of Opportunistic Expert Activation (OEA), a decode-time routing optimization that the assistant had just implemented and deployed. The message is brief—a single benchmark invocation and its results—but it sits at a critical juncture in a months-long optimization campaign, representing the transition from implementation to empirical validation.

The Context: A Methodical Optimization Campaign

To understand why this message exists, one must appreciate the broader arc of the session. The assistant had been systematically working through a prioritized list of optimization strategies for the GLM-5-NVFP4 model on 8× NVIDIA RTX PRO 6000 Blackwell GPUs. Earlier segments had established the core bottleneck: small per-expert GEMM operations on SM120 compute architecture, where the FP4 kernel efficiency was limiting throughput. The assistant had already tested and documented multiple approaches—Piecewise CUDA graphs (blocked by framework limitations), MSCCLPP allreduce (minimal gains), Expert Parallelism EP8 (crashed under load), and others—each time writing up findings in glb5improvement-xx.md documents.

The immediate precursor to this message was a major update: the assistant had updated sglang to its latest commit, which alone yielded a 2× throughput improvement at 256 concurrency compared to earlier baselines. Then came the implementation of OEA, inspired by a paper (arXiv:2511.02237 by Oncescu, Wu, Chung, et al.). The idea was elegant: during decode, instead of loading a completely new set of experts for each token in the batch, opportunistically redirect tokens to experts that are already loaded. If a batch of tokens collectively uses 20 unique experts, but many of those experts are only used by a single token, OEA attempts to "piggyback" low-probability expert selections onto experts that are already being loaded for other tokens in the same batch. This reduces the total number of unique experts that must be loaded per decode step, which in turn reduces memory bandwidth pressure—the core bottleneck on Blackwell's SM120 architecture.

The assistant had carefully implemented OEA as a patch to sglang's topk.py file ([msg 1106]), adding a new function opportunistic_expert_activation() gated by the SGLANG_OEA_K0 environment variable. The implementation was non-trivial: it had to handle the unsorted top-k output from the router, use proper sigmoid scores for weight gathering, and renormalize the resulting weights. The patch was applied to the running server, which was then restarted with OEA enabled (SGLANG_OEA_K0=5, meaning keep the top-5 baseline experts per token and try to piggyback the remaining 3 slots onto already-loaded experts).

The Message Itself: A Sanity Check

Message [msg 1118] begins with the assistant confirming the server is ready, then immediately running a "quick sanity check" benchmark:

Server is ready. Now let me run the benchmark suite — first a quick sanity check, then the full comparison

The benchmark uses sglang.bench_serving with 10 prompts, random input/output lengths of 128 tokens, and an effectively infinite request rate (999). This is a low-concurrency test—with only 10 prompts in flight, the batch size during decode will be small, typically around 8-10 tokens.

The results show:

What This Message Reveals About the Assistant's Thinking

The most striking aspect of this message is what it reveals about the assistant's experimental methodology. Several design choices are visible:

First, the assistant runs a sanity check before the full benchmark suite. This is a deliberate choice to validate that the server is functioning correctly and that OEA hasn't broken anything. The low concurrency (10 prompts) is intentional: it's a quick test that takes only seconds to complete, and it provides immediate confirmation that the model is serving requests correctly. If OEA had introduced a bug—crashed the server, produced garbage outputs, or dramatically changed latency—this test would catch it before the assistant invests time in a comprehensive benchmark run.

Second, the assistant explicitly frames this as "first a quick sanity check, then the full comparison." This reveals a staged experimental protocol. The assistant is not just running benchmarks randomly; it has a plan: validate, then measure, then compare. The "full comparison" would involve running the same benchmark at multiple concurrency levels (1, 10, 256, 1024) and comparing against the baseline numbers already collected.

Third, the assistant chose concurrency ~10 for the sanity check. This is not an accident. At very low concurrency (1-2 tokens per batch), OEA is expected to have zero effect because there's no opportunity for piggybacking—the _OEA_MIN_BATCH variable is set to 2, meaning OEA is skipped entirely for batches smaller than 2 tokens. At very high concurrency (256+ tokens per batch), the baseline already uses most or all of the 256 experts, so there's little room for OEA to reduce unique expert count. The sweet spot for OEA is medium concurrency (around 10-64 tokens per batch), where there's significant expert overlap potential. By testing at concurrency ~10, the assistant is probing the lower end of this range.

Assumptions and Their Validity

The assistant makes several assumptions in this message:

Assumption 1: The OEA implementation is correct. The patch was applied and the server started successfully, logging "OEA enabled: k0=5, min_batch=2." But correctness in the sense of "doesn't crash" is different from correctness in the sense of "produces valid outputs." The assistant implicitly assumes that if the server serves requests without errors, the outputs are valid. This is a reasonable assumption for a sanity check, but the assistant later runs more thorough validation.

Assumption 2: Low-concurrency results are a valid sanity check. This is correct: if OEA had broken the model's routing logic, the outputs would likely be garbage and the latency metrics would show anomalies. The fact that the numbers are nearly identical to baseline is actually good news—it means OEA is not introducing overhead at low concurrency (where it's supposed to be skipped or have minimal effect).

Assumption 3: The benchmark is reproducible. The assistant compares against a baseline run from minutes earlier, assuming that the system state is stable enough for comparison. Given that the same hardware, same model, same server configuration (except OEA), and same benchmark parameters are used, this is reasonable.

Assumption 4: The tail of the output (TPOT) is not needed for the sanity check. The assistant truncated the output with tail -20, which cuts off the TPOT and throughput lines. This is a deliberate choice—for the sanity check, the E2E latency and TTFT are sufficient to confirm the server is working. The detailed throughput numbers will come in the full comparison.

Input Knowledge Required

To fully understand this message, one needs:

  1. Knowledge of the GLM-5-NVFP4 model architecture: It's a mixture-of-experts model with 256 experts, using FP4 quantization via NVIDIA's modelopt. The routing mechanism selects top-8 experts per token from 256.
  2. Knowledge of the OEA algorithm: The paper's idea of reducing unique loaded experts by piggybacking on already-loaded experts during decode. The implementation uses a two-phase approach: first select top-k0 baseline experts, then fill remaining slots with experts that are already in the batch-wide baseline set.
  3. Knowledge of the benchmark tool: sglang.bench_serving is SGLang's built-in benchmarking utility. The parameters --num-prompts 10 --request-rate 999 create a burst of 10 requests sent as fast as possible, resulting in a small batch.
  4. Knowledge of the hardware: 8× RTX PRO 6000 Blackwell GPUs with NVLink/NVSwitch for P2P communication. The SM120 architecture has specific shared memory limits (100KB) that constrain GEMM tile sizes.
  5. Knowledge of the optimization campaign's history: Previous attempts (Piecewise CUDA graphs, MSCCLPP, EP8) and their outcomes, documented in glb5improvement-xx.md files.

Output Knowledge Created

This message creates several pieces of knowledge:

  1. OEA does not degrade performance at low concurrency. The E2E latency and TTFT are essentially identical to baseline, confirming that OEA's overhead (the extra sorting, masking, and weight gathering) is negligible for small batch sizes.
  2. The OEA server is stable. It handled 10 requests without errors, crashes, or anomalies. This is a necessary precondition for running the full benchmark suite.
  3. A baseline for comparison is established. The assistant now has a reference point: at concurrency ~10, OEA-enabled server = 5686 ms E2E, baseline = 5681 ms E2E. Any significant deviation in the full benchmark would be attributable to OEA's effect at higher concurrency.
  4. The experimental protocol is validated. The assistant's staged approach (sanity check → full comparison) is working correctly. The server responds to requests, the benchmark tool produces output, and the metrics are interpretable.

The Deeper Significance

This message, for all its brevity, embodies a philosophy of empirical optimization that pervades the entire session. Every idea is implemented, tested, measured, and either adopted or discarded based on real data. The assistant does not assume OEA will work—it builds it, deploys it, and measures it. The sanity check is the first step in a rigorous evaluation pipeline.

The message also reveals the assistant's patience and systematic approach. After waiting 60 seconds for the server to load (monitored with a polling loop in [msg 1117]), the assistant immediately runs the benchmark rather than, say, manually inspecting the model's outputs or running a custom validation script. The benchmark is the ultimate arbiter: if throughput improves, OEA stays; if not, it goes.

What follows this message (in [msg 1119] and beyond) is the full benchmark suite, which will reveal that OEA's average throughput improvement is near zero on random data—a disappointing but honest result that the assistant will document faithfully in glm5findings.md. The sanity check in this message, showing identical performance to baseline, is the first hint of that outcome. The assistant doesn't know it yet, but the numbers are already telling a story: at low concurrency, OEA does nothing (as expected), and at higher concurrency, it will also do nothing (unexpected, but confirmed by measurement).

This is the essence of rigorous engineering: form a hypothesis, implement it, test it, and accept the results—even when they're negative. Message [msg 1118] is the moment of testing, the point where speculation meets measurement, and the first data point in the verdict on Opportunistic Expert Activation.