Segment 9
In this sub-session, the assistant systematically executed the next wave of optimization work: updating sglang to the latest commit (which alone gave a 2x throughput improvement at 256 concurrency vs earlier baselines), implementing Opportunistic Expert Activation (OEA) as a decode-time routing optimization gated by an environment variable (clean A/B benchmarks showed near-zero average improvement on random data, confirming OEA's benefit depends on non-uniform expert routing), retrying Expert Parallelism (EP8) with a memory-safe config (loaded successfully but crashed during warmup due to CUTLASS 128×256×128 tile exceeding SM120's 100KB shared memory limit, though the server ultimately recovered and ran), benchmarking concurrency 1 and 2 (10.36 tok/s single-stream, 19.29 tok/s dual-stream with excellent linear scaling), writing the comprehensive 500+ line glm5findings.md document covering all discoveries and lessons learned, and beginning to compute the theoretical maximum single-stream performance for this exact model/hardware combination by gathering model config parameters to establish HBM bandwidth-limited and PCIe allreduce-limited throughput ceilings.
The Methodical Campaign: From SGLang Update to Theoretical Ceiling in the GLM-5 Inference Project