The Moment of Execution: Benchmarking the Tuned GLM-5-NVFP4 Server

In the long arc of optimizing inference for the GLM-5-NVFP4 model on eight RTX PRO 6000 Blackwell GPUs, message [msg 936] represents a critical inflection point — the moment when theory meets practice, when hours of investigation into kernel internals, shared memory constraints, and alternative computation paths crystallize into a single, decisive experiment. The message is deceptively simple: a confirmation that the server is configured with two key parameters, followed by a benchmark command that sweeps across four concurrency levels. But beneath this surface lies the culmination of a deep reasoning chain that traversed CUTLASS tile geometry, SM120 microarchitecture limits, cuBLASLt FP4 performance characteristics, and the fundamental economics of Mixture-of-Experts (MoE) inference.

The Context: A Long Road of Dead Ends and Discoveries

To understand why [msg 936] was written, one must appreciate the journey that preceded it. The assistant had been engaged in an intensive optimization campaign for the GLM-5-NVFP4 model — a 256-expert MoE transformer running in FP4 quantization on NVIDIA's newest Blackwell architecture (SM120). Earlier in segment 7, the assistant had confirmed through TP4+PP2 benchmarking that the model was compute-bound, not communication-bound, ruling out allreduce latency as the primary bottleneck ([chunk 7.0]). This was a crucial insight: if the model were communication-bound, the optimization strategy would focus on reducing cross-GPU traffic; being compute-bound meant the path to higher throughput lay in better utilizing the GPU's arithmetic units.

A deep investigation into FP4 GEMM kernel efficiency on SM120 had revealed a sobering picture. During actual decode operations, per-expert batch sizes of roughly 16–64 tokens achieved merely 0.8–55 TFLOPS — a tiny fraction (0.02–3%) of the GPU's theoretical peak. The CUTLASS kernels powering these operations plateaued at approximately 1,300 TFLOPS (70% of dense peak) only for very large matrix dimensions, far beyond what the MoE routing produced in practice. The GPUs themselves were drawing only ~235W out of a 600W TDP during inference, a stark indicator of severe underutilization.

The assistant had also explored and ruled out several optimization paths. The 99KB shared memory limit on SM120 prevented using larger CUTLASS tile configurations (M128×N256 and M256×N128 failed to initialize during autotune), a dead end documented across messages [msg 914] through [msg 923]. cuBLASLt FP4 integration was investigated via a research subagent ([msg 927]) and found to be no faster than FlashInfer's existing CUTLASS path — in fact, it was 5–8% slower for large matrix sizes and, critically, could not be used for grouped GEMM operations at all, making it irrelevant for the MoE layers that dominated computation.

The Hypothesis: Batch Size as the Master Lever

With the low-level kernel paths exhausted, the assistant pivoted to a higher-level insight. The GLM-5-NVFP4 model uses 256 experts with 8 active per token. At 1024 concurrent requests — the previous configuration — each expert received on average only 32 tokens per decode step. The FP4 GEMM kernels, designed for throughput at scale, were starving for work.

The lever was clear: increase the number of concurrent tokens in flight to push per-expert batch sizes higher. Two SGLang server parameters controlled this: --max-running-requests (the maximum number of concurrent requests the server would accept) and --num-continuous-decode-steps (how many decode steps to batch together before re-scheduling). The assistant raised the former from 1024 to 2048 and the latter from 4 to 8, effectively doubling the potential token pool per scheduling event.

Message [msg 936] opens with the confirmation that these parameters are in place: "Confirmed: max_running_requests=2048, num_continuous_decode_steps=8." This brief line carries significant weight — it signals that the server has been successfully restarted with the new configuration after a series of operational hiccups. Messages [msg 930] through [msg 933] show the assistant struggling with a failed launch (the shell script wasn't written properly due to heredoc issues), checking process status, finding the server not running, recreating the script, and finally launching successfully. The confirmation in [msg 936] is the all-clear after this troubleshooting.

The Benchmark: A Systematic Sweep

The assistant then executes a benchmark sweep across four concurrency levels — 256, 512, 1024, and 2048 — using SGLang's built-in bench_serving tool. The choice of a geometric progression (doubling each step) is deliberate: it allows the assistant to observe how throughput scales with concurrency and identify the point of diminishing returns. Each benchmark uses 128 input tokens and 128 output tokens with random data, a standard workload that exercises both prefill and decode phases.

The command itself reveals several assumptions. The --request-rate 999 flag effectively disables rate limiting, allowing the benchmark to fire all requests as fast as possible — this measures the server's maximum sustainable throughput rather than its behavior under controlled load. The grep filter extracts only the key metrics: output token throughput, total token throughput, peak output, mean TPOT (time per output token), and achieved concurrency. This structured output format makes the results machine-readable for comparison.

The Results Begin to Appear

The message shows results for the first two concurrency levels. At 256 concurrency, the server achieves 705 output tok/s and 1,384 total tok/s with a mean TPOT of 202 ms. At 512 concurrency, these figures rise to 1,111 output tok/s and 2,216 total tok/s. These numbers are essentially identical to the previous configuration ([msg 937] would later confirm this: "Same" for both 256 and 512), which makes sense — at lower concurrency levels, the server isn't pushing enough tokens through to benefit from the increased batch capacity.

The message is truncated at the 1024 and 2048 results, but the subsequent message ([msg 937]) reveals the payoff: at 2048 concurrency, output throughput jumps from 1,640 to 2,095 tok/s (a 28% improvement), and total throughput from 3,249 to 4,151 tok/s. The peak output rate hits an impressive 7,216 tok/s. This validates the assistant's hypothesis: with ~2048 concurrent tokens, each expert receives roughly 64 tokens on average — double the previous figure — finally pushing the FP4 GEMM kernels into a more efficient operating regime.

Assumptions Embedded in the Experiment

Several assumptions underpin this benchmark. The assistant assumes the KV cache capacity of 495,488 tokens (verified in [msg 924]) is sufficient for 2048 concurrent requests at 256 tokens each (128 input + 128 output), which requires approximately 524,288 tokens — slightly over the cache size. In practice, the achieved concurrency at 2048 requests was 2,073 ([msg 937]), suggesting the server dynamically throttled to fit within available memory. The assistant also assumes that the random dataset with fixed input/output lengths is representative of real workloads, though real traffic patterns would likely show more variance.

A subtle but important assumption is that the num-continuous-decode-steps 8 parameter interacts correctly with the disable-cuda-graph flag. CUDA graphs are disabled (--disable-cuda-graph), which means each decode step incurs kernel launch overhead — but this overhead is acceptable because the alternative (CUDA graph replay) would require fixed graph shapes that don't accommodate the variable batch sizes that continuous batching produces.

The Thinking Process: From Kernel Internals to System Tuning

What makes [msg 936] remarkable is not the benchmark itself but the reasoning chain that led to it. The assistant had to navigate a complex decision tree:

  1. Is the model compute-bound or communication-bound? → Benchmark TP4+PP2 vs TP8 → Compute-bound (TP8 is 2× faster)
  2. Can we fix the failing CUTLASS tiles? → Investigate SM120 99KB limit → Dead end (128×256 tile needs 91KB mainloop alone)
  3. Is cuBLASLt FP4 a better kernel path? → Research subagent → No better, and no grouped GEMM support
  4. What's the actual bottleneck during decode? → Measure per-expert batch sizes → 16–64 tokens, far too few
  5. How do we increase per-expert batch sizes? → Tune server parameters → max-running-requests and num-continuous-decode-steps Each of these steps required deep knowledge of different domains: GPU microarchitecture (SM120 shared memory limits, FP4 GEMM efficiency curves), CUDA kernel compilation (CUTLASS tile builder, StageCountAutoCarveout), MoE inference dynamics (expert routing, per-expert batch distribution), and SGLang server architecture (continuous batching, KV cache management). The assistant synthesized these disparate knowledge domains into a coherent optimization strategy, then executed it with a well-designed benchmark.

The Broader Significance

Message [msg 936] is a textbook example of the systems approach to ML inference optimization: when low-level kernel tuning hits fundamental hardware limits, the next lever is often at the system level — changing how work is batched and scheduled. The 28% throughput improvement achieved here is not the result of a single magic bullet but of systematically ruling out alternatives until the most impactful lever becomes clear. The message also illustrates the iterative nature of performance engineering: each experiment generates data that informs the next hypothesis, and even "negative" results (the CUTLASS tile dead end, the cuBLASLt non-improvement) are valuable for narrowing the search space.

The assistant's approach of documenting each optimization path as glb5improvement-xx.md files (mentioned in the chunk summary) reflects a mature engineering discipline: insights are captured for future reference, dead ends are recorded so they need not be re-explored, and the ranked optimization plan provides a roadmap for continued work. Message [msg 936] is the execution node in this plan — the point where analysis yields to experiment, and theory confronts reality.