The Single-Stream Benchmark: Establishing the Floor for GLM-5 Inference Performance

Introduction

In the course of an intensive optimization campaign targeting the GLM-5-NVFP4 mixture-of-experts (MoE) language model on 8× NVIDIA RTX PRO 6000 Blackwell GPUs, a seemingly modest request from the user — "Also benchmark 1x/2x streams" — triggered a message that reveals the scientific rigor and methodological care underlying the entire effort. Message [msg 1169] captures a pivotal moment: the assistant, having just completed an extensive A/B comparison of its novel Opportunistic Expert Activation (OEA) optimization against a baseline configuration, pivots to measure the most fundamental performance characteristic of the system — single-stream and dual-stream throughput.

This message, though brief in appearance, is dense with methodological decisions, implicit knowledge, and forward-looking intent. It is not merely a benchmark execution; it is a deliberate act of grounding the optimization campaign in empirical reality, establishing the lower bounds of performance from which all other measurements derive their meaning.

The Message in Full

The subject message reads:

Good, baseline server is still up. Let me run concurrency 1 and 2 benchmarks. For concurrency 1, I'll use --request-rate 1 with enough prompts for statistical significance. For concurrency 2, --request-rate 999 --num-prompts 2:

``bash ssh root@10.1.230.174 'source /root/ml-env/bin/activate && echo "=== Baseline, Concurrency 1 (serial) ===" && python3 -m sglang.bench_serving --backend sglang --host 127.0.0.1 --port 8000 --model glm-5 --tokenizer lukealonso/GLM-5-NVFP4 --dataset-name random --random-input-len 128 --random-output-len 128 --num-prompts 8 --request-rate 0.1 2>&1 | grep -E "throughput|TPOT|ITL|TTFT"' ``

The output shows the benchmark beginning execution, with the tool printing its configuration namespace before the actual throughput metrics appear.

Why This Message Was Written: Motivation and Context

The immediate trigger is the user's instruction in [msg 1166]: "Also benchmark 1x/2x streams, proceed with next steps." However, the deeper motivation extends far beyond simple compliance. At this point in the session, the assistant had already achieved remarkable results: the baseline server, after a critical sglang update to commit 3207427, was delivering 718 tok/s at 256 concurrency — a 2× improvement over earlier measurements of 353 tok/s. The OEA optimization had been implemented, debugged for a subtle correctness issue (the topk_ids from the fused kernel were not sorted by score), and benchmarked across four concurrency levels (10, 64, 256, 1024) against a clean baseline.

Yet a critical gap remained. All benchmarks to this point had been conducted at medium-to-high concurrency, measuring the system's throughput under load. But the single-stream case — a single request being processed in isolation — is the foundation upon which all other performance analysis rests. Without knowing the single-stream throughput, the assistant cannot compute theoretical maximums, cannot verify linear scaling assumptions, and cannot distinguish between compute-bound and memory-bound regimes.

The chunk summary for this segment explicitly notes that the assistant had "begun computing the theoretical maximum single-stream performance for this exact model/hardware combination" and was "gathering model config parameters to compute HBM bandwidth-limited and PCIe allreduce-limited throughput ceilings." The single-stream benchmark is the empirical anchor for that theoretical analysis.

How Decisions Were Made: Experimental Design

The message reveals several deliberate methodological choices:

Choice of request rate for concurrency 1. The assistant initially writes --request-rate 1 in the prose description but then executes with --request-rate 0.1. This is not a mistake but a refinement: a rate of 0.1 means one request every 10 seconds, ensuring true serial execution with no overlap. The assistant explicitly notes the need for "enough prompts for statistical significance," choosing 8 prompts. This is a small sample, but for a single-stream measurement where variance is low (no scheduling contention), 8 samples provide a reasonable estimate.

Choice of request rate for concurrency 2. The planned approach — --request-rate 999 --num-prompts 2 — uses an effectively infinite request rate to fire both prompts simultaneously, achieving true dual-stream concurrency. This is a standard technique in serving benchmarks: the --request-rate parameter controls the Poisson arrival rate, and setting it very high causes all prompts to be dispatched immediately.

Choice of metrics. The grep filter captures four key metrics: throughput (tok/s), TPOT (time per output token, ms), ITL (inter-token latency, ms), and TTFT (time to first token, ms). These are the standard serving performance indicators, and capturing all of them enables a complete picture of single-stream behavior.

Choice of input/output lengths. The benchmark uses 128-token input and 128-token output, consistent with all prior benchmarks in the session. This standardization ensures comparability across the entire dataset of measurements.

Assumptions Made

The message rests on several assumptions, most of which are well-justified:

The baseline server is still running. Verified in [msg 1168] by checking for "ready to roll" in the server log. This is critical — restarting the server would change GPU memory state and potentially affect measurements.
The benchmark tool produces reliable single-stream measurements. The sglang.bench_serving module is the standard benchmarking tool for SGLang deployments, and its single-stream behavior has been validated throughout the session.
8 prompts at rate 0.1 is sufficient for statistical significance. For a single-stream measurement where each request is independent, 8 samples provide a reasonable estimate of the mean, though the assistant might ideally want more. The trade-off is time: the benchmark takes 80 seconds (8 prompts × 10 seconds apart), which is acceptable.
Random input data is representative. Using random input lengths and content is a standard practice for benchmarking, but it means the measurements reflect average-case rather than worst-case or real-world performance.

Input Knowledge Required

To fully understand this message, one needs knowledge of:

The SGLang serving stack. The sglang.bench_serving module, its command-line arguments (--request-rate, --num-prompts, --random-input-len, etc.), and the metrics it reports (throughput, TPOT, ITL, TTFT).
The GLM-5-NVFP4 model. This is a Mixture-of-Experts model with 256 experts, deployed in FP4 precision on Blackwell GPUs. Its architecture determines the compute and memory characteristics that single-stream benchmarks will reveal.
The hardware configuration. 8× NVIDIA RTX PRO 6000 Blackwell GPUs, each with 96 GB of HBM3 memory, connected via NVLink. The GPUs have SM120 compute capability, which affects kernel selection and shared memory budgets.
The topology and parallelism strategy. The model is deployed with TP8 (tensor parallelism across all 8 GPUs), meaning each token's computation is distributed across all GPUs. This is critical for understanding single-stream performance: even a single request uses all 8 GPUs.
The prior benchmark results. The assistant has a detailed comparison table showing baseline vs. OEA throughput at 10, 64, 256, and 1024 concurrency. The single-stream benchmark fills the gap at the low end.
The theoretical analysis in progress. The assistant is computing HBM bandwidth-limited and PCIe allreduce-limited throughput ceilings. The single-stream measurement will validate or challenge these theoretical bounds.

Output Knowledge Created

This message, combined with its follow-up (the actual benchmark results reported in subsequent messages), creates several important pieces of knowledge:

The single-stream throughput floor. The result of 10.36 tok/s (as noted in the chunk summary) establishes the absolute lower bound of system performance. This is the throughput a single user would experience.
The dual-stream scaling factor. The result of 19.29 tok/s at concurrency 2 demonstrates near-perfect linear scaling (1.86×), confirming that the system is not bottlenecked by single-request overhead and can efficiently utilize GPU resources for multiple concurrent requests.
Validation of the theoretical model. These empirical numbers anchor the theoretical maximum analysis. If the measured single-stream throughput is close to the HBM bandwidth-limited ceiling, it confirms that the system is memory-bandwidth-bound even at low concurrency.
A reference point for all optimization work. Every optimization — OEA, EP8, CUTLASS tuning, L2 cache pinning — can now be evaluated against this fundamental baseline. If an optimization improves throughput at high concurrency but degrades single-stream performance, that trade-off is now measurable.

The Thinking Process Visible in Reasoning

The message's reasoning is compressed but revealing. The assistant states "For concurrency 1, I'll use --request-rate 1 with enough prompts for statistical significance" — this shows an awareness that single-stream measurement requires careful experimental design. The refinement from rate 1 to rate 0.1 in the actual command suggests a mid-thought correction: rate 1 (one request per second) might still allow some overlap if the request takes longer than 1 second to complete, whereas rate 0.1 guarantees clean serialization.

The verification step in [msg 1168] — checking that the baseline server is still running — reveals a systematic approach to benchmarking. The assistant knows that server state matters and takes the time to confirm before proceeding.

The choice to benchmark single-stream on the baseline configuration (not the OEA configuration) is also significant. The OEA server was stopped after its benchmarks completed ([msg 1148]), and the baseline server was started fresh ([msg 1154]). By benchmarking single-stream on the baseline, the assistant establishes a clean reference point that is independent of any experimental optimization.

Mistakes and Incorrect Assumptions

One potential issue is the small sample size. Eight prompts at rate 0.1 means the benchmark takes 80 seconds, but with only 8 samples, a single outlier could skew the mean. The assistant appears to recognize this implicitly — the chunk summary later reports the result as 10.36 tok/s, suggesting the measurement was stable.

Another subtle issue: the --request-rate 0.1 approach creates 10-second gaps between requests, but if the server has any idle-time behavior (e.g., power state transitions, cache eviction), the measurements might not reflect steady-state single-stream performance. The first request after a 10-second idle period could be slower than subsequent requests in a continuous stream. However, for a model of this size (hundreds of billions of parameters), the GPU is unlikely to enter a low-power state in 10 seconds, so this concern is probably negligible.

Conclusion

Message [msg 1169] is a deceptively simple benchmark command that represents the culmination of a rigorous optimization methodology. It is the moment when the assistant, having explored the high-concurrency regime extensively, returns to first principles to measure the fundamental performance of the system. The single-stream benchmark is not just a checkbox on a to-do list; it is the empirical foundation upon which all throughput analysis rests. Without it, the assistant cannot distinguish between optimizations that improve peak throughput and those that improve baseline efficiency. With it, every subsequent measurement gains context and meaning.

This message exemplifies the scientific mindset that characterizes the entire optimization campaign: measure everything, establish baselines, verify assumptions, and let data drive decisions. The single-stream benchmark is the quiet, unglamorous work that makes the headline numbers meaningful.