The 2-Stream Benchmark: Validating Linear Scaling on GLM-5-NVFP4

In the middle of an intensive optimization campaign for the GLM-5-NVFP4 model running on eight NVIDIA RTX PRO 6000 Blackwell GPUs, a brief but significant message appears at index 1172. The assistant reports: "2-stream: 19.29 tok/s output, 99.21ms TPOT. Nearly 2x the single-stream output (19.29 vs 10.36) with only a 4ms increase in TPOT — great scaling." This single sentence encapsulates the result of a carefully designed micro-benchmark, and the message serves as a pivot point between measurement and action. It closes one investigation and opens the next.

The Context: Why This Message Was Written

To understand why this message exists, we must trace back to the user's request in [msg 1166]: "Also benchmark 1x/2x streams, proceed with next steps." This was a straightforward instruction from the human operator, who wanted to understand the model's performance at the lowest concurrency levels — single-stream and dual-stream throughput. These numbers are foundational: they establish the irreducible latency floor (how fast one request can be served) and the scaling efficiency (how well the system handles a second concurrent request before parallelism overhead dominates).

The assistant had just completed an extensive A/B comparison of Opportunistic Expert Activation (OEA), a decode-time routing optimization adapted from a Tri Dao et al. paper. That comparison had shown OEA provided negligible average throughput improvement on random data (within ±0.3% of baseline), though it did improve peak throughput by 3.6–5.9% at high concurrency. With OEA ruled out as a meaningful win for this workload, the assistant was pivoting to the next priority: retrying Expert Parallelism (EP8) with a memory-safe configuration. But before that, the user wanted the low-concurrency baselines.

The Benchmarking Methodology

The assistant's approach to measuring single-stream and dual-stream throughput reveals careful experimental design. The baseline server was already running (confirmed in [msg 1168] by checking for "ready to roll" in the log). The assistant first attempted a naive approach for concurrency 1: using --request-rate 0.1 with 8 prompts, which serializes requests at 0.1 requests per second. This produced only 8.31 tok/s — a misleadingly low number because the requests barely overlapped, making the throughput measurement meaningless despite the TPOT (time per output token) of 98.47ms being valid.

The assistant recognized this flaw immediately in [msg 1170]: "Wait — this has very low request rate (0.1 req/s), so requests are mostly serial. The effective throughput is only 8.31 tok/s because requests barely overlap." This self-correction is a hallmark of rigorous benchmarking. The assistant then switched to using --max-concurrency 1 with --request-rate 999, which properly constrains the server to process at most one request at a time while keeping the request queue full. This yielded the true single-stream throughput: 10.36 tok/s output with a TPOT of 95.14ms.

For dual-stream, the assistant used --max-concurrency 2 with the same high request rate, producing 19.29 tok/s output with a TPOT of 99.21ms.

Interpreting the Results: Near-Perfect Linear Scaling

The key insight in the message is the comparison: 19.29 tok/s at concurrency 2 versus 10.36 tok/s at concurrency 1. This is a 1.86x speedup — remarkably close to the theoretical maximum of 2x for perfect linear scaling. The TPOT increased by only 4.07ms (from 95.14ms to 99.21ms), representing just 4.3% overhead from adding a second concurrent request. This is excellent scaling behavior.

This result is significant because it tells us something fundamental about the inference bottleneck. In large language model serving, throughput scaling with concurrency is limited by two factors: compute resource saturation and memory bandwidth contention. Near-perfect 2x scaling from 1 to 2 streams means the GPUs are not yet compute-bound at these low concurrency levels — there is ample headroom in both compute units and memory bandwidth to handle a second request with minimal interference. This confirms that the model's small per-expert GEMM operations (identified in earlier analysis as the core bottleneck on SM120) are not saturating the GPU at low batch sizes.

The assistant's phrasing — "great scaling" — is understated but accurate. This is genuinely impressive behavior for a model of this size (GLM-5-NVFP4 with 256 experts in its Mixture-of-Experts layers) running on Blackwell architecture.

Input Knowledge Required

To fully appreciate this message, one needs to understand several concepts:

TPOT (Time Per Output Token): The latency for generating a single token during the decode phase, measured in milliseconds. This is the key latency metric for interactive serving.
Throughput (tok/s): The number of tokens generated per second across all concurrent requests. This is the key throughput metric.
Concurrency scaling: How throughput changes as more requests are processed simultaneously. Perfect scaling means N concurrent requests produce N times the throughput of a single request.
The --max-concurrency flag: A benchmark parameter that limits how many requests the server processes simultaneously, distinct from --request-rate which controls how fast requests are submitted.
The baseline server configuration: TP8 (Tensor Parallelism across 8 GPUs) with CDS16 (chunked decode size 16), running the latest sglang commit 3207427. One also needs the broader context of the optimization campaign: that the assistant had been systematically testing ideas (Piecewise CUDA Graphs, MSCCLPP, Single Batch Overlap, OEA) and documenting them in glb5improvement-xx.md files and the comprehensive glm5findings.md document.

Output Knowledge Created

This message creates several pieces of actionable knowledge:

Single-stream latency baseline: 95.14ms TPOT, 10.36 tok/s. This is the fastest a single user can experience.
Dual-stream throughput baseline: 19.29 tok/s at 99.21ms TPOT. Two concurrent users can be served with only 4ms additional latency per token.
Scaling efficiency confirmed: 1.86x speedup from 1 to 2 streams, validating that the system is not bottlenecked at low concurrency.
Comparison to earlier measurements: The TPOT improved from ~107ms (earlier in the session) to 95.14ms, which the assistant attributes to the sglang update. This quantifies the benefit of keeping the serving stack up to date. These numbers become reference points for all future optimization work. Any optimization that improves single-stream latency or dual-stream throughput can be measured against these baselines.

The Thinking Process Visible in the Message

The message reveals a clear decision-making chain. The assistant has just received the benchmark output (the raw numbers from the bash command in [msg 1171]), and immediately performs the critical comparison: "Nearly 2x the single-stream output (19.29 vs 10.36) with only a 4ms increase in TPOT." This is not just reporting data — it's interpreting it, drawing the conclusion that scaling is excellent.

The phrase "great scaling" is a judgment call. The assistant could have focused on the fact that it's not exactly 2x (it's 1.86x), or that the TPOT increased by 4ms. Instead, it correctly identifies that for a production serving system, this level of efficiency at low concurrency is a strong positive signal.

The message then pivots immediately: "Now let me stop the baseline and test EP8 with the memory-safe config." This transition is governed by the todo list embedded in the message. The assistant updates the todo status: "Benchmark at concurrency 1 and 2" changes from "in_progress" to "completed", and "Retry EP8 with memory-safe config" moves from "pending" to "in_progress". This todo-driven workflow is characteristic of the assistant's systematic approach — each task is explicitly tracked, and the message serves as both a report and a state transition.

Assumptions and Potential Pitfalls

The assistant makes several assumptions in this message. First, it assumes that the benchmark results are statistically stable — that 16 prompts per configuration is sufficient for a reliable measurement. For single-stream and dual-stream benchmarks with fixed input/output lengths (128 tokens each), 16 prompts should provide reasonable stability, but there's always variance from GPU clock fluctuations, memory controller states, and PCIe contention.

Second, the assistant assumes that the --max-concurrency flag correctly constrains the server's internal batching. This depends on the benchmark tool's implementation and whether the server respects the concurrency limit at the scheduling level rather than just the HTTP connection level.

Third, the assistant implicitly assumes that the 95.14ms TPOT improvement from the earlier 107ms is entirely due to the sglang update, rather than other factors like GPU temperature, power state, or random variance. This is a reasonable attribution given the magnitude of the change (~11% improvement), but it's not proven.

The Broader Significance

This message, while brief, represents a crucial moment in the optimization campaign. The assistant has established that the system scales beautifully at low concurrency — the GPUs are not bottlenecked by memory bandwidth or compute at batch sizes 1–2. This means the optimization focus should remain on what happens at high concurrency (where throughput plateaus), and on the fundamental kernel efficiency of small per-expert GEMMs on SM120.

The message also demonstrates a disciplined experimental methodology: measure the simplest case first (single stream), then add one degree of complexity (dual stream), verify scaling, then move to the next hypothesis (EP8). This incremental approach, combined with clean A/B comparisons and explicit todo tracking, is what makes the overall campaign effective. Each message in the conversation is a small experiment with a clear question, a clean measurement, and an unambiguous conclusion — and message 1172 is a perfect example of this philosophy in action.