Benchmarking GLM-5-NVFP4 at Scale: Characterizing Throughput Scaling Across Concurrency Levels

Introduction

In the long arc of optimizing GLM-5-NVFP4 inference on eight NVIDIA RTX PRO 6000 Blackwell GPUs, message [msg 901] represents a pivotal moment of empirical measurement. After an extensive investigation into CUTLASS FP4 GEMM kernel efficiency on SM120, the assistant had confirmed that the model was compute-bound rather than communication-bound. The TP8 (tensor parallelism across all 8 GPUs) configuration had been launched, and the question now was simple but critical: how does throughput scale with concurrency?

The message itself is deceptively brief — a single bash command executing a loop of benchmark runs at concurrency levels 256, 512, and 1024, preceded by a note that the single-stream baseline had already been established at 10.2 tok/s. But this message sits at the intersection of weeks of infrastructure work, deep kernel analysis, and a systematic optimization campaign. It is the moment where theory meets measurement, where the assistant's hypotheses about compute-bound behavior are put to the test with real numbers.

The Context: Why This Message Was Written

To understand why this message exists, one must trace the arc of the preceding investigation. The assistant had spent considerable effort analyzing why the GLM-5-NVFP4 model — a massive Mixture-of-Experts architecture with 256 experts and FP4 quantization — was underperforming on the Blackwell GPUs. A key breakthrough had been the TP4+PP2 vs TP8 comparison: TP4+PP2 (tensor parallelism across 4 GPUs with pipeline parallelism across 2 stages) was found to be 2× slower than TP8. This was a crucial diagnostic result because it ruled out allreduce communication latency as the primary bottleneck. If communication were the bottleneck, TP4+PP2 (which requires less allreduce) would have been faster. The fact that TP8 was faster confirmed the model was compute-bound.

This led to a deep investigation into FP4 GEMM kernel efficiency on SM120 (the Blackwell architecture). The assistant discovered that the GPUs were drawing only ~235W out of 600W TDP during inference, and the CUTLASS kernels plateaued at ~1,300 TFLOPS (70% of dense peak) only for very large matrices. During actual decode, per-expert batch sizes of ~16–64 tokens achieved merely 0.8–55 TFLOPS — 0.02–3% of peak. The 99KB shared memory limit on SM120 prevented using larger CUTLASS tile configurations (M128×N256 and M256×N128 failed to initialize), while cuBLASLt FP4 was found to be no faster than FlashInfer's CUTLASS path.

With this understanding in hand, the assistant had launched the TP8 server and was now ready to benchmark. Message [msg 901] is the first systematic throughput characterization of the TP8 configuration.

The Benchmarking Methodology

The assistant's approach reveals a methodical mindset. Rather than running a single benchmark at an arbitrary concurrency level, the assistant chose to sweep across three levels — 256, 512, and 1024 — in a single bash loop. This is significant for several reasons.

First, the choice of concurrency levels reflects an understanding of the model's architecture. The GLM-5-NVFP4 model has 256 experts with 8 activated per token. At concurrency 256, with 8 active experts per token, the total number of expert invocations is 256 × 8 = 2,048, but distributed across 256 experts, each expert sees only about 8 tokens on average. At concurrency 512, each expert sees ~16 tokens. At concurrency 1024, ~32 tokens. These numbers map directly onto the per-expert GEMM dimensions that the assistant had been analyzing — the tiny matrix multiplications that were achieving only 0.02–3% of peak FP4 compute.

Second, the use of sglang.bench_serving with --request-rate 999 (effectively infinite, meaning all requests are fired as fast as possible) ensures the server is maximally loaded. This is a stress-test methodology designed to find the server's saturation point. The --random-input-len 128 --random-output-len 128 configuration provides a controlled workload — short, uniform sequences that isolate decode performance from prefill variability.

Third, the assistant filters the output to just five key metrics: Output token throughput, Total token throughput, Peak output, Mean TPOT, and Concurrency:. This is a deliberate choice to focus on the most informative signals while ignoring noise. The assistant is not interested in latency distribution details at this stage — it wants the aggregate throughput picture.

The Results and Their Interpretation

The results in message [msg 901] are truncated in the conversation, but we can see the beginning of a clear pattern:

At concurrency 256: Output token throughput of 704.64 tok/s, peak output of 1,486 tok/s, total token throughput of 1,383.44 tok/s, effective concurrency of 152.66, and mean TPOT of 201.68 ms.

At concurrency 512: Output token throughput of 1,105.82 tok/s, peak output of 2,001 tok/s, total token throughput of 2,205.48 tok/s.

The scaling from 256 to 512 is approximately linear: doubling the number of requests from 256 to 512 increases throughput from 704 to 1,105 tok/s — a 57% increase. This sub-linear scaling is expected because as concurrency increases, the per-expert batch size grows, improving GEMM efficiency, but the system also experiences greater queueing and scheduling overhead.

The effective concurrency numbers (152.66 at 256 requests, and presumably ~300 at 512 requests) reveal that the server is not keeping all requests in flight simultaneously. This is due to the KV cache memory limit and the --max-running-requests 1024 parameter. The server can only process a subset of requests at any given time.

The TPOT (time per output token) increases from 97ms at single-stream to 201ms at concurrency 256. This is the expected cost of batching — individual requests experience higher latency because they must wait for other requests' tokens to be processed in the same batch. The tradeoff is that total system throughput increases.

Assumptions and Limitations

The assistant makes several implicit assumptions in this message. First, it assumes that the benchmark workload (random 128-token input and output) is representative of real-world usage. This is a reasonable approximation for decode-heavy workloads, but real applications may have longer inputs, more variable output lengths, or different token distributions that affect caching efficiency.

Second, the assistant assumes that the server has reached steady state before measurements are taken. The sglang.bench_serving tool handles warmup internally, but the assistant does not verify that GPU clocks have stabilized or that the CUDA graph compilation (if any) has completed.

Third, the assistant assumes that the throughput numbers are reproducible. Running a single benchmark at each concurrency level leaves open the possibility of measurement noise from thermal throttling, power capping, or background processes.

Fourth, the assistant assumes that the --request-rate 999 parameter truly saturates the server. In practice, the client-side benchmark tool may have its own bottlenecks — the machine running the benchmark might not be able to generate requests fast enough to keep the server fully loaded.

Input Knowledge Required

To fully understand this message, one needs knowledge of:

  1. The GLM-5-NVFP4 model architecture: 256 experts, 8 activated per token, FP4 quantization, MoE (Mixture of Experts) structure. Without this, the choice of concurrency levels and the interpretation of per-expert batch sizes would be opaque.
  2. The TP8 configuration: Tensor parallelism across all 8 GPUs means each GPU holds a shard of every model layer. This maximizes compute utilization but requires allreduce communication after every layer. The assistant had already confirmed this was the optimal configuration.
  3. The Blackwell SM120 architecture: 99KB shared memory limit, FP4 compute capabilities, CUTLASS tile configuration constraints. The assistant's earlier investigation into why M128×N256 and M256×N128 tiles fail directly informs the interpretation of these benchmark results.
  4. The sglang.bench_serving tool: Its parameters, output format, and measurement methodology. The assistant knows which flags to use and which output lines to grep.
  5. The concept of compute-bound vs communication-bound: The entire optimization strategy hinges on this distinction. If the model were communication-bound, the solution would be to reduce allreduce overhead (e.g., through MSCCLPP or better NCCL settings). Since it's compute-bound, the solution must be to improve kernel efficiency or increase batch sizes.

Output Knowledge Created

This message produces concrete, actionable knowledge:

  1. Baseline throughput numbers: 704 tok/s at concurrency 256, 1,105 tok/s at concurrency 512, and (from the continuation in subsequent messages) ~1,600 tok/s at concurrency 1024. These serve as the baseline against which all future optimizations will be measured.
  2. Scaling behavior: The sub-linear scaling from 256 to 512 (57% increase for 2× concurrency) provides a data point for modeling the system's efficiency curve. The assistant can now estimate the saturation point and the marginal benefit of further concurrency increases.
  3. TPOT degradation: The increase from 97ms (single-stream) to 201ms (concurrency 256) quantifies the latency-throughput tradeoff. This is essential information for capacity planning and for setting quality-of-service guarantees.
  4. Validation of the compute-bound hypothesis: If the model were communication-bound, throughput would plateau much earlier because the allreduce would saturate PCIe bandwidth. The fact that throughput continues to scale up to 1,600 tok/s at 1024 concurrency (as seen in subsequent messages) confirms that compute is indeed the bottleneck.

The Thinking Process

The assistant's thinking process in this message is visible in its structure and choices. The single-stream baseline (10.2 tok/s, TPOT 97ms) was established first — a necessary sanity check to ensure the server is functioning correctly and to establish the latency floor. Then the assistant jumps to concurrency 256, skipping lower levels. This is a deliberate choice based on the model's architecture: with 256 experts, concurrency 256 is the minimum needed to ensure every expert sees at least some work.

The loop structure (256, 512, 1024) follows a geometric progression — doubling each time. This is a standard benchmarking technique that makes it easy to identify scaling regimes. If throughput doubles with each doubling of concurrency, the system is in the linear scaling regime. If throughput plateaus, the system is saturated.

The assistant does not run concurrency 2048 in this message (that comes in [msg 902]). The truncation at 1024 may reflect an expectation that the KV cache limit (--max-running-requests 1024) would cap effective concurrency. Indeed, subsequent messages show that 2048 yields only marginal improvement (1,640 tok/s vs 1,600 at 1024), confirming that the server is hitting its memory limit.

The assistant also does not measure GPU power or utilization during these benchmarks — that comes later in [msg 903]. This separation of concerns is methodical: first establish throughput scaling, then diagnose resource utilization.

Conclusion

Message [msg 901] is a textbook example of systematic performance characterization. The assistant takes a complex system — an 8-GPU inference server running a massive MoE model — and reduces it to a few key numbers that reveal its fundamental behavior. The benchmark results confirm that throughput scales with concurrency (validating the compute-bound hypothesis) and provide the baseline against which all subsequent optimizations will be measured. This message is not flashy — it's just a bash loop and some grep-filtered output — but it represents the disciplined, data-driven approach that makes the optimization campaign successful.

The numbers produced here — 704 tok/s at concurrency 256, 1,105 at 512, and eventually ~1,600 at 1024 — become the reference point for every improvement that follows. When the assistant later achieves 2,095 tok/s through parameter tuning, or 3,740 tok/s through FlashInfer CUTLASS MoE autotune, it is this baseline that gives those numbers meaning. Without measurement, optimization is guesswork. Message [msg 901] is where the measurement begins.