Benchmarking GLM-5-NVFP4 on 8 Blackwell GPUs: Finding the Throughput Ceiling

Introduction

In the middle of a complex deployment session for the GLM-5-NVFP4 model — a 744-billion parameter Mixture-of-Experts (MoE) language model running on eight NVIDIA RTX PRO 6000 Blackwell GPUs — a single assistant message marks a critical transition. Message 227 is the moment the assistant stops validating that the model works and starts systematically measuring how fast it can go. After resolving a crippling NaN crash during decode that had plagued multiple earlier attempts, the assistant now has a stable server configuration and is methodically pushing it to its limits.

The message reports the results of a benchmark with 64 concurrent requests at a moderate request rate, then immediately launches a saturated test designed to find the absolute maximum throughput the system can sustain. It is a textbook example of progressive benchmarking: establish a baseline, increase load, observe behavior, then push harder.

The Context: From NaN Crash to Working Deployment

To understand message 227, one must appreciate the journey that preceded it. The assistant had spent significant effort deploying GLM-5-NVFP4 on SM120 Blackwell GPUs, encountering persistent NaN (Not-a-Number) crashes during the decode phase. These crashes produced garbage output and forced server restarts. The root cause was traced to incompatible NSA (Native Sparse Attention) backends — the flashmla_kv and flashmla_sparse backends produced NaN values on SM120 hardware, while the trtllm backend worked correctly.

The fix was deployed in earlier messages (msg 215–218): launching the server with --nsa-decode-backend trtllm --nsa-prefill-backend trtllm. This produced coherent reasoning chains and correct answers, confirming that the model was finally operational. The assistant then shifted focus from "does it work?" to "how fast does it work?"

A series of progressively larger benchmarks followed. Message 224 tested 16 prompts at request rate 2, yielding 50.55 output tok/s. Message 225 tested 32 prompts at rate 8, achieving 144 output tok/s. Message 226 tested 64 prompts at rate 16, with server-side decode throughput peaking at ~208 tok/s. Each step increased the load while verifying that the server remained stable — no crashes, no NaN, no errors.

The Subject Message: Benchmark Results and the Saturation Push

Message 227 opens with the results of the 64-request test at request rate 16:

Good progress. With 64 concurrent requests at rate=16: - Output throughput: 196 tok/s (up from 144 at rate=2) - Total throughput: 451 tok/s - All 64 requests succeeded - Peak concurrency hit 64 (all requests in flight simultaneously)

These numbers represent a significant improvement over the previous test. Output throughput increased from 144 to 196 tok/s — a 36% gain — simply by increasing the request rate from 8 to 16. The total throughput of 451 tok/s includes both prefill and decode tokens, giving a more complete picture of the server's workload. Critically, all 64 requests succeeded, and peak concurrency reached 64, meaning the server was handling every request simultaneously without dropping any.

The assistant then immediately launches the next test:

Let me now try to max out the server with even higher concurrency, and also run a saturated test (request_rate=inf) to find the max throughput:

This is followed by a bash command executing sglang.bench_serving with the same parameters except --request-rate inf. The inf flag tells the benchmarking tool to send all requests as fast as possible, without rate limiting, effectively saturating the server to find its maximum sustainable throughput.

Why This Message Matters

Message 227 is significant for several reasons. First, it represents the first time in the session that the assistant has a fully working, stable deployment of GLM-5-NVFP4 on SM120 hardware. All prior work was either setup, debugging, or crash recovery. This message is the payoff.

Second, the assistant's methodology reveals a clear mental model of how to benchmark an inference server. Rather than jumping straight to maximum load, it progressively increases concurrency: 16 requests, then 32, then 64, each time increasing the request rate. This allows the assistant to observe how throughput scales with load and to detect any instability at each level before pushing further.

Third, the decision to use request_rate=inf is a deliberate choice to find the saturation point. At lower request rates, the server may have idle cycles between batches. The saturated test eliminates idle time entirely, revealing the true hardware bottleneck — whether it's GPU compute, memory bandwidth, PCIe transfers, or CPU scheduling.

Assumptions Embedded in the Message

The assistant makes several assumptions in this message, some explicit and some implicit.

The most important assumption is that the server configuration is stable enough to handle saturation. The previous test at rate 16 succeeded, but saturation is a qualitatively different stressor — it can expose race conditions, memory fragmentation, or queue overflow bugs that don't appear at lower loads. The assistant implicitly trusts that the trtllm NSA backend is robust under extreme conditions.

Another assumption is that the native SGLang /generate endpoint provides meaningful throughput numbers. As noted in earlier messages (msg 224), the native backend loses per-token timing information — Time-to-First-Token (TTFT), Time-per-Output-Token (TPOT), and Inter-Token Latency (ITL) are all reported as zero or negative. The assistant compensates by relying on aggregate throughput (total tok/s and output tok/s), but this means it cannot diagnose whether the bottleneck is in prefill or decode.

The assistant also assumes that benchmarking with random input/output lengths of 256 tokens is representative. Real-world usage may have different token distributions, longer contexts, or variable output lengths. The random dataset is a reasonable proxy for initial benchmarking but may not reflect production performance.

What the Assistant Got Right

The assistant's systematic approach is a strength. Each benchmark builds on the previous one, and the assistant carefully records results before moving to the next test. The progression from 16 to 32 to 64 requests, with corresponding increases in request rate, follows sound experimental design.

The decision to use the native SGLang backend instead of the OAI chat backend was also correct. Earlier attempts (msg 222–223) showed that the chat backend crashed on reasoning models because it expected content to be a string, but reasoning models return content: null with reasoning_content instead. The native backend handles this correctly.

The assistant also correctly interprets the metrics. It notes that "peak concurrency hit 64 (all requests in flight simultaneously)" — this is important because it confirms the server can handle the full workload without queueing or dropping requests.

Limitations and What the Assistant Didn't Yet Know

At the time of message 227, the assistant was not yet aware of several important factors that would later become central to the tuning effort. CUDA graphs were disabled (--disable-cuda-graph was in the server launch command), which likely limited throughput. The assistant would later (msg 229+) attempt to enable CUDA graphs and increase the memory fraction, discovering that CUDA graphs could be captured successfully with the trtllm backend.

The assistant also hadn't yet examined the PCIe topology or virtualization overhead. Later in the session (Chunk 0 of Segment 2), the assistant would discover that the system is running inside a Proxmox/KVM virtual machine with no direct GPU peer-to-peer support, forcing all cross-GPU transfers through host memory. This virtualization overhead would prove to be a significant bottleneck, with small all-reduce messages achieving only ~1 GB/s bandwidth.

These discoveries would come later. At the moment of message 227, the assistant is focused on establishing a clean baseline with the current configuration before making any changes.

Knowledge Flow: Input and Output

To understand message 227, the reader needs several pieces of input knowledge: familiarity with SGLang's benchmarking tool (sglang.bench_serving), understanding of the --request-rate parameter and the special inf value, awareness that the model is GLM-5-NVFP4 running on 8x RTX PRO 6000 GPUs with tensor parallelism, and knowledge of the earlier NaN crash and its resolution via trtllm NSA backends.

The message creates new knowledge: baseline throughput numbers for the current configuration (196 output tok/s, 451 total tok/s at rate 16), confirmation that the server handles full concurrency without crashing, and a clear next step (the saturated test that is launched within the same message). This knowledge feeds directly into the subsequent tuning work — the assistant will use these numbers as a reference point when evaluating the impact of configuration changes.

The Thinking Process

The assistant's reasoning is visible in the structure of the message. It first summarizes the results of the just-completed test, highlighting the key metrics and the fact that all requests succeeded. Then it explicitly states the next goal: "max out the server with even higher concurrency." The transition from "what we just learned" to "what we need to learn next" is seamless.

The choice of request_rate=inf is particularly telling. The assistant could have tried rate 32 or rate 64, but instead chose saturation. This suggests the assistant believes the server is not yet at its throughput ceiling at rate 16, and wants to find that ceiling directly rather than incrementally. It's a pragmatic decision — one saturated test can reveal the maximum throughput, while a series of incremental tests would take longer and might not converge cleanly.

The assistant also shows awareness of the benchmarking tool's limitations. Earlier messages (msg 223–224) discussed the crash with the OAI chat backend and the loss of per-token timing with the native backend. The assistant has adapted its methodology to work within these constraints, focusing on aggregate throughput rather than per-token metrics.

Conclusion

Message 227 is a pivotal moment in the GLM-5-NVFP4 deployment session. It marks the transition from debugging to performance optimization, from "does it work?" to "how fast can it go?" The assistant's systematic benchmarking methodology — progressively increasing concurrency, recording results at each step, and then pushing to saturation — provides a clear picture of the server's baseline performance. The throughput numbers (196 output tok/s, 451 total tok/s) establish a reference point against which all future tuning will be measured.

The message also demonstrates the assistant's ability to work within tool constraints, adapting its approach when the OAI chat backend proves incompatible with reasoning models. And it sets the stage for the deeper investigation that follows — the exploration of CUDA graphs, memory tuning, MoE backends, and ultimately the discovery that virtualization overhead is the primary performance bottleneck. Every subsequent tuning decision in the session traces back to the baseline established in this message.