Benchmarking a 744B MoE on Blackwell GPUs: The Methodical Pursuit of Throughput

Introduction

In the high-stakes world of large language model deployment, getting a model to produce coherent output is only half the battle. The other half — the one that determines whether a deployment is practically useful — is performance: how many tokens can the system generate per second under realistic concurrent load? Message [msg 226] captures a pivotal moment in this journey, where the assistant transitions from celebrating a working model to systematically characterizing its throughput ceiling. This message, appearing after the resolution of a critical NaN decode crash that had plagued earlier attempts, represents the beginning of the performance tuning phase for the GLM-5-NVFP4 model running on eight NVIDIA RTX PRO 6000 Blackwell GPUs.

The Road to This Message

To understand the significance of [msg 226], one must appreciate the arduous path that preceded it. Earlier in the session, the assistant had been locked in a battle with a persistent NaN (Not a Number) crash that occurred during the decode phase of inference. Every attempt to serve the GLM-5-NVFP4 — a 744-billion-parameter Mixture-of-Experts model quantized with NVIDIA's ModelOpt FP4 scheme — resulted in garbage output or outright crashes when using default attention backends. The breakthrough came with the discovery that the --nsa-decode-backend trtllm and --nsa-prefill-backend trtllm flags were essential for the SM120 architecture of the Blackwell GPUs, while other NSA backends like flashmla_kv and flashmla_sparse produced NaN results ([msg 218]).

Once the model was producing coherent output — correctly answering "What is 2+2?" with "4" and writing a proper is_prime Python function — the assistant immediately pivoted to performance characterization. Message [msg 225] established a baseline: 32 concurrent requests yielded approximately 144 output tokens per second and 304 total tokens per second, with an average concurrency of 18.4 requests. All 32 requests succeeded without crashes, confirming that the configuration was stable under load.

The Message Itself

Message [msg 226] opens with a critical observation drawn from the server logs:

The server-side decode throughput peaks at ~208 tok/s with 21 concurrent requests. This is decent for a 744B MoE on PCIe-only GPUs without CUDA graphs.

This single sentence packs remarkable density. The assistant has extracted a server-side metric — 208 tokens per second decode throughput — from the SGLang runtime logs, which report per-batch statistics. The "21 concurrent requests" indicates the peak batch size at which this throughput was observed. The qualification "decent" is carefully calibrated: 208 tok/s for a 744B parameter model is objectively impressive, especially given the hardware constraints the assistant explicitly acknowledges.

The phrase "PCIe-only GPUs" is a crucial qualifier. These eight RTX PRO 6000 GPUs, despite being connected via PCIe, lack the high-speed NVLink interconnects that typically bind multi-GPU configurations in NVIDIA's data center lineup. This means all cross-GPU communication — and for a model of this scale using tensor parallelism across eight GPUs, there is constant cross-GPU communication — must traverse the relatively limited PCIe bus. The assistant implicitly understands that this is a fundamental bottleneck that no amount of software tuning can fully eliminate.

The mention of "without CUDA graphs" is equally telling. CUDA graphs are a performance optimization that allows the GPU to launch a pre-compiled sequence of operations without repeated CPU-side launch overhead. The assistant flags this as an available optimization that has not yet been applied, signaling that the 208 tok/s figure is a baseline that can potentially be improved.

The Benchmark Scaling Strategy

The second half of the message reveals the assistant's methodical approach to performance characterization:

Now let me run a bigger benchmark to get stable numbers and test throughput scaling with higher concurrency:

The assistant then launches a benchmark with 64 prompts at a request rate of 16 per second, using 256-token input and output lengths. This is a deliberate escalation from the previous 32-prompt benchmark. The logic is sound: to find the true throughput ceiling of the server, one must saturate it with enough concurrent requests that the batching mechanism can operate at maximum efficiency. Larger batch sizes allow the GPU to amortize the fixed costs of kernel launches and memory transfers across more tokens, but they also increase memory pressure and latency.

The choice of 64 prompts at rate 16 is strategic. With a request rate of 16 per second and each request producing 256 output tokens at roughly 12.5 tok/s per request (the single-request throughput observed earlier), each request would take about 20 seconds to complete. At rate 16, after just 4 seconds, 64 requests would be in flight simultaneously, fully saturating the server's capacity. This is precisely the condition needed to measure peak throughput.

The assistant also chooses the sglang native backend for benchmarking rather than the OpenAI-compatible chat endpoint. This decision, informed by earlier failures where the chat endpoint crashed on reasoning models that return content: null ([msg 223]), shows adaptive learning from previous mistakes. The native /generate endpoint handles the model's output format correctly.

Assumptions and Their Validity

The message rests on several implicit assumptions. First, the assistant assumes that higher concurrency will yield higher aggregate throughput — a reasonable assumption for a batched inference server, but one that only holds until memory or compute becomes saturated. Second, it assumes that the 256-token input and output lengths are representative of the workload the model will face in production. Third, it assumes that the server configuration — particularly the --mem-fraction-static 0.85 setting — provides enough GPU memory headroom for 64 concurrent requests without out-of-memory errors.

The assistant also assumes that CUDA graphs are not currently enabled. This is confirmed by the server log in [msg 225], which shows "cuda graph: False" in the decode batch statistics. The implication is that further tuning — enabling CUDA graphs, adjusting memory fractions, or switching MoE runner backends — could push throughput higher, and the assistant is building a baseline against which those improvements can be measured.

What This Message Creates

Message [msg 226] produces several forms of output knowledge. The most concrete is the benchmark result that appears in the subsequent message [msg 227]: with 64 concurrent requests at rate 16, the server achieves 196 output tok/s and 451 total tok/s, with all 64 requests succeeding and peak concurrency hitting the full 64. This confirms that the server scales nearly linearly from 32 to 64 concurrent requests (144 → 196 output tok/s), though with diminishing returns — doubling the concurrency yields only a 36% throughput increase, suggesting the system is approaching its saturation point.

More broadly, the message establishes a methodology for performance evaluation that will guide the remainder of the session. The assistant has demonstrated a pattern: measure, analyze, escalate, repeat. Each benchmark informs the next, with parameters adjusted based on observed behavior. This systematic approach is what transforms a working prototype into a production-ready deployment.

The Thinking Process

What is most striking about [msg 226] is what it reveals about the assistant's reasoning. The observation of "~208 tok/s with 21 concurrent requests" is not a number pulled from thin air — it is extracted from the server-side log line gen throughput (token/s): 170.24 seen in [msg 225], which the assistant mentally adjusts upward based on the peak observed across multiple batches. The assistant is not merely executing commands; it is interpreting telemetry, forming hypotheses about system behavior, and designing experiments to test those hypotheses.

The phrase "This is decent for a 744B MoE on PCIe-only GPUs without CUDA graphs" reveals a sophisticated mental model of the performance landscape. The assistant is benchmarking against an implicit expectation of what such a configuration should achieve, and the word "decent" signals that the result is within the expected range — not disappointing, but also not the ceiling. The explicit enumeration of constraints (744B parameters, MoE architecture, PCIe-only interconnect, no CUDA graphs) shows that the assistant is accounting for each factor that limits throughput, rather than treating the raw number as an absolute measure of performance.

Conclusion

Message [msg 226] is a transitional moment in the session — the pivot point between "does it work?" and "how fast does it work?" — but it is far more than a simple status update. It encapsulates the assistant's methodical approach to performance engineering: extracting insights from server telemetry, forming calibrated judgments about system capability, and designing increasingly demanding benchmarks to probe the throughput frontier. The 208 tok/s figure serves as both a validation of the current configuration and a baseline against which future optimizations will be measured. In the broader narrative of deploying GLM-5-NVFP4 on Blackwell GPUs, this message marks the moment when the conversation shifted from debugging to optimization — from survival to performance.