The Saturation Point: Benchmarking GLM-5-NVFP4 on Eight Blackwell GPUs

In message 228 of this opencode session, the assistant reaches a critical inflection point. After hours of battling NaN crashes, incompatible attention backends, and configuration errors, the GLM-5-NVFP4 model is finally running stably on eight RTX PRO 6000 Blackwell GPUs. The message reports the results of a saturated throughput benchmark and then pivots decisively from "making it work" to "making it fast." This brief but pivotal message — just a few lines of summary followed by a GPU memory check — encapsulates the transition between two fundamentally different phases of ML deployment engineering.

The Road to Saturated Throughput

To understand message 228, one must appreciate the journey that preceded it. The assistant had been struggling for hours to get the GLM-5-NVFP4 model — a massive 744B-parameter Mixture-of-Experts reasoning model — to produce coherent output on NVIDIA's SM120 (Blackwell) architecture. Earlier attempts using default NSA (Nested Sparse Attention) backends like flashmla_kv and flashmla_sparse had resulted in NaN crashes during decode, producing garbage output. The breakthrough came in message 215 when the assistant discovered that --nsa-decode-backend trtllm and --nsa-prefill-backend trtllm produced clean, correct reasoning chains.

Once the model was producing valid output, the assistant methodically established baseline throughput. Message 224 showed a meager 50.55 tok/s with 16 prompts. Message 225 improved to 144 output tok/s with 32 concurrent requests. Message 227 reached 196 output tok/s with 64 requests at rate 16. Each step increased concurrency, pushing the server harder while verifying that no crashes occurred under load.

Message 228 represents the capstone of this benchmarking phase: the saturated throughput test with --request-rate inf, which floods the server with all 64 requests simultaneously to find its maximum sustainable throughput. The results — 225 output tok/s and 516 total tok/s — are the first reliable measurement of what this eight-GPU Blackwell system can deliver with the current, untuned configuration.

Interpreting the Numbers

The benchmark results in message 228 tell a nuanced story. The output throughput of 225 tokens per second across eight GPUs translates to roughly 28 tokens per second per GPU — modest for a 744B-parameter model, but remarkable given that these GPUs are connected only via PCIe with no NVLink. The model's MoE architecture means each token activates only a subset of experts, but the all-reduce communication for expert routing still requires significant cross-GPU data movement.

The total throughput of 516 tok/s includes prefill tokens, which are computationally cheaper than decode tokens. The ratio of total to output throughput (about 2.3:1) indicates that prefill work constitutes a significant portion of the server's workload, which is expected for a reasoning model that generates long chains of thought before producing final answers.

Perhaps the most important number is the one not explicitly stated: all 64 requests succeeded. After hours of debugging NaN crashes, this confirmation of stability under maximum load is the real achievement. The model is not just working — it's working reliably under stress.

The Tuning Hypothesis

What makes message 228 particularly interesting is the assistant's immediate pivot to optimization. The very next sentence after reporting the benchmark results is: "Now let me consider tuning options. The main bottleneck is probably the lack of CUDA graphs (disabled because of SM120 compatibility)."

This statement reveals several layers of the assistant's mental model. First, it identifies CUDA graphs as the primary performance lever. CUDA graphs allow the GPU to capture and replay sequences of operations without CPU involvement, reducing launch latency and improving throughput. The assistant knows that CUDA graphs were disabled — likely because the SM120 architecture requires specific graph capture support that wasn't available or was causing issues.

Second, the assistant identifies memory pressure as the secondary concern. By checking --mem-fraction-static, it's considering whether to allocate more GPU memory to the KV cache, which would allow larger batch sizes and potentially higher throughput. The current memory usage of 80,679 MiB out of 97,887 MiB per GPU (about 82.4%) leaves roughly 17 GB of headroom per GPU — significant capacity that could be redirected to the KV cache.

The assistant's reasoning chain is visible in the sequence of actions: benchmark to establish baseline, identify the likely bottleneck (CUDA graphs), check resource availability (GPU memory), and then plan the next configuration change. This is classic performance tuning methodology — measure, hypothesize, verify.

The GPU Memory Check

The nvidia-smi command in message 228 reveals the current memory allocation: 80,679 MiB used, 16,574 MiB free out of 97,887 MiB total per GPU. This is a critical data point for the tuning decision. With 17 GB of free memory per GPU, there is substantial headroom to increase --mem-fraction-static from its current value (likely 0.85 based on the server launch command shown in message 213).

The assistant's decision to check memory before making tuning changes demonstrates good engineering discipline. Increasing the memory fraction without checking available headroom could cause out-of-memory errors, crashing the server and wasting time. By verifying that 17 GB per GPU is available, the assistant establishes that there is room to grow the KV cache allocation.

However, the assistant makes an implicit assumption here: that increasing the KV cache will improve throughput. This is true only if the current bottleneck is batch-size limited by KV cache capacity rather than by compute or communication. Given the PCIe-only interconnect, communication latency might be the actual bottleneck, and increasing the KV cache could simply shift the bottleneck without improving overall throughput. The assistant would need to test this hypothesis empirically.

Knowledge Required and Created

To fully understand message 228, the reader needs knowledge of several domains: GPU memory hierarchy and allocation strategies in LLM serving, the role of CUDA graphs in reducing kernel launch overhead, the architecture of MoE models and their communication patterns, and the benchmarking methodology used by SGLang's bench_serving tool. The message also assumes familiarity with the specific configuration parameters (--mem-fraction-static, --disable-cuda-graph, --request-rate) and their interactions.

The message creates new knowledge about the performance characteristics of GLM-5-NVFP4 on SM120 hardware: a baseline saturated throughput of 225 output tok/s across eight PCIe-connected Blackwell GPUs, confirmation that the trtllm NSA backend configuration is stable under maximum load, and a data point about memory utilization (80.7 GB per GPU) that informs subsequent tuning decisions.

A Pivotal Moment

Message 228 is a hinge point in the conversation. Everything before it was about survival — getting the model to produce coherent output without crashing. Everything after it will be about optimization — squeezing every drop of performance from the hardware. The assistant's disciplined approach — establishing a reliable baseline before making changes, checking resource availability before adjusting parameters, and articulating clear hypotheses about bottlenecks — exemplifies the systematic methodology required for production ML deployment.

The message also reveals the assistant's awareness of its own limitations. The phrase "probably the main bottleneck" is hedged, acknowledging that the diagnosis is a hypothesis rather than a certainty. This intellectual honesty is crucial in complex systems where bottlenecks are often non-obvious and interactions between components can produce counterintuitive results. The assistant is setting up an experiment, not declaring a conclusion.

In the broader narrative of the opencode session, message 228 marks the moment when the deployment shifts from fragile to robust. The model is no longer a science experiment that might crash at any moment — it is a service with measurable, reproducible performance characteristics. The work of optimization can now begin from a position of stability and knowledge, not desperation and guesswork.