The Moment of Validation: CUDA Graphs Go Live on GLM-5-NVFP4

Introduction

In any complex engineering effort, there comes a moment when a risky change either pays off or collapses into failure. For this opencode session—deploying the massive GLM-5-NVFP4 mixture-of-experts model across eight NVIDIA RTX PRO 6000 Blackwell GPUs—that moment arrived in message [msg 238]. After a long struggle with NaN crashes during decode, incompatible attention backends, and the ever-present threat of out-of-memory errors, the assistant typed a simple confirmation: "CUDA graphs work perfectly! Correct answer 'Paris' with clean reasoning." Then, without pause, it launched a saturated throughput benchmark to quantify the improvement.

This message is deceptively brief. On its surface, it is a single tool call—a bash command running sglang.bench_serving with 64 concurrent prompts. But beneath that surface lies the culmination of hours of debugging, the validation of a high-risk configuration change, and the transition from "does it work?" to "how fast does it work?" It is the hinge point of the entire deployment effort.

The Context: Why CUDA Graphs Were the Prize

To understand why message [msg 238] matters, one must appreciate the journey that led to it. The GLM-5-NVFP4 model is a 744-billion-parameter MoE (Mixture of Experts) transformer, quantized to NVIDIA's NVFP4 format. Deploying it across eight GPUs using tensor parallelism on a system limited by PCIe interconnects (the GPUs are in a Proxmox VM without direct peer-to-peer NVLink support) is inherently challenging.

Earlier in the session, the assistant had discovered that the default NSA (Native Sparse Attention) backends—flashmla_kv and flashmla_sparse—produced NaN values during decode on the SM120 Blackwell GPUs. Only the trtllm NSA backend produced coherent output ([msg 218]). This discovery was critical, but it left performance on the table. The server was running with --disable-cuda-graph, meaning SGLang's CUDA graph optimization—which precompiles CUDA kernels for common batch sizes to reduce launch overhead—was turned off.

CUDA graphs are particularly valuable for inference serving because they amortize kernel launch costs across many iterations. For a model this large, running on PCIe-bound GPUs, every microsecond of overhead matters. But enabling CUDA graphs is risky: they require capturing a snapshot of the GPU's execution state, and if the memory footprint doesn't fit, the capture fails with an OOM error or, worse, produces silently incorrect results. Earlier attempts with different NSA backends had crashed. The question was whether the trtllm backend would behave differently.

The Decision: Higher Memory Fraction and CUDA Graphs

In message [msg 229], the assistant presented the user with a choice: restart the server with --mem-fraction-static 0.92 (up from 0.85) and try enabling CUDA graphs, or take a safer path. The user chose the aggressive option: both tunings simultaneously.

This was a calculated gamble. Increasing the memory fraction from 0.85 to 0.92 reclaims more GPU memory for the KV cache—moving from 370K tokens to 498K tokens—but leaves less headroom for activation memory during batched decode. CUDA graph capture itself also consumes memory. The assistant had to trust that the trtllm NSA backend, which had proven stable during decode, would also be compatible with CUDA graph capture.

The server was killed (<msg id=231-233>), and a new launch command was issued ([msg 234]). The key differences from the previous working configuration were:

The Validation: "CUDA graphs work perfectly!"

Message [msg 237] showed the server coming up successfully. The log confirmed that CUDA graphs were captured for batch sizes [1, 2, 4, 8, 12, 16, 24, 32, 40, 48, 56, 64]—every planned batch size—without OOM. The KV cache had grown to 498K tokens, and ~5.2GB remained free per GPU.

But capture success is not the same as inference correctness. The assistant then sent a test request via curl: "What is the capital of France? Answer in one word." The response came back: &#34;content&#34;:&#34;Paris&#34; with a clean reasoning chain. No NaN, no crash, no garbled output.

This is the moment captured in message [msg 238]. The assistant opens with a declarative summary: "CUDA graphs work perfectly! Correct answer 'Paris' with clean reasoning." The tone is matter-of-fact, but the weight of that statement is enormous. After hours of debugging NaN crashes, after trying multiple NSA backends, after carefully managing memory fractions and compilation flags—the risky configuration change had succeeded.

The Benchmark: Measuring the Improvement

Without pausing to celebrate, the assistant immediately pivots to measurement. The bash command in message [msg 238] runs sglang.bench_serving with the same parameters used to establish the baseline in message [msg 228]: 64 prompts, 256 input tokens, 256 output tokens, infinite request rate. This is a deliberate apples-to-apples comparison.

The benchmark arguments reveal several assumptions baked into the testing methodology:

What Happened Next: An Unexpected Result

The result, revealed in message [msg 239], was surprising: the saturated throughput with CUDA graphs was 194 output tok/s, compared to the baseline of 225 output tok/s without CUDA graphs. The CUDA graphs had actually made throughput slightly worse.

This counterintuitive result highlights several important lessons. First, CUDA graphs primarily reduce kernel launch overhead, which matters most for small batch sizes and short sequences. For large, compute-bound models like GLM-5-NVFP4 running on PCIe-bound GPUs, the bottleneck is data transfer and compute, not kernel launch. Second, the --disable-cuda-graph flag in the baseline configuration may have been masking other differences—the baseline had --mem-fraction-static 0.85 while the new config had 0.92, and the memory allocation changes could interact with the scheduler's batching behavior.

The assistant's response in message [msg 239] shows intellectual honesty: "Hmm, that's actually slightly slower." Rather than dismissing the result, the assistant checks the server-side log (which shows cuda graph: True with 200.40 tok/s, confirming graphs are active) and then pivots to a longer-output benchmark (512 output tokens) to test a more decode-heavy workload where CUDA graphs might show benefit.

Assumptions and Their Consequences

Several assumptions are embedded in message [msg 238]:

  1. CUDA graphs would improve throughput: This was the working hypothesis, and it turned out to be incorrect for this specific workload. The assumption was reasonable—CUDA graphs are widely recommended for inference serving—but it didn't account for the PCIe-bound nature of this particular deployment.
  2. The benchmark parameters would produce comparable results: The assistant carefully matched the benchmark configuration to the baseline, which is methodologically sound. However, the change in memory fraction (0.85 to 0.92) means the comparison isn't solely measuring CUDA graph impact—it's measuring the combined effect of higher KV cache allocation and CUDA graphs.
  3. Random token generation is a valid proxy for real workloads: This is a standard assumption in LLM benchmarking, but it's worth noting that real user traffic has different patterns (variable input lengths, cache hits, etc.) that could interact differently with CUDA graphs.
  4. The server was in a comparable state: The assistant verified that the old server was fully killed and memory was freed before starting the new one. This is good practice and avoids contamination from leftover processes.

Input and Output Knowledge

To fully understand message [msg 238], one needs input knowledge of:

The Thinking Process

The assistant's reasoning in message [msg 238] follows a clear pattern:

  1. Validate correctness first: Before running any benchmark, confirm the model produces correct output. The curl test in message [msg 237] served this purpose, and message [msg 238] explicitly confirms the result.
  2. Measure systematically: Use the exact same benchmark parameters as the baseline to enable direct comparison. The assistant doesn't change multiple variables at once—it isolates the CUDA graph change.
  3. Document the configuration: The benchmark output includes the full Namespace of arguments, creating an auditable record of what was tested.
  4. Proceed without delay: There's no gap between validation and measurement. The assistant moves immediately from "it works" to "how fast?" This reflects an engineering mindset where correctness is the prerequisite, not the goal.

Conclusion

Message [msg 238] is a study in the rhythm of engineering work: validate, measure, iterate. It captures the satisfaction of a risky change succeeding, the discipline of rigorous benchmarking, and the humility of being surprised by the results. The CUDA graphs worked perfectly in terms of correctness—no NaN, no crash—but the throughput improvement was negligible or slightly negative. This is not a failure; it's data. The assistant's ability to accept the unexpected result and pivot to a different workload (longer outputs in message [msg 239]) demonstrates the adaptive mindset that characterizes effective technical work.

In the broader narrative of deploying GLM-5-NVFP4, message [msg 238] marks the end of the "does it work?" phase and the beginning of the "how do we make it faster?" phase. The answer to the first question was a definitive yes. The answer to the second would require deeper investigation into PCIe bottlenecks, virtualization overhead, and the fundamental limits of serving a 744B-parameter model on non-NVLink GPUs.