The CUDA Graph Milestone: Validating Inference Stability on SM120 with GLM-5-NVFP4

In the high-stakes world of deploying large language models on novel hardware, few moments are as tense as the first inference request after a risky configuration change. Message [msg 237] captures precisely such a moment: the assistant has just restarted the SGLang server serving the 744B-parameter GLM-5-NVFP4 model across eight RTX PRO 6000 Blackwell GPUs (SM120 architecture), this time with CUDA graphs enabled and a higher memory fraction. The message reports that CUDA graphs were captured successfully across a wide range of batch sizes, the KV cache was expanded by 35%, and — crucially — a test inference produces the correct answer "Paris" with a clean reasoning chain. This message is the culmination of a long debugging arc and a critical validation gate before the session moves into performance tuning.

The Road to This Message

To understand why message [msg 237] carries such weight, one must trace the preceding struggle. The GLM-5-NVFP4 deployment had been plagued by NaN crashes during decode — the model would load, begin generating, and then produce garbage or crash entirely. The breakthrough came in [msg 218] when the assistant discovered that the --nsa-decode-backend trtllm and --nsa-prefill-backend trtllm flags (using NVIDIA's TensorRT-LLM attention backends) produced coherent output on SM120, while the other NSA backends (flashmla_kv, flashmla_sparse) consistently produced NaN. This was a narrow escape: the model was functional, but with CUDA graphs disabled (the --disable-cuda-graph flag was in the working configuration), leaving performance on the table.

The baseline benchmarks in [msg 228] established a throughput of approximately 225 output tokens per second and 516 total tokens per second with 64 concurrent requests under saturated conditions. The user then posed a tuning question in [msg 229]: should they restart with a higher memory fraction (0.92 instead of 0.85), try enabling CUDA graphs (which had crashed previously with different NSA backends), or both? The assistant chose the most aggressive option — both tunings simultaneously — reasoning that the earlier CUDA graph crashes were caused by incompatible NSA backends, not by CUDA graphs themselves. This was a calculated risk.

What the Message Reports

The message opens with a triumphant announcement: "CUDA graphs captured successfully!" It then enumerates three concrete improvements:

  1. CUDA graphs captured for batch sizes [1, 2, 4, 8, 12, 16, 24, 32, 40, 48, 56, 64] — all without OOM. This is the most important line. CUDA graphs are a technique that captures the entire GPU kernel launch sequence into a compiled graph, eliminating CPU-side launch overhead for repeated operations. The fact that all twelve batch sizes captured without out-of-memory errors means the server can handle everything from single requests to full 64-request batches with graph-accelerated decoding.
  2. KV cache now 498K tokens (up from 370K with mem-fraction 0.85). The higher memory fraction (--mem-fraction-static 0.92) allocates more GPU memory to the key-value cache, increasing capacity by 128K tokens — a 35% improvement. This directly translates to the ability to handle longer sequences or more concurrent requests before hitting memory limits.
  3. ~5.2GB free per GPU after everything. This headroom is important for activation memory during batched decode and provides a safety margin against OOM during peak usage. The assistant then immediately pivots to validation: "Now let me test that it actually works and doesn't NaN with CUDA graphs during decode." This reveals the core anxiety — the earlier NaN crashes were specifically decode-related, and CUDA graphs introduce a new path through the GPU execution that could potentially trigger the same numerical instability. The test uses a trivial question ("What is the capital of France? Answer in one word.") with conservative parameters (256 max tokens, zero temperature). The response comes back clean: "content":"Paris" with a coherent reasoning chain showing the model thinking through the problem step by step.

The Reasoning and Decision-Making Process

The message reveals several layers of reasoning. First, the assistant is operating under a clear priority hierarchy: stability first, then performance. The decision to test with a simple curl request before running benchmarks demonstrates this — there is no point measuring throughput if the server produces garbage output or crashes under load. The choice of test question is deliberate: it requires factual recall (not reasoning chains that might expose numerical issues), has a single correct answer, and completes in very few tokens, minimizing the window for decode errors.

Second, the assistant's decision to combine both tunings in a single restart reflects a judgment about risk correlation. The earlier CUDA graph failures (mentioned in [msg 229]) occurred with different NSA backends — likely flashmla_kv or flashmla_sparse. The assistant implicitly assumes that the CUDA graph mechanism itself is not the root cause; rather, the incompatibility lies at the intersection of specific NSA backends and graph capture. By switching to trtllm backends, the assistant hypothesizes that CUDA graphs will work. This is a reasonable inference, but it is an assumption nonetheless — one that could have resulted in yet another crash and another wasted server restart cycle.

Third, the message shows the assistant managing multiple constraints simultaneously. The memory fraction increase (0.92) risks OOM during the graph capture phase, which itself requires additional memory for storing the captured graphs. The assistant monitors this carefully, noting that all batch sizes captured "without OOM" and that ~5.2GB remains free. The KV cache expansion to 498K tokens is a secondary benefit of the higher memory fraction, but it also increases memory pressure during decode — a tradeoff the assistant implicitly accepts.

Input Knowledge Required

To fully understand this message, one needs familiarity with several concepts:

Output Knowledge Created

This message creates several important pieces of knowledge:

  1. CUDA graphs are compatible with the trtllm NSA backend on SM120. This was not known before — earlier attempts with other NSA backends had failed. The successful capture across all batch sizes up to 64 provides strong evidence that the incompatibility was backend-specific, not architecture-wide.
  2. The expanded KV cache (498K tokens) is stable under the trtllm NSA configuration. Simply increasing --mem-fraction-static to 0.92 could theoretically cause OOM during decode if the activation memory requirements of the trtllm backend are higher than expected. The successful inference test confirms this is not the case.
  3. The server produces correct output with CUDA graphs enabled. The "Paris" response with its reasoning chain demonstrates that numerical stability is preserved through the graph-captured execution path. This is non-trivial — CUDA graphs can introduce subtle timing changes that expose race conditions or numerical precision issues in GPU kernels.
  4. A validated methodology for testing configuration changes. The pattern of "restart → verify capture → test single inference → benchmark" established here becomes the template for subsequent tuning iterations.

What the Message Does Not Yet Know

In a twist that only becomes apparent in the following messages ([msg 238] and [msg 239]), the CUDA graphs ultimately provide minimal throughput improvement. The saturated benchmark shows ~194 output tok/s with CUDA graphs, compared to ~225 without — actually slightly slower. The assistant correctly diagnoses this: "the bottleneck is the MoE expert computation + all-reduce over PCIe, not kernel launch overhead." The CUDA graphs were solving a problem that wasn't the bottleneck.

This does not diminish the importance of message [msg 237]. The validation that CUDA graphs work correctly on SM120 with trtllm backends is itself valuable knowledge — it rules out one class of potential optimizations and narrows the search for the true bottleneck. In engineering debugging, proving what doesn't help is often as important as proving what does. The message represents a hypothesis tested to completion, with clear evidence either way.

The Broader Context

Message [msg 237] sits at a pivotal moment in the deployment story. The NaN crash saga is definitively behind the team — the model is stable, the server is tuned, and the focus shifts from "will it work?" to "how fast can it go?" The subsequent investigation into PCIe bottlenecks, virtualization overhead, and expert parallelism (documented in the chunk summary) all flow from the foundation established here. Without the successful CUDA graph capture and validation, the team would still be wondering whether graph capture was viable on SM120, and the diagnosis of the true bottleneck (cross-GPU communication latency in the virtualized environment) would have been delayed.

The message also exemplifies a key pattern in AI-assisted engineering: the assistant acts as both operator and analyst, executing commands while simultaneously reasoning about their implications. The decision to test with a simple factual question, the careful monitoring of memory headroom, and the implicit hypothesis about NSA backend compatibility all demonstrate a level of systems thinking that goes beyond rote command execution. This is the assistant functioning as a junior engineer — competent, methodical, and learning from each iteration.