The Counterintuitive Optimization: When CUDA Graphs Make Things Worse

In the high-stakes world of large language model deployment, every millisecond counts. When you're serving a 744-billion-parameter Mixture-of-Experts model like GLM-5-NVFP4 across eight NVIDIA RTX PRO 6000 Blackwell GPUs, the difference between a well-tuned system and a mediocre one can be measured in orders of magnitude of throughput. So when the assistant in this coding session finally managed to enable CUDA graphs — a powerful optimization that captures GPU kernel launch sequences to eliminate CPU-side scheduling overhead — the expectation was clear: throughput should go up. Instead, it went down.

Message [msg 239] captures the moment of this unexpected result, and it is a masterclass in scientific reasoning under the pressure of a complex deployment scenario.

The Setup: A Long Road to CUDA Graphs

To understand why this message matters, we must first understand what led to it. The assistant had been battling the GLM-5-NVFP4 deployment for hours across multiple segments. The model, a quantized 744B MoE using NVIDIA's ModelOpt FP4 format, presented numerous challenges. The most critical was a persistent NaN crash during decode — the model would produce coherent output for the first few tokens, then collapse into numerical instability. This was eventually resolved in [msg 219] through [msg 227] by selecting the correct NSA (NVIDIA Sparse Attention) backends: --nsa-decode-backend trtllm and --nsa-prefill-backend trtllm. These backends, designed for the SM120 architecture of the Blackwell GPUs, finally produced stable output.

With the model running, the assistant established baseline throughput: approximately 225 output tokens per second and 516 total tokens per second with 64 concurrent requests under saturated conditions ([msg 228]). This was respectable but far from the theoretical potential of eight 96GB GPUs. The next logical step was to enable CUDA graphs — a feature that captures the GPU kernel launch sequence into a reusable graph, eliminating the CPU-side overhead of launching individual kernels for each decode step. CUDA graphs are particularly beneficial for small batch sizes where kernel launch overhead dominates.

The assistant had previously been running with --disable-cuda-graph because earlier attempts with different NSA backends had crashed. With the trtllm backends working, the assistant decided to try again. In [msg 234], the server was restarted with --mem-fraction-static 0.92 (up from 0.85) and CUDA graphs enabled. The graphs captured successfully for batch sizes [1, 2, 4, 8, 12, 16, 24, 32, 40, 48, 56, 64] without running out of memory ([msg 237]). A quick sanity check with a chat completion request confirmed the model still produced coherent output: "Paris" with clean reasoning.

The Moment of Truth: Message 239

The message begins with the assistant running the same saturated benchmark used for the baseline — 64 random prompts with 256 input tokens and 256 output tokens, at infinite request rate — against the newly CUDA-graph-enabled server. The result is delivered in the opening line:

Hmm, that's actually slightly slower (194 vs 225 tok/s).

This single sentence encapsulates the entire emotional and intellectual arc of the message. The word "Hmm" signals surprise, a moment of cognitive dissonance. The assistant had every reason to expect improvement. CUDA graphs are a well-established optimization in the SGLang ecosystem. They had captured successfully. The model was producing correct output. Everything pointed to a win. Yet the numbers told a different story: 194 output tok/s versus the previous 225 output tok/s — a regression of approximately 14%.

The Hypothesis: Variable-Length Tokens and the /generate Endpoint

What makes this message so valuable as a piece of technical reasoning is what comes next. The assistant does not panic, does not revert the change, and does not blame the tooling. Instead, it immediately formulates a hypothesis:

The CUDA graphs may not be helping with the /generate endpoint and random tokens (which have variable output lengths).

This hypothesis reveals a deep understanding of how CUDA graphs work and where their benefits manifest. CUDA graphs capture a fixed sequence of kernel launches. When the SGLang server uses the native /generate endpoint (as opposed to the OpenAI-compatible /v1/chat/completions endpoint), the decoding path may involve different kernel configurations depending on the token being generated. Random tokens with variable output lengths mean that the batch composition changes every decode step — some requests finish, new ones arrive, and the batch size fluctuates. CUDA graphs are most effective when the same kernel sequence repeats predictably. If every decode step has a different shape, the captured graph may not match, forcing fallback to the uncaptured path or incurring replay overhead.

The assistant also implicitly acknowledges that the benchmark uses the --backend sglang flag, which communicates via the native /generate endpoint. This endpoint returns results as a single response rather than streaming individual tokens, which means the benchmark tool cannot measure per-token latency (TTFT/TPOT) — it can only measure aggregate throughput. The assistant had noted this limitation earlier in [msg 224], where the native backend produced "negative TPOT, 0ms ITL" metrics.

The Diagnostic Response: Two Parallel Actions

The assistant takes two actions in parallel, both designed to gather more information:

  1. Check the server-side log to see what the server reports for decode throughput with CUDA graphs enabled.
  2. Run a decode-heavy benchmark with longer output (512 tokens vs 256) and shorter input (128 tokens vs 256) to create a workload where decode dominates and CUDA graphs should have more opportunity to help. The server log reveals a decode batch with 22 running requests, 7360 tokens, and a gen throughput of 200.40 tok/s with cuda graph: True. This confirms that CUDA graphs are indeed being used during the benchmark. The server-side measurement (200.40 tok/s) is consistent with the client-side measurement (194 tok/s), validating that the regression is real and not an artifact of the benchmark tool. The second benchmark, with decode-heavy parameters (128 input, 512 output), produces slightly better results — 236 output tok/s — but still within the same range as the baseline without CUDA graphs. The assistant will later conclude in [msg 240] that "the throughput is capped around 200-236 tok/s regardless of CUDA graphs," and that "the bottleneck is the MoE expert computation + all-reduce over PCIe, not kernel launch overhead."

Assumptions and Their Refinement

This message reveals several assumptions that the assistant held, and which were challenged by the data:

Assumption 1: CUDA graphs would improve throughput on Blackwell GPUs. This was a reasonable assumption based on the optimization's reputation in the SGLang community. However, the assistant implicitly assumed that kernel launch overhead was a significant fraction of the total decode time. The results suggest otherwise — on this particular hardware configuration, the bottleneck lies elsewhere.

Assumption 2: The benchmark methodology was adequate to measure the improvement. The assistant assumed that the saturated benchmark with 64 concurrent requests at infinite request rate would capture any throughput gains. In retrospect, this benchmark may have been operating in a regime where CUDA graphs provide minimal benefit. CUDA graphs are most valuable for small batch sizes where kernel launch overhead is proportionally larger. At high concurrency with large batches, the compute time dominates.

Assumption 3: The /generate endpoint would benefit from CUDA graphs in the same way as the streaming endpoint. The assistant's hypothesis about the /generate endpoint interaction with CUDA graphs is astute but unverified. It's possible that the streaming endpoint would show a different result, but the benchmark tool's limitations prevented testing this directly.

Assumption 4: Variable output lengths from random tokens would not significantly impact CUDA graph effectiveness. The assistant correctly identified that random tokens with variable output lengths create a heterogeneous batch environment where CUDA graphs may struggle.

Input Knowledge Required

To fully understand this message, one needs:

Output Knowledge Created

This message produces several valuable insights:

  1. CUDA graphs do not universally improve throughput on this configuration. The 14% regression is a concrete data point that challenges the assumption that this optimization is always beneficial.
  2. The bottleneck is likely elsewhere. The fact that throughput remains stable at ~200-236 tok/s regardless of CUDA graph status suggests that the primary constraint is not kernel launch overhead but rather the MoE expert computation and the all-reduce communication across PCIe.
  3. A diagnostic methodology for optimization evaluation. The assistant demonstrates a rigorous approach: establish a baseline, apply a single change, measure with the same methodology, compare results, formulate a hypothesis when results are unexpected, and design a follow-up experiment to test the hypothesis.
  4. The importance of workload characteristics in benchmarking. The decode-heavy benchmark (128 in, 512 out) produces different results than the balanced benchmark (256 in, 256 out), highlighting that optimization effectiveness depends on the specific workload mix.

The Broader Significance

This message is a microcosm of the challenges faced when deploying cutting-edge AI infrastructure. The assistant is operating at the intersection of multiple complex systems: a novel model architecture (GLM-5 with NVFP4 quantization), experimental GPU hardware (Blackwell SM120), a rapidly evolving serving framework (SGLang nightly), and a virtualized infrastructure (Proxmox KVM). Each layer introduces its own constraints and failure modes.

The counterintuitive result — that an established optimization makes things worse — is a reminder that performance optimization is not a matter of applying a checklist of best practices. It requires careful measurement, hypothesis formation, and iterative experimentation. The assistant's response to this setback is exemplary: rather than discarding the optimization or blindly accepting the result, it digs deeper, checks the server-side metrics, and designs a targeted experiment to isolate the variable of interest.

In the subsequent messages ([msg 240] and beyond), the assistant will pivot to investigating other bottlenecks: MoE runner backends, expert parallelism feasibility, and the virtualization-induced PCIe latency that ultimately emerges as the primary constraint. But message [msg 239] stands as the turning point — the moment when the assistant realized that the easy optimizations had been exhausted and the real bottleneck lay deeper in the system architecture.