The Turning Point: Enabling CUDA Graphs for GLM-5-NVFP4 on Blackwell GPUs

In the long and arduous journey of deploying the GLM-5-NVFP4 model on eight RTX PRO 6000 Blackwell GPUs, few moments carry as much weight as the server restart command found in message 234. This single message represents a deliberate pivot from "making it work" to "making it fast." After hours of debugging NaN crashes, discovering that only the trtllm NSA backend produced coherent output on SM120 hardware, and establishing a baseline throughput of approximately 225 output tokens per second (516 total tok/s), the assistant now takes a calculated risk: restarting the server with CUDA graphs enabled and a higher memory fraction, aiming to push performance toward the ambitious target of 1,000+ total tokens per second.

The Context: A Baseline Established, A Ceiling Identified

To understand why message 234 matters, one must first appreciate what preceded it. The assistant had just completed a grueling debugging session spanning multiple server restarts ([msg 216] through [msg 228]). The critical breakthrough was discovering that the --nsa-decode-backend trtllm and --nsa-prefill-backend trtllm flags produced coherent, non-NaN output on the SM120 GPUs — a fix that had eluded several earlier attempts with flashmla_kv and flashmla_sparse backends.

With the model finally generating correct responses (including a working Python is_prime function with docstrings and type hints), the assistant turned to performance tuning. The baseline benchmarks told a clear story:

• 16 concurrent requests: ~50 tok/s output throughput
• 32 concurrent requests: ~144 tok/s output, ~304 total tok/s
• 64 concurrent requests (saturated): ~225 tok/s output, ~516 total tok/s All 64 requests succeeded without crashes — a testament to the stability of the trtllm NSA backend. But the assistant knew this was just the starting point. The server was running with --mem-fraction-static 0.85 and, critically, CUDA graphs were disabled. The previous attempts to enable CUDA graphs had ended in crashes, but those were with different NSA backends. The trtllm backend had never been tested with CUDA graphs.

The Decision: A Calculated Risk

Message 234 is the execution of a plan that was carefully negotiated with the user in the preceding messages. In [msg 229], the assistant laid out four options:

1. Restart with --mem-fraction-static 0.92 + CUDA graphs enabled (recommended)
2. Restart with 0.92 mem-fraction only, keep CUDA graphs disabled
3. Both tunings
4. Keep current config The user chose option 1 — "Both tunings (Recommended)" — and the assistant updated its todo list in [msg 230] to reflect the new priority: "Restart with mem-fraction 0.92 + CUDA graphs enabled." But before executing the restart, the assistant performed essential cleanup. In [msg 231], it killed the old server process. In [msg 232], it issued a forceful pkill -9 -f sglang to ensure no zombie processes remained. In [msg 233], it verified that all GPUs showed 0 MiB memory usage — a clean slate. This meticulous cleanup reflects an important assumption: that a fresh process start is necessary to change memory allocation parameters, and that residual GPU memory from the previous server could interfere with the new configuration.

The Message Itself: Anatomy of a Server Restart

The subject message opens with a single word: "Clean." This confirms the GPU memory verification from the previous message — all eight GPUs are at 0 MiB, ready for a fresh allocation.

The assistant then issues a complex bash command that does several things at once:

source ~/ml-env/bin/activate && \
export NCCL_IB_DISABLE=1 && \
export NCCL_P2P_LEVEL=PHB && \
export NCCL_ALLOC_P2P_NET_LL_BUFFERS=1 && \
export NCCL_MIN_NCHANNELS=8 && \
export OMP_NUM_THREADS=8 && \
export SAFETENSORS_FAST_GPU=1 && \
export CUDA_HOME=/usr/local/cuda-12.8 && \
nohup python3 -m sglang.launch_server \
  --model lukealonso/GLM-5-NVFP4 \
  --served-model-name glm-5 \
  --reasoning-parser glm45 \
  --tool-call-parser glm47 \
  --trust-remote-code \
  --tp 8 \
  --mem-fraction-static 0.92 \
  --max-running-requests 64 \
  --kv-cache-dtype auto \
  --quantization modelopt_fp4 \
  --attention-backend flashinfer \
  --fp8-gemm-backend cutlass \
  --nsa-decode-backend trtllm \
  --nsa-prefill-backend trtllm \
  --moe-runner-backend flashinfer_cutlass \
  --disable-custom-all-reduce \
  --enable-flashinfer-allreduce-fusion \
  --host 0.0.0.0 \
  --port 8000 \
  > ~/sglang-glm5.log 2>&1 &

The command is rich with meaning. Let us examine each piece.

The Environment Variables: Assumptions About the Networking Stack

The NCCL environment variables reveal the assistant's assumptions about the hardware topology:

• NCCL_IB_DISABLE=1: Disables InfiniBand communication. This is correct for a system where GPUs communicate over PCIe rather than NVLink or InfiniBand. The RTX PRO 6000 Blackwell GPUs in this setup are connected through the PCIe fabric, not through dedicated GPU interconnects.
• NCCL_P2P_LEVEL=PHB: Sets the peer-to-peer communication level to "PHB" (PCIe Host Bridge). This tells NCCL to route GPU-to-GPU communication through the host's PCIe hierarchy rather than attempting direct GPU peer-to-peer access. This is a critical setting — earlier in the session (as noted in the chunk summary), the assistant discovered that the system is a KVM/QEMU virtual machine (Proxmox) with no direct GPU peer-to-peer support (NS status in NVLink/NVSwitch detection). All cross-GPU transfers must go through host memory, making this setting essential for correct operation.
• NCCL_ALLOC_P2P_NET_LL_BUFFERS=1: Allocates low-latency network buffers for P2P communication, an optimization for the virtualized PCIe path.
• NCCL_MIN_NCHANNELS=8: Sets the minimum number of NCCL communication channels to 8, matching the 8-GPU topology. This ensures sufficient parallelism in the communication fabric.
• OMP_NUM_THREADS=8: Limits OpenMP threads to 8, preventing CPU thread oversubscription during parallel operations.
• SAFETENSORS_FAST_GPU=1: Enables GPU-accelerated safetensor loading, which speeds up model weight loading from disk to GPU memory.
• CUDA_HOME=/usr/local/cuda-12.8: Points to CUDA 12.8 toolkit. This is notable because the system also has CUDA 13.1 installed (from the environment setup in segment 0). The assistant deliberately uses CUDA 12.8 here, likely because the flash-attn and other compiled kernels were built against CUDA 12.8 during the earlier build process.

The Server Flags: What Changed and What Stayed

Comparing this launch command to the previous one (which produced the working baseline), the key differences are:

What changed:

• --mem-fraction-static increased from 0.85 to 0.92
• The --disable-cuda-graph flag is absent (implicitly enabling CUDA graphs) What stayed the same:
• --nsa-decode-backend trtllm and --nsa-prefill-backend trtllm — the critical fix for SM120
• --moe-runner-backend flashinfer_cutlass — the MoE kernel backend
• --attention-backend flashinfer — the attention backend
• --fp8-gemm-backend cutlass — the FP8 GEMM backend
• --disable-custom-all-reduce — disables NCCL custom all-reduce (necessary for virtualized environments without direct P2P)
• --enable-flashinfer-allreduce-fusion — enables fused all-reduce in flashinfer
• --tp 8 — tensor parallelism across 8 GPUs
• --max-running-requests 64 — maximum concurrent requests The absence of --disable-cuda-graph is the most significant change. CUDA graphs allow the GPU to capture and replay entire sequences of kernel launches as a single unit, eliminating kernel launch overhead and improving throughput — especially for the repetitive decode phase of transformer inference. However, CUDA graphs are notoriously fragile. They can crash if the captured graph doesn't match the runtime execution pattern, and they had crashed in earlier attempts when used with the flashmla_kv and flashmla_sparse NSA backends. The assistant's reasoning, made explicit in [msg 229], was that the crashes might have been backend-specific: "try enabling CUDA graphs to see if they work with the trtllm NSA backend." This is a reasonable hypothesis — different attention backends use different CUDA kernel compositions, and some may be more graph-compatible than others. The memory fraction increase from 0.85 to 0.92 is also significant. The HF model card for GLM-5-NVFP4 recommends 0.95, but the assistant chose 0.92 to leave headroom for activation memory during batched decode. With 8 GPUs each having 97,887 MiB of memory, this change allocates approximately 7% more GPU memory to the KV cache — from roughly 83 GiB to 90 GiB per GPU. The previous configuration left ~16.5 GiB free per GPU; the new configuration would reduce that to ~8 GiB, which the assistant judged sufficient for batch processing.

The Timeout: A Deliberate Handoff

The bash command timed out after 30 seconds — the default timeout for the bash tool. This is not an error; it's expected behavior for a server launch command that runs indefinitely in the background. The nohup and & at the end of the command ensure the server continues running even after the SSH session returns. The timeout simply means the tool stopped waiting for the command to complete, which it never will (the server runs until killed).

The assistant's next messages ([msg 235] through [msg 237]) show the server coming up successfully. The log reveals that CUDA graphs were captured for batch sizes [1, 2, 4, 8, 12, 16, 24, 32, 40, 48, 56, 64] — all without out-of-memory errors. The KV cache expanded from 370K to 498K tokens, and ~5.2 GiB remained free per GPU. Most importantly, the model continued to produce correct output ("Paris" for the capital of France) with CUDA graphs enabled.

The Thinking Process: Methodical Optimization

What makes message 234 a compelling subject for analysis is what it reveals about the assistant's thinking process. The assistant is following a clear optimization loop:

1. Establish baseline: Run benchmarks with conservative settings to get a performance floor.
2. Identify constraints: Determine what's limiting performance (in this case, CUDA graphs disabled, low memory fraction).
3. Formulate hypotheses: Propose specific changes and predict their effects.
4. Validate with user: Present options and get buy-in before making disruptive changes.
5. Clean state: Kill old processes, verify clean GPU memory.
6. Execute with monitoring: Launch with new settings, then verify through log inspection and test queries.
7. Measure impact: Run benchmarks to quantify the improvement (or regression). This is textbook performance engineering. The assistant resists the temptation to throw every optimization at the problem at once. Instead, it makes one controlled change (or a small set of related changes), measures the result, and iterates.

Assumptions and Risks

Several assumptions underpin message 234:

1. CUDA graph compatibility with trtllm: The assistant assumes that the trtllm NSA backend will work with CUDA graphs even though other NSA backends crashed. This is an educated guess, not a certainty.
2. Memory headroom: The assistant assumes that 0.92 memory fraction leaves enough room for activation memory during batched decode with 64 concurrent requests. If this is wrong, the server could OOM under load.
3. NCCL configuration correctness: The NCCL environment variables are tuned for a virtualized PCIe environment. If the virtualization layer behaves differently than assumed, communication could be suboptimal or fail.
4. CUDA toolkit version: Using CUDA 12.8 rather than 13.1 assumes that the compiled kernels (flash-attn, etc.) are compatible with this version. This was established during the earlier build process.

Conclusion

Message 234 is a turning point in the deployment story. It marks the transition from debugging to optimization, from fragility to performance. The assistant's methodical approach — cleaning state, making targeted changes, and verifying results — exemplifies how complex ML infrastructure deployment should be done. The success of this restart (confirmed in subsequent messages) paved the way for the deeper performance analysis that followed, including the discovery that virtualization-induced PCIe latency was the fundamental bottleneck limiting throughput. In a single command, the assistant moved the project from "it works" to "how fast can we make it?"