The CUDA Graph That Almost Wasn't: Memory Triage in a Multi-GPU LLM Deployment

Introduction

In the high-stakes world of large language model deployment, the difference between a working server and a crashed one often comes down to a single command-line flag. Message 158 of this opencode session captures one such turning point: the moment when an 8-GPU Blackwell system, after an initial Out-of-Memory (OOM) failure, successfully begins capturing CUDA graphs for the GLM-5-NVFP4 model. This message is deceptively brief—a few lines of log output and a comment—but it represents the culmination of a critical debugging loop that reveals deep truths about memory management in modern inference serving stacks.

The Message

Let us quote the message exactly as it appears in the conversation:

Much better. Now capturing CUDA graphs for bs [1, 2, 4, 8, 12, 16, 24, 32, 40, 48, 56, 64] with 8.2GB headroom. KV cache has 425K tokens. Let me wait for graph capture to finish:

>

[bash] sleep 180 && ssh 10.1.230.175 'tail -10 ~/sglang-glm5.log' [2026-02-19 00:01:36 TP0] Warning: You are sending unauthenticated requests to the HF Hub. Please set a HF_TOKEN to enable higher rate limits and faster downloads. [2026-02-19 00:01:37 TP0] max_total_num_tokens=425664, chunked_prefill_size=8192, max_prefill_tokens=16384, max_running_requests=64, context_len=202752, available_gpu_mem=6.60 GB [2026-02-19 00:01:38] INFO: Started server process [7343] [2026-02-19 00:01:38] INFO: Waiting for application startup. [2026-02-19 00:01:38] INFO: ...

At first glance, this reads like a routine progress update. The assistant reports that CUDA graphs are being captured for a wide range of batch sizes, that there is 8.2 GB of headroom per GPU, and that the KV cache can hold 425,664 tokens. The server process has started and is waiting for application startup. But to understand why this message matters, we must understand what came before it.

The Context: An OOM in the Machine Room

The story begins with the first attempt to launch the GLM-5-NVFP4 model. This is a Mixture-of-Experts model quantized to NVIDIA's NVFP4 format, weighing approximately 400 GB. The target hardware is a machine with 8 RTX PRO 6000 Blackwell GPUs, each with 96 GB of VRAM, for a total of 768 GB—comfortably more than the model's footprint.

In message 130, the assistant launched the sglang server with --mem-fraction-static 0.95. This flag controls what fraction of available GPU memory is reserved for the KV cache (the key-value store that enables the model to maintain context across tokens). The remaining 5% is left for model weights, CUDA graph buffers, and runtime overhead. With 96 GB per GPU, 0.95 meant reserving roughly 91.2 GB for KV cache, leaving only 4.8 GB for everything else.

The model weights loaded successfully. But when sglang attempted to capture CUDA graphs—a performance optimization that records GPU kernel execution sequences for reuse—the process crashed with a torch.OutOfMemoryError. The error message (visible in message 150) revealed the problem: GPU 4 had only 247.44 MiB free out of 94.97 GiB total capacity. The 0.95 mem-fraction had allocated so much memory to the KV cache that there was no room left for the CUDA graph capture buffers.

The User's Correction

The user's response in message 149 is a masterclass in concise debugging feedback: "crashed; memory isn't really tight, the model fits very comfortably (nvfp4, 400G and we have >700g vram)." This single sentence corrects a key assumption. The assistant had implicitly assumed that memory was scarce and that a high mem-fraction was necessary to maximize throughput. The user points out that with 768 GB of VRAM and a 400 GB model, there is 368 GB of headroom—plenty of space. The problem was not insufficient total memory, but an overly aggressive allocation policy that starved the CUDA graph capture process.

The Fix: From 0.95 to 0.88

In message 150, the assistant acknowledges the correction and investigates the crash log. In message 153, it relaunches the server with two critical changes:

1. --mem-fraction-static reduced from 0.95 to 0.88: This frees approximately 6.7 GB per GPU (from ~91.2 GB to ~84.5 GB reserved for KV cache), providing the headroom needed for CUDA graph capture.
2. --max-running-requests increased from 8 to 64: With more memory available, the server can handle more concurrent requests, improving throughput. Message 157 shows the immediate result: KV cache is allocated with 425,664 tokens per GPU, consuming 24.37 GB, and 8.61 GB of memory is available per GPU after allocation. This is the foundation for message 158.

What Message 158 Reveals

When the assistant writes "Much better" in message 158, it is expressing genuine relief. The comparison is implicit but stark. In the first attempt (message 147), CUDA graph capture began with only 0.96 GB of available memory and was limited to batch sizes [1, 2, 4, 8]—the bare minimum. Now, with 8.2 GB of headroom (the log shows 6.60 GB available after further allocation, but the assistant reports 8.2 GB from an earlier checkpoint), the graph capture can cover batch sizes up to 64: [1, 2, 4, 8, 12, 16, 24, 32, 40, 48, 56, 64]. This is a dramatic improvement.

The batch size range matters because CUDA graphs are captured once and reused. A graph captured for batch size 8 cannot be used for a request with batch size 12. By capturing graphs for a dense set of batch sizes up to the maximum running requests limit (64), the server ensures that every possible request size has an optimized execution path. This is essential for production throughput.

The KV cache capacity of 425,664 tokens is also significant. With a context length of 202,752 tokens (visible in the log), this means the server can handle multiple long-context requests simultaneously. The chunked_prefill_size of 8192 tokens and max_prefill_tokens of 16384 indicate that the server will process large prefills in chunks, balancing memory usage and latency.

The Thinking Process Visible in This Message

The assistant's reasoning is visible in several dimensions. First, there is the explicit comparison: "Much better" followed by the specific numbers that justify the assessment. The assistant is not just reporting progress; it is evaluating the new state against the old one.

Second, the assistant demonstrates a systematic monitoring approach. It issues a sleep 180 command—a three-minute wait—before checking the logs. This is not arbitrary; CUDA graph capture is documented to take "up to several minutes" (as seen in message 147). The assistant is balancing the need for timely updates against the risk of interrupting a long-running operation.

Third, the assistant chooses to show the log output verbatim rather than summarizing it. This is a deliberate transparency decision. By including the raw log lines, the assistant allows the user (or a future reader) to verify the numbers independently. The log confirms the server process has started (PID 7343) and is waiting for application startup, which means the CUDA graph capture is proceeding in the background.

Fourth, the assistant implicitly acknowledges the user's earlier correction. The phrase "Much better" is a direct response to the user's "crashed" in message 149. The assistant is saying, in effect, "Your diagnosis was correct, and the fix is working."

Assumptions and Their Consequences

This message, and the debugging loop that preceded it, reveals several assumptions that were made and either validated or corrected:

Assumption 1: Higher mem-fraction is always better. The assistant initially chose 0.95 because it maximizes KV cache capacity, which in theory allows longer context windows and more concurrent requests. The assumption was that any memory not used for KV cache is wasted. This assumption was wrong in practice because CUDA graph capture requires a non-trivial amount of free memory. The user's correction was essential.

Assumption 2: The OOM was due to insufficient total memory. The assistant's initial reaction (message 150) was to check for memory pressure. The user's intervention clarified that the issue was allocation policy, not capacity. This is a subtle but important distinction: having enough total memory does not help if the allocation policy reserves it all for one purpose.

Assumption 3: CUDA graph capture would succeed with minimal headroom. The first attempt had 0.96 GB free, which proved insufficient. The second attempt has 8.2 GB free, which appears to be working. The assistant is implicitly assuming that 8.2 GB is enough—an assumption that proves correct in subsequent messages.

Input Knowledge Required

To fully understand message 158, one needs knowledge in several areas:

• SGLang's memory model: Understanding that --mem-fraction-static controls the fraction of GPU memory reserved for KV cache, and that the remainder is used for weights, activations, and runtime buffers.
• CUDA graph capture: Knowing that this is a performance optimization that records GPU kernel launches for reuse, and that it requires additional memory for the graph buffers.
• Tensor parallelism: Understanding that with --tp 8, the model is split across 8 GPUs, and each GPU holds a shard of the weights and KV cache.
• NVFP4 quantization: Knowing that this is a 4-bit floating-point format that reduces model size significantly (400 GB instead of ~800 GB for FP16), which is why the model fits comfortably in 768 GB.
• Mixture-of-Experts architectures: Understanding that GLM-5 uses MoE, which has implications for memory layout and inference scheduling.

Output Knowledge Created

This message creates several pieces of actionable knowledge:

1. A validated configuration: --mem-fraction-static 0.88 with --max-running-requests 64 works for GLM-5-NVFP4 on 8× Blackwell GPUs. This is a reusable recipe for similar deployments.
2. Capacity benchmarks: The KV cache holds 425,664 tokens per GPU, the context length is 202,752 tokens, and the server can handle up to 64 concurrent requests. These numbers inform capacity planning.
3. CUDA graph coverage: The batch size range [1, 2, 4, 8, 12, 16, 24, 32, 40, 48, 56, 64] means the server has optimized execution paths for all expected request sizes.
4. A debugging methodology: The pattern of "launch, monitor, diagnose, adjust, relaunch" demonstrated here is a template for deploying large models on new hardware.

The Broader Significance

Message 158 is, in one sense, a routine progress update. But it is also a microcosm of the challenges involved in deploying cutting-edge AI models. The GLM-5-NVFP4 model is at the frontier of what is possible—a 400 GB MoE model quantized to 4-bit precision, running on brand-new Blackwell GPUs with a nightly build of sglang. Every component is bleeding-edge, and every configuration choice is a guess until validated.

The debugging loop that produced this message—launch, crash, diagnose, fix, relaunch—is the fundamental rhythm of systems engineering. The assistant's willingness to show its work, share raw logs, and respond to user corrections is what makes this session a valuable case study. Message 158 is the moment when the rhythm pays off: the crash is behind them, the server is starting, and the long wait for CUDA graph capture begins.