The 5% Trap: Diagnosing a CUDA Graph Capture OOM on an 8-GPU Blackwell Server

In a single, deceptively short message from an opencode coding session, an AI assistant identifies the root cause of a server crash during deployment of a 400-billion-parameter quantized MoE model across eight NVIDIA RTX PRO 6000 Blackwell GPUs. The message — message index 151 in the conversation — contains just two sentences of analysis followed by a bash command to kill the failed process. Yet within those few lines lies a rich story of diagnostic reasoning, hardware-software boundary negotiation, and the subtle ways that configuration parameters interact with GPU memory in modern ML serving stacks.

The Message in Full

The subject message reads:

OOM during CUDA graph capture. The 0.95 mem fraction is over-allocating KV cache, leaving no room for CUDA graph buffers. Let me kill it and relaunch with a lower --mem-fraction-static (0.88 should be fine — still gives plenty of KV cache with 768GB total).

>

`` ssh 10.1.230.175 'pkill -9 -f "sglang" 2>/dev/null; sleep 3; pgrep -a sglang || echo "all killed"' ``

This is the moment of insight — the pivot point where a failed deployment is correctly diagnosed and a targeted fix is formulated. To understand why this message matters, we must reconstruct the chain of events that led to it.

The Deployment Context

The session had been building toward deploying the GLM-5-NVFP4 model — a massive Mixture-of-Experts architecture compressed to 4-bit NVFP4 quantization — on a machine equipped with eight RTX PRO 6000 Blackwell GPUs, each with 96 GB of VRAM. That gives a total of 768 GB of GPU memory. The model itself weighs approximately 400 GB in its quantized form, leaving a comfortable 368 GB of headroom on paper.

The assistant had already navigated a series of complex infrastructure challenges to reach this point. It had installed sglang from the main branch to obtain a critical SM120 shared memory fix (PR #14311) that corrects attention block sizes for Blackwell's 100K shared memory limit. It had upgraded Transformers to version 5.2.0 to support the novel glm_moe_dsa architecture. It had launched the server with tensor parallelism 8, FP4 quantization, and flashinfer attention backends.

The initial launch command used --mem-fraction-static 0.95, meaning 95% of available GPU memory would be reserved for the model weights and KV cache. This seemed reasonable: with 768 GB total and a 400 GB model, 0.95 × 768 = 729.6 GB would be allocated, leaving 38.4 GB across all GPUs (roughly 4.8 GB per GPU) for runtime overhead. The assistant's reasoning was that this aggressive allocation was safe because the model fit "very comfortably" — a phrase the user had used in message 149 when reporting the crash.

The Crash and Its Diagnosis

The model loaded successfully. Weights were distributed across all eight GPUs. Each GPU showed approximately 94 GB of memory used out of 96 GB. Then CUDA graph capture began — a process where the serving framework pre-compiles and caches CUDA kernel launches for common batch sizes to reduce latency during inference. The capture process requires temporary buffer allocations for graph inputs, outputs, and workspace.

And it crashed. The error, visible in the preceding message (msg 150), was a textbook CUDA out-of-memory (OOM) error:

torch.OutOfMemoryError: CUDA out of memory. Tried to allocate 694.00 MiB. GPU 4 has a total capacity of 94.97 GiB of which 247.44 MiB is free.

GPU 4 had only 247 MB free — less than 0.3% of its capacity. The CUDA graph capture needed 694 MB for a temporary buffer, and there simply wasn't room.

The Incorrect Assumption

The critical mistake was an assumption about memory allocation granularity. The assistant had reasoned: "The model is 400 GB, we have 768 GB, so we have 368 GB of headroom — 0.95 mem fraction is safe." This reasoning conflated static model weight allocation with dynamic runtime memory requirements.

The --mem-fraction-static parameter in sglang controls what fraction of GPU memory is reserved upfront for the KV cache and model weights. The remaining fraction is left for dynamic allocations: CUDA graph capture buffers, activation tensors during the capture process, temporary workspace for attention kernels, and other runtime overhead. Setting this to 0.95 left only 5% — approximately 4.8 GB per GPU — for all dynamic needs.

The assistant had overlooked that CUDA graph capture is a memory-intensive transient operation. During graph capture, the framework records CUDA kernel launches by running them, which requires allocating and freeing temporary buffers. If the static reservation consumes nearly all memory, the capture process has no room to breathe. The 694 MB allocation that triggered the OOM was modest by GPU standards, but it found only 247 MB free.

This is a classic systems failure pattern: a configuration that works at smaller scales or with different hardware fails catastrophically when pushed to the limit. The assistant had tuned for steady-state inference throughput (maximizing KV cache capacity) without accounting for the transient memory spike during initialization.

The Diagnostic Reasoning

The assistant's reasoning, visible in the message's concise analysis, demonstrates a clear mental model of the serving stack's memory architecture:

1. Observation: The crash occurred during CUDA graph capture, not during model loading or inference.
2. Correlation: The OOM error showed near-zero free memory on the affected GPU.
3. Causal inference: The high --mem-fraction-static value (0.95) had pre-allocated too much memory for KV cache, leaving insufficient headroom for the graph capture buffers.
4. Fix formulation: Reduce the fraction to 0.88, which frees 12% (instead of 5%) for dynamic needs — approximately 11.5 GB per GPU instead of 4.8 GB.
5. Validation check: 0.88 × 768 = 675.8 GB reserved, leaving 92.2 GB for runtime. The model needs ~400 GB, KV cache can use the remaining ~276 GB of the reservation, and the freed 92 GB provides ample room for graph capture and other transients. The choice of 0.88 is not arbitrary. It represents a calculated tradeoff: enough headroom to avoid the OOM during initialization while still reserving generous capacity for KV cache during inference. The assistant explicitly notes "still gives plenty of KV cache with 768GB total" — confirming that the fix preserves the primary goal of high-throughput serving.

Input Knowledge Required

To fully understand this message, a reader needs knowledge spanning several domains:

• GPU memory hierarchy: Understanding that VRAM is shared between static allocations (model weights, KV cache) and dynamic allocations (temporary buffers, CUDA graphs, activation memory).
• Serving framework internals: Knowing that --mem-fraction-static controls the split between reserved and dynamic memory, and that CUDA graph capture is a separate phase with its own memory requirements.
• Hardware specifics: The RTX PRO 6000 Blackwell GPUs have 96 GB each, and the SM120 architecture's shared memory constraints (100K vs 160K+ on Hopper) affect kernel behavior.
• Model characteristics: GLM-5-NVFP4 is a ~400 GB quantized MoE model, meaning its weight distribution across 8 GPUs is ~50 GB per GPU, leaving ~46 GB per GPU for KV cache and other allocations under normal conditions.
• CUDA graph mechanics: Understanding that graph capture involves running kernels to record their launches, which requires allocating workspace buffers that are freed after capture completes.

Output Knowledge Created

This message produces several valuable pieces of knowledge:

1. A documented failure mode: The interaction between --mem-fraction-static and CUDA graph capture on Blackwell GPUs is now explicitly identified. Future deployments can avoid this pitfall.
2. A calibrated parameter value: The shift from 0.95 to 0.88 provides a concrete reference point for similar configurations (8×96 GB GPUs, ~400 GB quantized model).
3. A diagnostic heuristic: When an OOM occurs during CUDA graph capture but not during model loading, the first suspect should be over-aggressive static memory reservation.
4. A process template: The assistant's kill-and-relaunch workflow (pkill -9 -f "sglang", sleep, verify) provides a reusable pattern for restarting failed serving processes.

The Broader Significance

This message exemplifies a recurring theme in the deployment of large language models on cutting-edge hardware: the gap between theoretical capacity and practical constraints. On paper, 768 GB of VRAM for a 400 GB model leaves 368 GB of headroom — nearly 50% spare capacity. Yet a single configuration parameter, set to what seemed like a conservative 95% reservation, caused a catastrophic failure during a routine initialization phase.

The lesson extends beyond this specific deployment. It highlights that GPU memory is not a fungible pool but a complex landscape of overlapping allocation phases. Model loading, KV cache reservation, CUDA graph capture, and inference each have distinct memory profiles. A configuration optimized for steady-state throughput may fail during initialization, and vice versa.

The assistant's response also demonstrates the value of understanding the order of operations in the serving stack. By recognizing that CUDA graph capture occurs after weight loading but before inference begins, the assistant correctly identified that the crash was a transient initialization issue rather than a fundamental incompatibility. The fix — reducing the static fraction — sacrifices a small amount of KV cache capacity (which was abundant anyway) to ensure the initialization phase completes successfully.

Conclusion

Message 151 captures a moment of diagnostic clarity in the midst of a complex deployment. In two sentences, the assistant identifies a subtle memory allocation bug, formulates a targeted fix, and prepares to relaunch. The message is brief, but it encapsulates the kind of systems-level reasoning that distinguishes successful deployments from frustrating failures: understanding not just what each configuration parameter does, but how it interacts with the temporal phases of the serving lifecycle. The 5% that was reserved for dynamic needs wasn't enough — and adjusting that boundary from 5% to 12% was the difference between a crashed server and a running model.