The Art of Memory Fractions: Relaunching GLM-5-NVFP4 After an OOM Crash

In the middle of a complex deployment session, a single message often carries the weight of everything that came before it. Message 153 in this opencode conversation is a deceptively simple shell command — a relaunch of the sglang inference server for the GLM-5-NVFP4 model across 8 Blackwell GPUs. But behind the one-liner lies a chain of reasoning that spans hardware constraints, framework internals, diagnostic detective work, and a crucial correction from the user. This message is the pivot point where a failed deployment gets its second chance.

The Crash That Preceded the Relaunch

To understand why this message was written, we must first understand what went wrong. The initial server launch (at <msg id=130>) used a --mem-fraction-static 0.95 parameter, which tells sglang to reserve 95% of available GPU memory for the KV cache and model weights. On paper, this seemed reasonable: the GLM-5-NVFP4 model is a quantized 4-bit Mixture-of-Experts architecture weighing approximately 400GB, and with 8× RTX PRO 6000 Blackwell GPUs providing 768GB of total VRAM, there appeared to be ample headroom.

The server loaded successfully. Weights were distributed across all 8 GPUs, each showing about 94GB utilized out of 96GB total. Then came the CUDA graph capture phase, where sglang precompiles optimized execution paths for common batch sizes. This is where it failed — with a torch.OutOfMemoryError on GPU 4, which had only 247 MiB free when the framework tried to allocate an additional 694 MiB for graph buffers.

The user's intervention at <msg id=149> was brief but crucial: "crashed; memory isn't really tight, the model fits very comfortably (nvfp4, 400G and we have >700g vram)." This correction reframed the problem. The issue wasn't insufficient total VRAM — it was that --mem-fraction-static 0.95 was consuming memory that CUDA graph capture needed for temporary buffers. The KV cache reservation was too aggressive, leaving no slack for the runtime's own allocations.

Diagnosing the Real Problem

The assistant's analysis at <msg id=150> and <msg id=151> shows the thinking process clearly. The OOM traceback revealed that PyTorch had allocated 87.43 GiB on the affected GPU, with another 6.44 GiB reserved but unallocated. The mem-fraction-static of 0.95 had caused sglang to pre-allocate KV cache memory so aggressively that the CUDA graph capture — which needs temporary workspace — had nothing left to work with.

The assistant correctly identified the fix: reduce --mem-fraction-static to a value that leaves breathing room for runtime allocations. The choice of 0.88 was not arbitrary. With 96GB per GPU, 0.88 reserves approximately 84.5 GiB for the model and KV cache, leaving about 11.5 GiB per GPU for CUDA graphs, activation buffers, and other runtime overhead. Across 8 GPUs, this still provides approximately 676 GiB for the model and cache — well above the 400GB model size — while ensuring the server won't choke during initialization.

The Message Itself: A Carefully Tuned Relaunch

The subject message at <msg id=153> issues the new launch command. Let us examine it in full:

nohup bash -c '\''
export PATH=$HOME/ml-env/bin:$PATH
export CUDA_HOME=/usr/local/cuda-12.8
export NCCL_IB_DISABLE=1
export NCCL_P2P_LEVEL=PHB
export NCCL_ALLOC_P2P_NET_LL_BUFFERS=1
export NCCL_MIN_NCHANNELS=8
export OMP_NUM_THREADS=8
export SAFETENSORS_FAST_GPU=1

python3 -m sglang.launch_server \
  --model lukealonso/GLM-5-NVFP4 \
  --served-model-name glm-5 \
  --reasoning-parser glm45 \
  --tool-call-parser glm47 \
  --trust-remote-code \
  --tp 8 \
  --mem-fraction-static 0.88 \
  --max-running-requests 64 \
  --kv-cache-dtype fp8_e4m3 \
  --quantization modelopt_fp4 \
  --attention-backend flashinfer \
  --moe-runner-backend flashinfer_cutlass \
  --disable-custom-all-reduce \
  --enable-flashinfer-allreduce-fusion \
  --host 0.0.0.0 \
  --port 8000
'\'' > ~/sglang-glm5.log 2>&1 &

The changes from the previous launch are subtle but meaningful. The --mem-fraction-static dropped from 0.95 to 0.88. The --max-running-requests increased from 8 to 64 — a decision made possible by the reduced memory reservation. With more memory left for KV cache slots after model weights are loaded, the server can now handle a larger concurrent request pool. Everything else — the tensor parallelism of 8, the FP4 quantization, the flashinfer attention backend, the NCCL environment variables — remains unchanged, reflecting the assistant's confidence that those choices were correct.

The Environment Variables: A Story in Themselves

The NCCL environment variables deserve attention. NCCL_IB_DISABLE=1 disables InfiniBand communication (this machine uses NVLink or PCIe for GPU-to-GPU transfers). NCCL_P2P_LEVEL=PHB sets the peer-to-peer communication level to "PHB" (PCIe Host Bridge), appropriate for a multi-GPU workstation without NVSwitch. NCCL_ALLOC_P2P_NET_LL_BUFFERS=1 enables low-latency buffer allocation for P2P communication, and NCCL_MIN_NCHANNELS=8 ensures enough communication channels for the 8-GPU tensor parallelism. These settings were refined in earlier segments of the conversation and represent accumulated knowledge about what works on this specific hardware configuration.

The SAFETENSORS_FAST_GPU=1 flag accelerates safetensors loading using GPU memory mapping, and OMP_NUM_THREADS=8 limits OpenMP parallelism to match the CPU core count available per process. Every variable in this preamble is the result of prior debugging and optimization.

Assumptions Embedded in the Launch

This message makes several assumptions worth examining. First, it assumes that 0.88 is the right memory fraction — high enough to maximize KV cache capacity, low enough to avoid OOM during graph capture. This is an educated guess, not a guaranteed value. The assistant has no way to know the exact memory requirements of CUDA graph capture without reading sglang's source code or running experiments. The value 0.88 is a heuristic, chosen because it leaves roughly 11.5 GiB of headroom per GPU, which should be sufficient for the graph capture buffers that previously needed only 694 MiB.

Second, the message assumes that the same attention backend (flashinfer) and MoE runner backend (flashinfer_cutlass) will work correctly on this hardware. Earlier in the conversation, the assistant had discovered that the DeepGemm backend was incompatible with the checkpoint's scale format (ue8m0), producing a warning about potential accuracy degradation on Blackwell. The flashinfer backends were chosen to avoid this issue, but the assumption is that they are numerically stable for this model.

Third, the message assumes that the server will complete its initialization without further intervention. The nohup wrapper and backgrounding (&) indicate that the assistant expects this to be a long-running process that should survive terminal disconnection. The log is redirected to ~/sglang-glm5.log, which the assistant will monitor in subsequent messages.

Input Knowledge Required

To understand this message fully, one needs knowledge of several domains. The sglang server's parameter semantics — what --mem-fraction-static controls, how --tp distributes layers across GPUs, what --kv-cache-dtype fp8_e4m3 means for memory efficiency — is essential. Understanding CUDA graph capture and why it needs temporary GPU memory is important for appreciating why the previous launch failed. Knowledge of NCCL communication modes (InfiniBand vs PCIe, P2P levels) explains the environment variable setup. Finally, familiarity with the GLM-5-NVFP4 model — its 400GB size, its glm_moe_dsa architecture, its requirement for Transformers ≥5.2.0 — provides the broader context for why this deployment is happening at all.

Output Knowledge Created

This message creates a running inference server that, if successful, serves the GLM-5-NVFP4 model on port 8000 with tensor parallelism across 8 GPUs. It also creates a log file that will document the server's initialization progress and any subsequent errors. For the assistant, this message is a checkpoint: if the server starts successfully, the conversation can move on to performance tuning and load testing. If it fails again, the log file will contain the evidence needed for the next diagnostic iteration.

The Thinking Process Visible in the Message

The reasoning in this message is mostly implicit — the actual deliberation happened in the preceding messages ([msg 150] and [msg 151]). But the parameter choices reveal the assistant's mental model. The reduction from 0.95 to 0.88 shows an understanding that memory fraction is not a "more is better" parameter; it must be balanced against the runtime's own allocation needs. The increase in --max-running-requests from 8 to 64 shows forward thinking: with more memory available after the fraction reduction, the server should be able to handle more concurrent requests, and setting this higher now avoids a restart later.

The decision to keep all other parameters identical is itself a reasoning artifact. The assistant judged that the OOM was solely caused by the memory fraction, not by the attention backend, quantization method, or NCCL configuration. Changing multiple parameters simultaneously would make it harder to isolate the cause of any future failure. This is good experimental practice: change one variable at a time.

Conclusion

Message 153 is a study in iterative debugging under real-world constraints. It is not a triumphant "it works" moment — it is a careful, measured second attempt informed by a crash, a user correction, and a diagnostic analysis. The message encapsulates the tension between maximizing hardware utilization and leaving room for the framework's own operational needs. The 0.88 memory fraction is a compromise between efficiency and stability, chosen with the understanding that the true optimal value would require empirical testing to find. In a deployment session full of complex build issues, CUDA version conflicts, and architecture-specific kernel fixes, this message stands out as a moment of calibration — adjusting a single number to turn a crash into a launch.