The Second Launch: Deploying GLM-5-NVFP4 Across 8 Blackwell GPUs

Introduction

In the high-stakes world of deploying cutting-edge large language models, few moments are as charged as the second launch attempt. Message 130 in this opencode session captures precisely such a moment: the assistant issues a bash command via SSH to launch the GLM-5-NVFP4 model server across eight NVIDIA RTX PRO 6000 Blackwell GPUs using a freshly built main-branch version of SGLang. This message is not merely a command execution—it is the culmination of an intensive debugging session that resolved two distinct blocking issues, and it sets the stage for a cascade of numerical stability problems that would dominate the subsequent conversation. Understanding this single message requires unpacking the technical context, the reasoning that led to its specific parameter choices, and the assumptions embedded in every flag and environment variable.

The Path to This Message

To appreciate why message 130 was written, one must trace the journey that preceded it. The session began with environment setup on an Ubuntu 24.04 machine equipped with eight RTX PRO 6000 Blackwell GPUs (96 GB each). After installing NVIDIA drivers, CUDA Toolkit 13.1, and a Python virtual environment with PyTorch, the assistant attempted a first launch of the GLM-5-NVFP4 model in message 109. That attempt failed immediately with a KeyError: 'glm_moe_dsa'—the installed Transformers library (version 4.57.1) did not recognize the model architecture used by GLM-5, a state-of-the-art Mixture-of-Experts model quantized to NVFP4 format. The fix was straightforward: upgrade Transformers to version 5.2.0, which added support for the glm_moe_dsa architecture.

But a second, more subtle issue emerged. The user interjected in message 117 with a critical observation: the installed SGLang version (0.5.8.post1) lacked support for the SM120 compute capability of the Blackwell GPUs. A specific pull request, #14311, had been merged into SGLang's main branch three weeks earlier to fix shared memory block sizes for the RTX PRO 6000's architecture. Without this fix, attention kernels would use Hopper-sized block sizes that exceeded the Blackwell GPU's smaller shared memory capacity (100 KB versus 160+ KB on Hopper), causing crashes or silent correctness issues. The assistant verified this by inspecting the source code: the installed version only had branches for compute capability ≥9 (Hopper) and ≥8 (Ampere), with no special handling for capability 12 (Blackwell). The solution was to install SGLang from the main branch, which required cloning the repository and building from source.

Message 130 is therefore the second launch attempt, issued only after both blockers were resolved: Transformers 5.2.0 for the model architecture, and SGLang main branch (commit 632c7af) for the GPU architecture.

Anatomy of the Launch Command

The command itself is a study in deliberate configuration. Let us examine its components in detail.

Environment variables set the stage for distributed inference across eight GPUs:

• CUDA_HOME=/usr/local/cuda-12.8 — Points to the CUDA 12.8 toolkit installed earlier in the session, which provides the necessary compiler tools and libraries for kernel compilation.
• NCCL_IB_DISABLE=1 — Disables InfiniBand communication, appropriate since these GPUs are connected via NVLink/NVSwitch rather than IB.
• NCCL_P2P_LEVEL=PHB — Sets the peer-to-peer communication level to "PHB" (PCIe Host Bridge), a conservative setting that avoids more aggressive NVLink topologies that might cause instability.
• NCCL_ALLOC_P2P_NET_LL_BUFFERS=1 — Enables low-latency network buffers for P2P communication.
• NCCL_MIN_NCHANNELS=8 — Ensures at least 8 communication channels, matching the 8-GPU topology.
• OMP_NUM_THREADS=8 — Limits OpenMP threads to prevent oversubscription.
• SAFETENSORS_FAST_GPU=1 — Enables GPU-accelerated safetensor loading. These NCCL settings reflect hard-won knowledge from deploying large models on multi-GPU systems. The assistant assumes that disabling InfiniBand and setting conservative P2P levels will provide stable communication without the custom all-reduce optimizations that might be incompatible with the Blackwell architecture. The SGLang server arguments are equally deliberate:
• --model lukealonso/GLM-5-NVFP4 — The HuggingFace model identifier for the NVFP4-quantized GLM-5 checkpoint, approximately 250 GB in size.
• --served-model-name glm-5 — A friendly name for the OpenAI-compatible API endpoint.
• --reasoning-parser glm45 and --tool-call-parser glm47 — Enable the model's native reasoning and tool-calling capabilities, indicating this deployment targets agentic workloads.
• --trust-remote-code — Allows loading custom model code from the HuggingFace repository, necessary for the non-standard glm_moe_dsa architecture.
• --tp 8 — Tensor parallelism across all eight GPUs, splitting each tensor operation across the devices.
• --mem-fraction-static 0.95 — Reserves 95% of GPU memory for the model, an aggressive setting that assumes the model will fit comfortably within 96 GB × 8 = 768 GB total (the model is ~250 GB, leaving ample room for KV cache).
• --max-running-requests 8 — Limits concurrent requests to 8, matching the tensor parallelism degree.
• --kv-cache-dtype fp8_e4m3 — Uses FP8 (8-bit floating point) for the key-value cache, reducing memory pressure.
• --quantization modelopt_fp4 — Specifies the quantization format used by the model weights (NVFP4, a 4-bit floating point format from NVIDIA ModelOpt).
• --attention-backend flashinfer and --moe-runner-backend flashinfer_cutlass — Select the FlashInfer attention and MoE runner backends, which are optimized CUDA kernels for transformer inference.
• --disable-custom-all-reduce — Disables custom all-reduce kernels, falling back to NCCL's built-in implementation. This is a conservative choice that prioritizes stability over peak performance.
• --enable-flashinfer-allreduce-fusion — Enables fusion of all-reduce operations with FlashInfer kernels, a performance optimization that may reduce communication overhead.

Assumptions Embedded in the Configuration

Every parameter in this launch command encodes an assumption about the hardware, software, and model behavior. Some of these assumptions would prove incorrect.

The most significant assumption is that the FlashInfer attention backend and the modelopt_fp4 quantization would work correctly together on Blackwell GPUs. The model card on HuggingFace recommended this exact configuration, and the assistant trusted those recommendations. However, as the subsequent log output would reveal (message 131), a warning appeared immediately: "DeepGemm is enabled but the scale_fmt of checkpoint is not ue8m0. This might cause accuracy degradation on Blackwell." This warning, which the assistant noted but could not act on at this stage, foreshadowed the NaN crashes that would plague the server during decode operations.

A second assumption concerns the Transformers 5.2.0 compatibility. The log warned: "Transformers version 5.2.0 is used for model type glm_moe_dsa. If you experience issues related to RoPE parameters, they may..." This warning hinted at potential positional encoding incompatibilities between the new Transformers release and the custom model architecture. The assistant acknowledged this warning but proceeded, hoping the --trust-remote-code flag would handle any discrepancies.

The assistant also assumed that the main-branch SGLang build, with the SM120 shared memory fix, would resolve all GPU-related issues. While the fix was necessary for correct attention kernel execution, it did not address the DeepGemm scale format problem, which would later manifest as catastrophic NaN values in the probability tensor during decode.

Input Knowledge Required

Understanding this message requires substantial domain knowledge. The reader must be familiar with:

• CUDA compute capabilities and the distinction between Hopper (SM90, capability 9) and Blackwell (SM120, capability 12) architectures, particularly their different shared memory sizes.
• Tensor parallelism and how --tp 8 distributes model layers across GPUs, requiring careful NCCL configuration for efficient communication.
• Quantization formats like NVFP4 (4-bit floating point) and FP8 (8-bit floating point), and the concept of scale formats (ue8m0) used in DeepGemm kernels.
• SGLang's architecture, including its attention backends (FlashInfer, Triton), MoE runners, and the distinction between custom all-reduce and NCCL-based communication.
• The GLM model family, specifically the glm_moe_dsa architecture with its Mixture-of-Experts structure and custom RoPE implementation.
• The HuggingFace ecosystem, including model cards, trust_remote_code, and the Transformers library's configuration mapping.

Output Knowledge Created

This message creates several important outputs:

1. A baseline launch configuration that would be iteratively modified as debugging progressed. Every subsequent change to attention backends, KV cache dtype, or CUDA graph settings would be measured against this initial attempt.
2. A process (PID 4640) running on the remote machine, whose log output would become the primary debugging interface. The assistant would monitor this log across the next ten messages, watching the model download progress from 31 GB to 296 GB.
3. A set of warnings (DeepGemm scale format, Transformers RoPE compatibility) that would guide the subsequent debugging direction. These warnings, visible in the log but not in the launch command itself, represent the first hints of the NaN crash problem.
4. A documented decision point in the session's history. Future readers can see exactly which parameters were chosen, and why, before the stability issues emerged.

The Thinking Process Visible in This Message

While the message itself is a single bash command, the reasoning behind it is visible through the preceding context. The assistant's thinking process reveals a methodical approach to troubleshooting:

First, it identified the root cause of the first launch failure (missing glm_moe_dsa in Transformers) and applied the fix (upgrade to 5.2.0). Then, when the user flagged the SM120 issue, the assistant did not simply trust the user's assertion—it verified by inspecting the source code of the installed SGLang, confirming that the SM120 fix was absent. It then located the relevant PR, identified the merge commit, and built from the main branch. After installation, it verified the fix by re-inspecting the source code for the CUDA_CAPABILITY[0] == 12 branch.

This verification step is crucial. The assistant could have assumed the fix was present in the latest release (v0.5.8.post1), but instead it checked the actual code. This habit of verification—testing assumptions against reality—is a hallmark of effective debugging.

The assistant also demonstrated awareness of dependency interactions. When installing SGLang from source, the uv pip install -e command downgraded Transformers back to 4.57.1 (a dependency constraint in the SGLang build). The assistant caught this regression and re-upgraded Transformers, ensuring both fixes remained in place.

The Broader Context: What Followed

Message 130 is a pivot point. The server launched successfully, and the model began downloading. Over the next several messages, the assistant monitored the download progress (31 GB → 70 GB → 145 GB → 254 GB → 296 GB) and watched the weight loading process. But when the server began handling decode requests, the NaN crashes appeared—a device-side assert triggered error caused by infinite or NaN values in the probability tensor.

The subsequent debugging would consume much of the session. The assistant would try switching attention backends (Triton, FlashMLA sparse), forcing FP8 GEMM backends to Cutlass, disabling CUDA graphs, and consulting a local research repository (FINDINGS.md) that documented similar issues with Kimi K2-Thinking deployments. The DeepGemm scale format warning from message 130's log would prove to be the key clue, though the solution would require deeper investigation into Blackwell's numerical precision requirements.

Conclusion

Message 130 is far more than a simple bash command. It represents the convergence of two parallel debugging threads—Transformers version compatibility and SGLang GPU architecture support—into a single launch attempt. It encodes dozens of assumptions about hardware behavior, software compatibility, and model correctness. Some of these assumptions held; others did not. The warnings that appeared in the subsequent log output would become the roadmap for the next phase of debugging.

For the technical writer studying this session, message 130 serves as a case study in the complexity of deploying cutting-edge AI models. It illustrates that a successful launch is not the end of the journey but the beginning of a new phase of investigation, where theoretical compatibility meets the unforgiving reality of numerical computation on novel hardware architectures.