The First Launch: Deploying GLM-5-NVFP4 Across 8 Blackwell GPUs

Introduction

In the course of deploying a cutting-edge quantized Mixture-of-Experts (MoE) model on brand-new NVIDIA Blackwell hardware, there comes a moment of culmination: the first launch command. Message [msg 109] in this opencode session is that moment — a single bash command that attempts to bring together days of environment setup, driver installation, dependency resolution, and hardware verification into a running inference server. This article examines that message in depth: the reasoning behind it, the decisions embedded in its flags and environment variables, the assumptions it makes, and the knowledge it both consumes and produces.

The Context: What Led to This Moment

To understand message [msg 109], one must appreciate the journey that preceded it. The session began in [segment 0] with a bare Ubuntu 24.04 machine and the task of setting up a full ML environment. The assistant installed NVIDIA drivers (590.48.01), CUDA Toolkit 13.1, created a Python virtual environment using uv, and wrestled with flash-attn compilation issues — reducing MAX_JOBS from 128 to 20 to avoid memory exhaustion during builds, and later rebuilding flash-attn against the correct PyTorch version after vLLM had downgraded it. The environment was eventually stabilized with PyTorch 2.9.1, flash-attn 2.8.3, and vLLM 0.15.1.

Then came the hardware upgrade: the machine was expanded from 2 GPUs to 8 NVIDIA RTX PRO 6000 Blackwell Server Edition GPUs, each with 96 GB of VRAM — a total of 768 GB. The user's instruction was clear: deploy the GLM-5-NVFP4 model, a 744-billion-parameter MoE model quantized to FP4 precision, using SGLang, then tune parameters and allow parallel queries.

By message [msg 109], the assistant had:

The Message: A Launch Command in Full

The assistant's message consists of a brief reasoning preamble followed by a single bash command executed over SSH. Here is the message exactly as written:

[assistant] Good — no existing sglang processes, persistence mode enabled. Now let me launch the server. I'll use the exact command from the HuggingFace model card, running it with nohup so it persists.
[bash] ssh 10.1.230.175 'nohup bash -c '\''
export PATH=$HOME/ml-env/bin:$PATH
export CUDA_HOME=/usr/local/cuda-12.8
export NCCL_IB_DISABLE=1
export NCCL_P2P_LEVEL=PHB
export NCCL_ALLOC_P2P_NET_LL_BUFFERS=1
export NCCL_MIN_NCHANNELS=8
export OMP_NUM_THREADS=8
export SAFETENSORS_FAST_GPU=1

python3 -m sglang.launch_server \
  --model lukealonso/GLM-5-NVFP4 \
  --served-model-name glm-5 \
  --reasoning-parser glm45 \
  --tool-call-parser glm47 \
  --trust-remote-code \
  --tp 8 \
  --mem-fraction-static 0.95 \
  --max-running-requests 8 \
  --kv-cache-dtype fp8_e4m3 \
  --quantization modelopt_fp4 \
  --attention-backend flashinfer \
  --moe-runner-backend flashinfer_cutlass \
  --disable-custom-all-reduce \
  --enable-flashinfer-allreduce-fusion \
  --host 0.0.0.0 \
  --port 8000
'\'' > ~/sglang-glm5.log 2>&1 &
echo "PID: $!"'
PID: 3534

The command itself is a multi-line invocation wrapped in nohup bash -c '...' > ~/sglang-glm5.log 2>&1 &. It sets seven environment variables and then calls python3 -m sglang.launch_server with fifteen command-line flags.

Reasoning and Motivation

The assistant's reasoning is stated explicitly: "I'll use the exact command from the HuggingFace model card." This is a deliberate strategy of deference to authoritative documentation. The model card on HuggingFace — which the assistant had fetched and read in [msg 89] — is the canonical source for how to deploy this specific model. The model is unusual: it uses FP4 quantization (via NVIDIA's ModelOpt framework), targets the new Blackwell GPU architecture (SM120 compute capability), and has a custom architecture type (glm_moe_dsa). The model card's recommended launch command represents the combined knowledge of the model developers about what flags and settings are needed.

The decision to use nohup with backgrounding (&) and log redirection is practical: the server needs to persist beyond the SSH session. Without this, the server would terminate when the SSH connection closes. The log file (~/sglang-glm5.log) serves both as a record of startup and as a diagnostic tool for monitoring progress.

The Environment Variables: Decisions About Multi-GPU Communication

The seven environment variables reveal the assistant's understanding of the hardware topology and networking constraints:

The Launch Flags: Mapping the Model Card's Recommendations

The fifteen flags passed to sglang.launch_server each encode a specific decision:

Assumptions Embedded in the Command

The message makes several assumptions, some of which would prove incorrect:

  1. That SGLang 0.5.8.post1 supports Blackwell GPUs. The assistant had verified the SM120 fix was present in the source code ([msg 120]), but this was a false positive — the check only looked for the string "120" in the source, which matched unrelated code. The actual fix (PR #14311, merged Jan 30, 2026) was not present in 0.5.8.post1, as discovered in [msg 122].
  2. That the installed Transformers version (4.57.1) supports the glm_moe_dsa architecture. The assistant hadn't verified this yet. In fact, Transformers 5.2.0 was required, and the server would crash with a KeyError: 'glm_moe_dsa' in the very next message ([msg 110]).
  3. That the HuggingFace model card's command is complete and correct for this environment. While the model card is authoritative, it may not account for all environment-specific issues like the Transformers version or the exact SGLang build.
  4. That the model will auto-download without issues. The model is approximately 250 GB, and the download could fail due to network issues, disk space, or HuggingFace authentication.
  5. That the NCCL configuration is appropriate. The NCCL_P2P_LEVEL=PHB and NCCL_IB_DISABLE=1 settings assume a specific interconnect topology that may not match the actual hardware.

Input Knowledge Required

To understand this message, a reader needs knowledge of:

Output Knowledge Created

This message produces several artifacts:

  1. A running SGLang server process (PID 3534) on the remote machine, which will attempt to download and load the model.
  2. A log file at ~/sglang-glm5.log that records the server's startup progress and any errors.
  3. A documented launch configuration that can be reused, modified, or debugged. The exact command with all flags serves as a baseline for future tuning.
  4. A point of failure diagnosis: When the server crashes (as it does in [msg 110]), the log file provides the error traceback that reveals the Transformers version mismatch. This output knowledge feeds directly into the next iteration of debugging.

The Thinking Process Visible in the Reasoning

The assistant's reasoning in this message is concise but revealing. The phrase "I'll use the exact command from the HuggingFace model card" indicates a deliberate strategy: when deploying an unfamiliar model on new hardware, start with the vendor's recommended configuration. This is a conservative, principled approach that minimizes the risk of misconfiguration.

The decision to use nohup with backgrounding shows awareness of the SSH session's transient nature. The log file redirection shows anticipation of needing to debug the launch. The choice to include all environment variables in the same command (rather than setting them separately) shows an understanding that the bash subshell needs them in scope.

What's not visible in the reasoning is any verification that the installed SGLang version actually contains the SM120 fix. The assistant had checked earlier ([msg 120]) and concluded the fix was present, but that check was superficial. This oversight would be caught in the next round of debugging (<msg id=121-122>), leading to a rebuild from the main branch.

Conclusion

Message [msg 109] is a pivotal moment in the deployment of GLM-5-NVFP4 on 8 Blackwell GPUs. It represents the transition from environment setup to actual model serving — the point where all the preparation is put to the test. The command is carefully constructed, drawing on the HuggingFace model card's recommendations, NCCL best practices for multi-GPU communication, and SGLang's configuration options. Yet it also contains assumptions that would prove incorrect, specifically about the SGLang version's Blackwell support and the Transformers version's model architecture support.

These assumptions are not failures of reasoning but rather the natural gaps in knowledge when working at the cutting edge: a model released days ago, a GPU architecture months old, a serving framework evolving weekly. The message captures the moment when theory meets practice, and the debugging that follows is the process of closing those gaps. In this sense, message [msg 109] is less a final answer and more a hypothesis — a testable proposition about how to deploy this model, whose refutation generates the knowledge needed for the correct configuration.