The Relaunch That Finally Worked: Deploying GLM-5-NVFP4 on Blackwell GPUs

Introduction

In the long arc of deploying a massive Mixture-of-Experts model across eight NVIDIA RTX PRO 6000 Blackwell GPUs, message [msg 635] represents a quiet turning point. It is not dramatic — no breakthroughs, no novel discoveries — but it is the moment when all the prior debugging, all the environment fixes, and all the compatibility patches finally converge into a single command that, this time, actually works. The message is a bash command that kills any lingering sglang processes and relaunches the inference server with an extensive set of flags and environment variables. By the time the next few messages roll around, the server will be serving requests and benchmarks will be running. This article examines why this particular message matters, what decisions it encodes, and what knowledge it presupposes.

The Immediate Context: Ninja-Build and FlashInfer

The message opens with "Good, ninja is installed. Now relaunch sglang." This terse acknowledgment belies the frustration of the preceding minutes. In [msg 633], the assistant discovered that the previous sglang launch attempt had crashed with a FileNotFoundError — the system was missing ninja, the build tool that FlashInfer requires for Just-In-Time (JIT) compilation of CUDA kernels. FlashInfer, a high-performance attention kernel library used by sglang, compiles specialized CUDA kernels on-the-fly at server startup. Without ninja-build installed, this JIT compilation step fails silently, causing the entire server to crash during initialization with a cryptic error trace.

The assistant's response in [msg 634] was to install ninja-build via apt-get install -y ninja-build. Message [msg 635] is the natural follow-up: with the missing dependency resolved, it is time to try again. The "Good, ninja is installed" is not just a status update — it is the culmination of a debugging chain that began with a server crash, led to reading the error log, identifying the missing build tool, installing it, and now retrying.

Anatomy of the Command

The bash command in [msg 635] is dense with meaning. It can be decomposed into three logical phases: cleanup, environment configuration, and server launch.

Phase 1: Cleanup

pkill -9 -f 'sglang' 2>/dev/null; sleep 2;

This forcefully terminates any residual sglang processes from the previous failed attempt. The -9 signal (SIGKILL) cannot be caught or ignored, ensuring that even stuck processes are removed. The 2>/dev/null suppresses error messages if no matching processes exist. The sleep 2 provides a brief window for the kernel to clean up resources — GPU memory allocations, network sockets, and shared memory segments — before the new instance starts. This is a pragmatic precaution: GPU processes sometimes leave CUDA contexts in an inconsistent state if killed abruptly, and a short delay reduces the risk of initialization errors.

Phase 2: Environment Variables

The command sets eight environment variables, each addressing a specific concern:

• PYTHONUNBUFFERED=1: Forces Python to flush stdout/stderr immediately rather than buffering output. This was a direct lesson from the previous failed launch (<msg id=622-623>), where the server appeared to hang because log output was buffered and not visible to the operator. With unbuffered output, any errors will appear in the log file in real time, enabling faster debugging.
• NCCL_IB_DISABLE=1: Disables InfiniBand support in NVIDIA Collective Communications Library (NCCL). The system has no InfiniBand hardware — the GPUs communicate over PCIe — so NCCL should not waste time probing for IB interfaces. This prevents potential initialization delays or errors.
• NCCL_P2P_LEVEL=5: Sets the NCCL peer-to-peer communication level. Level 5 corresponds to NVLINK or PXB (PCIe switch) topology, which is the expected communication path between the eight GPUs. This tells NCCL to use the most direct communication channel available.
• NCCL_MIN_NCHANNELS=8: Sets the minimum number of NCCL communication channels to 8. For tensor-parallel inference with 8 GPUs, having enough channels ensures efficient all-reduce operations during model execution. This is a performance tuning parameter.
• OMP_NUM_THREADS=8: Limits OpenMP threads to 8, preventing CPU thread oversubscription on the server's multi-core system.
• SAFETENSORS_FAST_GPU=1: Enables GPU-accelerated loading of safetensors checkpoint files. The GLM-5-NVFP4 model consists of 83 shard files totaling hundreds of gigabytes; loading them efficiently is critical. This flag tells the safetensors library to use CUDA for decompression and deserialization, significantly reducing load time.
• CUDA_HOME=/usr/local/cuda-12.8: Points to the CUDA 12.8 toolkit installation. This is essential because the system has multiple CUDA versions installed (12.8 and 13.1), and FlashInfer's JIT compilation needs to find the correct NVCC compiler and CUDA headers. The CUDA 12.8 toolkit was installed earlier in the session specifically to support flash-attn compilation ([chunk 0.0]).

Phase 3: The Server Launch

The sglang server command itself carries twenty parameters, each representing a design decision:

• --model lukealonso/GLM-5-NVFP4: The HuggingFace model identifier. The model is a 4-bit NVFP4 quantized version of GLM-5, a large MoE language model.
• --served-model-name glm-5: The name exposed via the OpenAI-compatible API endpoint.
• --reasoning-parser glm45 and --tool-call-parser glm47: Enable GLM-specific parsing for chain-of-thought reasoning and tool-calling capabilities.
• --trust-remote-code: Allows loading model code from the HuggingFace repository. This was a point of failure earlier (<msg id=604-608>) when transformers didn't recognize the glm_moe_dsa model type.
• --tp 8: Enables tensor parallelism across all 8 GPUs. This is the core distribution strategy for large models that cannot fit on a single GPU.
• --mem-fraction-static 0.92: Allocates 92% of available GPU memory for the model and KV cache. The remaining 8% is reserved for temporary allocations and CUDA graphs.
• --max-running-requests 64: Limits the number of concurrent in-flight requests. This prevents memory exhaustion under high load.
• --kv-cache-dtype auto: Lets sglang choose the optimal data type for the key-value cache based on the model's quantization format.
• --quantization modelopt_fp4: Specifies the 4-bit NVFP4 quantization format, which is the model's native format. This flag ensures the server loads the quantized weights correctly.
• --attention-backend flashinfer: Uses FlashInfer for attention computations. This was the backend that required ninja-build for JIT compilation.
• --fp8-gemm-backend cutlass: Uses NVIDIA's CUTLASS library for FP8 matrix multiplications.
• --nsa-decode-backend trtllm and --nsa-prefill-backend trtllm: Use TensorRT-LLM for the NSA (Native Sparse Attention) decode and prefill operations. This was a critical choice: earlier attempts with other NSA backends caused NaN crashes during decode ([chunk 1.0]).
• --moe-runner-backend flashinfer_cutlass: Uses FlashInfer with CUTLASS for the Mixture-of-Experts runner, which handles the expert routing and computation.
• --enable-flashinfer-allreduce-fusion: Enables a performance optimization that fuses the all-reduce communication with FlashInfer kernel execution, reducing latency.
• --host 0.0.0.0 --port 8000: Listens on all network interfaces on port 8000.

Assumptions Embedded in the Command

The command makes several assumptions, most of which are well-founded but some of which are worth examining:

1. The model weights are already cached locally. The command does not download the model; it assumes the HuggingFace cache at /root/.cache/huggingface/hub/ contains the 83 safetensors shards. This is correct — earlier messages confirmed the cache exists.
2. CUDA 12.8 is the correct toolkit for FlashInfer. The CUDA_HOME points to CUDA 12.8 rather than 13.1. This assumption is based on earlier debugging where flash-attn required CUDA 12.8 for compatibility. FlashInfer inherits this dependency.
3. The trtllm NSA backends will not crash. This assumption is grounded in the earlier debugging session ([chunk 1.0]) where the assistant systematically tested NSA backends and found that only trtllm avoided NaN crashes during decode.
4. The LXC container provides bare-metal GPU access. The assistant verified P2P topology at 53 GB/s same-NUMA ([chunk 5.0]), confirming that the LXC approach bypasses the VFIO/IOMMU limitations of the KVM VM.
5. All eight GPUs are homogeneous and functional. The command assumes all RTX PRO 6000 Blackwell GPUs have identical memory and compute capability. The earlier nvidia-smi checks confirmed this.

Input Knowledge Required

To understand this message fully, one needs knowledge of:

• The debugging history: That ninja-build was missing (msg 633-634), that CUDA initialization failed due to HMM (segment 5), that transformers needed upgrading to 5.2.0 (msg 610-612), and that NSA backends caused NaN crashes (segment 1-2).
• The hardware topology: That the system has 8 Blackwell GPUs in an LXC container on Proxmox, with PCIe-based communication rather than NVLink.
• The model architecture: That GLM-5-NVFP4 is a Mixture-of-Experts model requiring tensor parallelism, that it uses NVFP4 quantization, and that it has NSA (Native Sparse Attention) layers.
• SGLang's architecture: That the server uses a multi-process TP worker model, that FlashInfer requires JIT compilation, and that attention backends and MoE runners are configurable.
• NCCL configuration: That NCCL_P2P_LEVEL controls communication topology detection and that NCCL_MIN_NCHANNELS affects all-reduce bandwidth.

Output Knowledge Created

This message produces several forms of knowledge:

1. A working server launch command that can be reused or modified for future deployments. The exact combination of flags and environment variables represents a validated configuration for GLM-5-NVFP4 on Blackwell GPUs.
2. Confirmation that the dependency chain is complete. The fact that this launch succeeds (as seen in subsequent messages) proves that CUDA 12.8, transformers 5.2.0, ninja-build, and the sglang main branch are all mutually compatible.
3. A baseline for performance tuning. The command includes performance-related flags (--enable-flashinfer-allreduce-fusion, --max-running-requests 64, --mem-fraction-static 0.92) that can be adjusted based on benchmark results.
4. Documentation of the correct NSA backend selection. The use of --nsa-decode-backend trtllm and --nsa-prefill-backend trtllm encodes the finding that other NSA backends cause NaN crashes on Blackwell GPUs.

The Thinking Process

The reasoning visible in this message is primarily about consolidation and retry. The assistant has been through multiple cycles of debugging:

1. The server crashed → read the log → identified missing ninja-build
2. Installed the missing dependency
3. Now relaunches with the same parameters, but with two key improvements: unbuffered Python output (for better observability) and the confirmed-working NSA backend configuration The assistant does not change the core server parameters — the model, parallelism, quantization, and backend choices remain identical to the previous attempt ([msg 630]). This is intentional: the previous attempt failed not because the parameters were wrong, but because a system dependency was missing. The fix is minimal and targeted. The use of pkill -9 rather than a graceful shutdown reflects the assistant's experience with stuck GPU processes. Earlier, the assistant observed that sglang processes remained in futex_wait_queue state ([msg 628]) and could not be cleanly terminated. SIGKILL is the only reliable option.

Conclusion

Message [msg 635] is a moment of synthesis in a long debugging session. It brings together fixes for CUDA initialization, model type recognition, NSA backend selection, and build tool dependencies into a single launch command. The message itself is just a bash command, but the knowledge encoded in its flags and environment variables represents hours of iterative troubleshooting across multiple subsystems — from GPU drivers to Python package compatibility to CUDA kernel compilation. When the server finally starts serving requests in the following messages, it is this command that makes it possible.