The Moment of Truth: Launching GLM-5-NVFP4 on Bare-Metal Blackwell GPUs

In the arc of any complex infrastructure project, there comes a single moment when all the debugging, all the kernel parameters, all the module reloads, and all the compatibility fixes converge into one decisive action. For this coding session—a multi-day odyssey to deploy the GLM-5-NVFP4 large language model across eight NVIDIA RTX PRO 6000 Blackwell GPUs—that moment arrives at message 572. After resolving a cascade of blockers spanning CUDA initialization failures, flash-attn compilation memory exhaustion, NaN decode crashes, VFIO-imposed PCIe topology limitations, and a critical Heterogeneous Memory Management (HMM) incompatibility, the assistant finally issues the command to launch the sglang inference server. The message is deceptively simple—a single bash invocation with a long list of flags—but it represents the culmination of an extraordinary debugging journey and the beginning of a new phase: performance characterization and optimization.

The Long Road to This Point

To understand the weight of this message, one must appreciate the obstacles that preceded it. The session began in segment 0 with a clean Ubuntu 24.04 installation and the task of setting up a machine learning environment. What followed was a series of increasingly intricate hardware-software boundary problems. The flash-attn library repeatedly failed to compile because the build system defaulted to 128 parallel jobs on a machine with limited memory, requiring manual intervention with MAX_JOBS=20. The NVIDIA driver stack required a secondary CUDA 12.8 toolkit alongside CUDA 13.1 because different components demanded different versions. The vLLM installation downgraded PyTorch, breaking the carefully constructed environment.

Then came the model deployment itself. When sglang was first launched in a KVM virtual machine (segment 1), the server crashed with NaN values during decode—a catastrophic failure that took hours to diagnose. The root cause was ultimately traced to incompatible attention backends, and the solution required selecting specific NSA (Native Sparse Attention) backends (trtllm for both decode and prefill). Even after the server ran, performance was bottlenecked by the virtualized PCIe topology: the KVM VM, using VFIO passthrough, could only achieve a PHB (PCIe Host Bridge) topology, meaning GPU-to-GPU communication had to traverse host memory rather than using direct peer-to-peer (P2P) DMA.

The realization that virtualization was the bottleneck triggered a pivot to LXC containers (segment 4), which promised bare-metal GPU access. But this path was blocked by a cryptic CUDA initialization failure: cuInit() returned error code 3 (CUDA_ERROR_NOT_INITIALIZED). The breakthrough came when the assistant discovered a GitHub issue linking the problem to the NVIDIA open kernel module's Heterogeneous Memory Management (HMM) feature, which was incompatible with the Proxmox VE kernel. The fix—setting uvm_disable_hmm=1 as a module parameter for nvidia_uvm—immediately resolved the issue, and CUDA detected all eight GPUs.

The Decision: Adapting the Configuration for Real P2P

With CUDA working, the assistant performed a quick topology verification that confirmed the LXC container provided true bare-metal GPU access. The nvidia-smi topo -m output showed a NODE/SYS topology: GPUs 0–3 were connected via NODE (same PCIe switch) on NUMA node 0, and GPUs 4–7 similarly on NUMA node 1, with SYS links between the two groups. A bandwidth test measured same-NUMA P2P at 53.76 GB/s and cross-NUMA at 40.24 GB/s—a dramatic improvement over the VFIO-limited VM.

This new topology directly informed the assistant's configuration decision. The message's reasoning is explicit: "I'll use the same configuration that worked in the KVM VM but adapted for the better topology (we can try without --disable-custom-all-reduce since we now have real P2P)." This is a textbook example of adaptive system design—the assistant recognized that a flag that was necessary in the virtualized environment (disabling custom all-reduce because VFIO couldn't support efficient GPU-to-GPU communication) could now be removed, and a new flag (--enable-flashinfer-allreduce-fusion) could be added to leverage the hardware's true capabilities.

Anatomy of the Launch Command

The sglang launch command is a study in the complexity of modern LLM serving infrastructure. Every flag encodes a design decision shaped by the model's architecture, the hardware's capabilities, and the lessons learned from previous failures.

The model specification (--model lukealonso/GLM-5-NVFP4 --served-model-name glm-5) targets a specialized quantized variant of the GLM-5 architecture. The NVFP4 suffix indicates NVIDIA FP4 quantization, which requires the --quantization modelopt_fp4 flag—a non-standard quantization format that demands specific kernel support. The GLM-specific parsers (--reasoning-parser glm45 --tool-call-parser glm47) reflect the model's unique architecture, which includes built-in reasoning and tool-calling capabilities that require custom parsing logic.

The tensor parallelism setting (--tp 8) distributes the model across all eight GPUs, a necessity for a model of this size. The high memory fraction (--mem-fraction-static 0.92) aggressively reserves GPU memory for the KV cache, maximizing throughput at the cost of reduced headroom. The --max-running-requests 64 parameter sets an upper bound on concurrent request processing.

The attention and MoE backend selections are the most technically significant. The assistant chose --attention-backend flashinfer for the core attention mechanism, but specified --nsa-decode-backend trtllm and --nsa-prefill-backend trtllm for the Native Sparse Attention operations. This hybrid approach reflects the lesson learned from the NaN decode crash in segment 2: the NSA operations require TensorRT-LLM backends for numerical stability, while standard attention can use the faster FlashInfer implementation. The MoE runner (--moe-runner-backend flashinfer_cutlass) combines FlashInfer's fused kernels with CUTLASS for the Mixture-of-Experts layers that form the backbone of the GLM-5 architecture.

The inclusion of --enable-flashinfer-allreduce-fusion is the key adaptation to the new topology. In a P2P-capable environment, fused all-reduce operations can overlap communication and computation, significantly improving throughput. This flag would have been counterproductive in the VFIO VM, where all-reduce had to go through host memory.

Assumptions and Their Aftermath

The message makes several assumptions, some of which proved incorrect. The assistant assumed that the verified package stack (torch 2.9.1, flashinfer 0.6.3, sgl_kernel 0.3.21, transformers 4.57.1) was fully compatible. In reality, the server crashed shortly after launch due to a torchvision import error—the installed torchvision version was incompatible with torch 2.9.1. The subsequent fix (upgrading torchvision) triggered a cascade of unintended upgrades: uv resolved torch 2.10.0, which in turn broke sgl_kernel due to ABI incompatibility. The assistant had to roll back torch to 2.9.1 and carefully install torchvision 0.24.1.

A second assumption was that transformers 4.57.1 would support the GLM-5-NVFP4 model's architecture. This turned out to be false—the model uses a glm_moe_dsa model type that required transformers 5.2.0. This was discovered and fixed in subsequent messages.

These cascading failures are not signs of poor planning; they are the natural consequence of operating at the frontier of ML infrastructure. When deploying cutting-edge models with nightly-build serving frameworks on brand-new GPU architectures, compatibility must be discovered empirically. The assistant's methodical approach—verify, launch, observe, fix—is the only viable strategy.

The Significance

Message 572 is the pivot point of the entire session. Everything before it was preparation: driver installation, environment setup, build debugging, topology investigation, and blocker resolution. Everything after it is optimization: throughput benchmarking, parameter tuning, kernel selection, and performance analysis. The launch command itself encodes the accumulated knowledge of the entire session—every flag reflects a lesson learned, a bug fixed, or a hardware capability discovered.

The message also demonstrates a crucial engineering principle: infrastructure decisions must be revisited when the underlying environment changes. The assistant did not blindly copy the KVM configuration; it reasoned about how the new P2P topology should change the configuration. This adaptive mindset, combined with systematic debugging and a willingness to trace problems from Python stack traces through strace output to kernel module parameters, is what ultimately made the deployment possible.

In the end, the server launched successfully, and the benchmarks that followed achieved up to 806 tokens per second at 128 concurrent requests—a testament to the power of bare-metal GPU topology and the importance of persevering through the long, undramatic work of fixing one blocker after another until the system finally works.