From Bare Metal to Blackwell: Deploying and Tuning GLM-5-NVFP4 Across Eight RTX PRO 6000 GPUs

Introduction

The deployment of large language models on cutting-edge GPU hardware is rarely a straight line from installation to inference. This article chronicles a single chunk of an opencode coding session that traverses the full lifecycle of ML infrastructure work: from installing NVIDIA drivers on a bare Ubuntu 24.04 system, through resolving nightmarish build issues with flash-attention, to deploying the GLM-5-NVFP4 model across eight RTX PRO 6000 Blackwell GPUs, and finally diagnosing virtualization-induced performance bottlenecks that constrain throughput. The work documented here is a testament to the depth and breadth of modern ML engineering — where success depends not just on knowing PyTorch APIs, but on understanding CUDA toolkit compatibility, shared memory limits of GPU architectures, distributed compilation strategies, and the intricacies of PCIe peer-to-peer communication in virtualized environments.

Phase 1: Building the Foundation — Drivers, CUDA, and Python Environment

The chunk opens with the most fundamental of infrastructure tasks: preparing an Ubuntu 24.04 machine for ML work. The assistant installs NVIDIA drivers version 590.48.01 alongside CUDA Toolkit 13.1, configures environment paths, and verifies that two RTX PRO 6000 Blackwell GPUs are operational. This is the digital equivalent of laying a foundation — unglamorous, but absolutely critical. Without the correct driver-kernel compatibility, nothing else works.

A Python virtual environment is created using uv, a fast Python package manager, and core packages including PyTorch are installed. This establishes a clean, isolated environment for the ML stack — a best practice that prevents the version conflicts that plague shared system Python installations.

Phase 2: The Flash-Attention Ordeal

The installation of flash-attn becomes the first major bottleneck, and it is here that the session reveals its true character. Building CUDA extensions from source is notoriously finicky, and this case is no exception. The system's CUDA 13.1 installation conflicts with PyTorch's CUDA 12.8 base, requiring the installation of a secondary CUDA 12.8 toolkit. This dual-CUDA setup is a common but painful workaround: PyTorch ships with its own CUDA dependencies, and when those differ from the system toolkit, source builds of CUDA extensions fail.

The build process then proceeds to exhaust system memory repeatedly. The assistant and user engage in a collaborative trial-and-error reduction of parallel compilation jobs (MAX_JOBS), dialing it down from 128 to 20 before the build succeeds. This is only possible after the machine is rebooted with an expanded 432GB of RAM — a reminder that building large CUDA projects is as much a hardware problem as a software one.

But the saga does not end there. When vLLM is installed, it downgrades PyTorch from the version against which flash-attn was compiled, breaking the binary. This forces a targeted rebuild of flash-attn against the correct PyTorch version (2.9.1). The final stabilized stack includes PyTorch 2.9.1, flash-attn 2.8.3, vLLM 0.15.1, and other ML tools — a carefully balanced set of versions that required multiple rounds of diagnosis and correction to achieve.

Phase 3: Scaling Up and Deploying GLM-5-NVFP4

With the environment stabilized, the conversation pivots sharply from infrastructure to application. The machine is upgraded from 2 to 8 GPUs, and the user tasks the assistant with deploying the GLM-5-NVFP4 model using a nightly build of SGLang. GLM-5-NVFP4 is a 256-expert Mixture-of-Experts (MoE) model using NVIDIA's NVFP4 quantization format, with a hidden size of 6144 and an intermediate size of 2048. At approximately 177GB in size, it requires careful distribution across the available GPU memory.

The deployment hits a critical snag: a NaN crash during decode. The assistant resolves this by selecting working NSA backends — --nsa-decode-backend trtllm and --nsa-prefill-backend trtllm — which produce coherent output on the SM120 GPUs. This fix is a direct consequence of the architecture-specific knowledge documented in the prior Kimi K2 deployment's FINDINGS.md (see [5]), which revealed that Triton-based MoE kernels require 147KB of shared memory while SM120 only provides 101KB, forcing the use of alternative backends.

Phase 4: Baseline Benchmarks and Server Tuning

With the model producing correct output, the assistant establishes baseline throughput metrics. Using 64 concurrent requests, the system achieves approximately 225 output tokens per second and 516 total tokens per second. These baselines serve as the reference point for all subsequent optimization work.

The assistant then experiments with server parameters. Increasing --mem-fraction-static to 0.92 allows the server to utilize more GPU memory for the KV cache. Enabling CUDA graphs captures successfully without out-of-memory errors. However, throughput remains stubbornly similar — ranging from 210 to 247 output tokens per second across different configurations. Various MoE runner backends are tested (flashinfer_cutlass, flashinfer_cutedsl), all producing comparable results. The flashinfer_trtllm path is found to be SM100-only and unavailable on the SM120 Blackwell architecture.

Phase 5: The MoE Tuning Investigation

Recognizing that MoE kernel configuration is one of the highest-leverage optimization points for transformer-based MoE models, the user initiates a subagent session to investigate whether previous MoE kernel tuning configurations from the Kimi K2 deployment could be reused for GLM-5-NVFP4 [1]. This investigation becomes a multi-round subagent conversation that exemplifies systematic engineering analysis.

The assistant begins with a reconnaissance phase, globbing the workspace to confirm all requested files exist and discovering an additional JSON config file the user had not explicitly mentioned [2]. It then reads all files in parallel — config files, tuning logs, and the comprehensive FINDINGS.md document [3]. A targeted grep of FINDINGS.md extracts 38 matches across the specified patterns, revealing the critical shared memory constraint that shapes all subsequent decisions [5].

The analysis delivers a definitive verdict: the existing configs cannot be reused directly [4]. The expert count differs (161 vs 256), the intermediate size per GPU differs (192 vs 256 for TP8), the quantization format differs (FP8 vs NVFP4), and SGLang's config lookup mechanism requires exact filename matches. However, the investigation identifies what is transferable: the SM120-specific wisdom about shared memory limits (num_stages ≤ 5, BLOCK_K=128-256), the tuning methodology itself (using tuning_fused_moe_triton.py with a 1920-configuration search space distributed via Ray), and the architectural insight that TP4 with pipeline parallelism may outperform TP8 for short generations on this hardware.

Phase 6: Expert Parallelism Analysis and Virtualization Diagnosis

The user, observing the PCIe-bound performance, raises the possibility of expert parallelism (EP) to improve throughput. The assistant performs a detailed memory analysis: the MoE experts alone require 453GB, while each GPU has only 96GB of VRAM. Full replication of experts across GPUs is impossible. Standard EP8 (distributing 256 experts across 8 GPUs, 32 experts per GPU) offers no advantage because the small hidden size (6144) results in communication volume similar to tensor parallelism, without any reduction in all-reduce overhead.

The user then pivots to a deeper question: is the cross-GPU latency from the Proxmox VM environment itself a bottleneck? Investigation confirms the system is a KVM/QEMU virtual machine with no direct GPU peer-to-peer support — the nvidia-smi topo -m output shows "NS" (not supported) for cross-GPU P2P, forcing all inter-GPU transfers through host memory via PCIe. A bandwidth test reveals the impact: approximately 32 GB/s for large transfers, but only about 1 GB/s for the small messages typical of all-reduce operations. This asymmetry — high bandwidth for large transfers, abysmal bandwidth for small messages — is characteristic of virtualization overhead and explains why the system is latency-limited despite having ample aggregate bandwidth.

Synthesis: The Full Stack View

What makes this chunk remarkable is the vertical跨度 of problems it addresses. In a single session, the work spans:

Lessons for ML Infrastructure Engineering

Several themes emerge from this chunk that generalize beyond the specific deployment:

1. Build issues are the hidden tax on CUDA development. The flash-attn ordeal — dual CUDA toolkits, memory exhaustion during compilation, version conflicts from dependency downgrades — is not unusual. It represents a significant but invisible cost of working with cutting-edge ML models that require custom CUDA extensions.

2. Hardware constraints propagate upward. The SM120's 101KB shared memory limit shapes backend selection, which affects throughput, which influences whether expert parallelism is worth pursuing. A constraint at the GPU architecture level ripples through every subsequent decision.

3. Virtualization adds a latency tax that tuning cannot fix. The Proxmox VM environment's lack of GPU P2P support means that no amount of kernel tuning can overcome the ~1 GB/s small-message bandwidth. This is a fundamental infrastructure limitation that must be addressed at the hypervisor or hardware level.

4. Prior work is a goldmine — if you know how to mine it. The MoE tuning investigation [1][2][3][4][5] demonstrates the value of methodically analyzing previous deployment artifacts. The configs themselves were not reusable, but the methodology, the hardware constraints, and the tuning patterns transferred directly.

5. Negative results are valuable. The conclusion that EP offers no advantage, that virtualization is the bottleneck, and that the existing MoE configs cannot be reused — these are all negative findings that save future effort by ruling out unproductive paths.

Conclusion

This chunk of the opencode session captures the full arc of modern ML infrastructure work: from bare-metal setup through deployment, tuning, and diagnosis. The assistant navigates CUDA toolkit conflicts, memory-exhausting builds, NaN crashes, shared memory limits, expert parallelism tradeoffs, and virtualization overhead — each challenge requiring a different kind of expertise. The MoE tuning investigation ([1]-[5]) stands as a model of systematic knowledge transfer, while the virtualization diagnosis reveals the hard infrastructure constraints that ultimately bound what tuning can achieve. For anyone deploying large language models on cutting-edge hardware, this session is a case study in the breadth of skills required and the importance of methodical, data-driven problem solving.## References

[1] "Decoding the MoE Tuning Investigation: A Deep Dive Into Cross-Model Kernel Configuration Analysis" — Analyzes the user's initial request to investigate MoE kernel tuning configs from previous Kimi K2 deployments, examining the strategic context, assumptions, and analytical framework of the investigation.

[2] "The Reconnaissance Phase: How an AI Assistant Systematically Discovers MoE Kernel Tuning Configs" — Documents the assistant's first response in the subagent session, where it runs glob operations to verify file existence and map the workspace structure before reading any contents.

[3] "The Data-Gathering Pivot: Parallel File Reads in an MoE Kernel Tuning Investigation" — Covers the assistant's parallel read of all requested files, transitioning from exploration to data collection.

[4] "The MoE Config Autopsy: How One Message Determined Whether Previous GPU Kernel Tuning Could Serve a New Model" — Presents the comprehensive analysis concluding that the existing configs cannot be reused for GLM-5-NVFP4 due to dimensional mismatches, while identifying transferable methodology and SM120-specific wisdom.

[5] "The Methodical Searcher: How a Single Grep Command Revealed Blackwell SM120's MoE Kernel Constraints" — Examines the grep of FINDINGS.md that extracted the critical finding about SM120's 101KB shared memory limit versus the 147KB required by Triton kernels.