From Bare Metal to Blackwell: The Full-Stack Journey of Deploying and Optimizing GLM-5-NVFP4 on Eight RTX PRO 6000 GPUs

Introduction

The gap between acquiring cutting-edge hardware and achieving peak inference performance on a frontier-scale language model is vast — filled with driver incompatibilities, build system failures, kernel immaturity, and architectural constraints that no amount of brute force can overcome. This article chronicles a single, intensive opencode coding session that traversed this entire gap: starting from a bare Ubuntu 24.04 installation on a machine with two NVIDIA RTX PRO 6000 Blackwell GPUs, and culminating in a deeply analyzed, systematically optimized deployment of the 744-billion-parameter GLM-5-NVFP4 mixture-of-experts (MoE) model across eight such GPUs.

The work documented in this chunk represents a full-stack odyssey — from installing NVIDIA drivers and wrestling with flash-attention build failures, through deploying the model with SGLang, benchmarking its performance, launching a multi-agent research campaign into alternative model formats, tuning server parameters for a 28% throughput improvement, and finally synthesizing everything into a ranked optimization plan with documented improvement strategies. Each phase of this journey reveals distinct challenges and insights about the current state of large-scale ML inference on Blackwell hardware.

Part I: Building the Foundation — Environment Setup on Ubuntu 24.04

The session began with a clean slate: a remote Ubuntu 24.04 machine equipped with two RTX PRO 6000 Blackwell GPUs. The first task was to establish a complete ML development environment from scratch — a process that, in theory, should be straightforward but in practice revealed the ragged edge of NVIDIA's software ecosystem.

Driver and CUDA Installation

The assistant installed the latest NVIDIA drivers (version 590.48.01) and CUDA Toolkit 13.1, configuring environment paths and verifying that both GPUs were operational. This phase went smoothly, establishing a solid foundation. The RTX PRO 6000 Blackwell GPUs were confirmed as SM120 architecture devices — the consumer/professional variant of Blackwell, distinct from the datacenter SM100 architecture found in B200/GB200 GPUs. This distinction would prove critical later, as SM120 has different instruction sets and kernel requirements than its datacenter counterpart.

Python Environment and PyTorch

Using uv — a fast Python package manager — the assistant created a virtual environment and installed PyTorch along with core dependencies. This phase was straightforward, establishing a clean workspace for the ML stack.

The Flash-Attention Ordeal

The installation of flash-attn became the session's first major bottleneck, consuming multiple rounds of trial and error. The core problem was a mismatch between the system's CUDA 13.1 toolkit and PyTorch's CUDA 12.8 base — flash-attention needed to be compiled against the same CUDA version that PyTorch used. This required installing a secondary CUDA 12.8 toolkit alongside the primary 13.1 installation.

But the CUDA version mismatch was only the beginning. The build process repeatedly exhausted system memory, crashing with out-of-memory errors. The machine had 128 parallel compilation jobs running simultaneously (MAX_JOBS=128), each consuming significant memory. The assistant progressively reduced this to 20 jobs, but even that wasn't enough until the machine was rebooted with an expanded 432GB of RAM.

Further complications arose when vLLM was installed and downgraded PyTorch, breaking the compiled flash-attn binary. This required a targeted rebuild of flash-attn against the correct PyTorch version (2.9.1). The final stable stack was PyTorch 2.9.1, flash-attn 2.8.3, and vLLM 0.15.1 — a carefully balanced combination that required significant debugging to achieve.

This ordeal illustrates a fundamental reality of modern ML infrastructure: the dependency graph is fragile, and even simple installations can become multi-hour debugging sessions when version mismatches compound. The assistant's methodical approach — isolating each failure, adjusting parameters, and rebuilding — mirrors how an experienced systems engineer would handle such challenges.

Part II: Deployment and Performance Analysis

With the environment stabilized, the assistant deployed the GLM-5-NVFP4 model using SGLang. The model — a 744-billion-parameter MoE architecture with 256 experts and top-8 routing — was loaded across the available GPUs. Initial benchmarks were conducted to establish baseline performance.

The Compute-Bound Discovery

A critical early finding came from benchmarking the tensor parallelism configuration. The assistant compared TP4+PP2 (4-way tensor parallelism with 2-way pipeline parallelism) against TP8 (8-way tensor parallelism). The result was striking: TP4+PP2 was 2× slower than TP8, definitively ruling out allreduce communication latency as the primary bottleneck. The model was compute-bound, not communication-bound.

This finding was counterintuitive. With eight GPUs connected via NVLink, one might expect communication overhead to dominate. But the data showed that the bottleneck was in the compute kernels themselves — specifically, the FP4 GEMM operations on SM120 tensor cores were not achieving their theoretical throughput.

Deep Kernel Analysis

The assistant then conducted a deep investigation into FP4 GEMM kernel efficiency on SM120. The findings were sobering:

Part III: The Alternative Formats Research Campaign

Recognizing that the current NVFP4 format might not be optimal for SM120 hardware, the user commissioned a subagent to conduct a systematic investigation of alternative model formats, quantization schemes, and inference engines. The research brief posed seven specific questions, each targeting a different potential escape route from the bandwidth-bound regime [1].

The Seven Lines of Inquiry

The subagent launched multiple rounds of parallel web searches, iteratively refining queries based on initial results [2][3][4][5][6][7]. The seven questions were:

  1. FP8 (W8A8) quantization: Could FP8, despite reading 2× more data per weight, actually be faster for small batches because it reaches the compute-bound regime sooner?
  2. GGUF/llama.cpp: Did a GGUF format exist for GLM-5, and could llama.cpp's consumer-GPU-optimized kernels outperform SGLang?
  3. AWQ/GPTQ/Marlin: Were there weight-only quantized versions of GLM-5 using these established formats?
  4. TensorRT-LLM: Did NVIDIA's flagship inference engine support GLM-5 with optimized SM120 kernels?
  5. Fewer experts: Was there a GLM-5 variant with fewer than 256 experts, allowing larger per-expert batch sizes?
  6. W4A16 mixed precision: Could FP4 weights be combined with BF16 activations via cuBLASLt's W4A16 mode for better-tuned kernels?
  7. ExLlama/Aphrodite: Did alternative inference engines have specific SM120 MoE optimizations?

The Synthesis: What Was Found

The subagent's comprehensive findings report (message 6) delivered a nuanced verdict [4]. The most critical insight was the roofline analysis of FP8 versus FP4: during decode with tiny per-expert batch sizes, the operations are entirely memory-bandwidth-bound regardless of precision. FP8 would make things worse by requiring 2× more bytes to be read per GEMM. FP4 was confirmed as the correct choice for decode.

For llama.cpp, the subagent found that GGUF quantizations existed but evaluated the engine as unsuitable for the multi-GPU setup — llama.cpp's layer-based splitting cannot match the expert-parallel or tensor-parallel schemes that SGLang provides. No AWQ, GPTQ, or Marlin quantized versions existed. TensorRT-LLM did not support GLM-5, and its SM120 support was buggy even for supported models. No variant with fewer experts existed — the 256-expert configuration is baked into the architecture. The W4A16 path was already in play as a fallback in vLLM when native NVFP4 kernels aren't available. ExLlamaV3 had not yet produced GLM-5 quants (the model was only eight days old).

The subagent's final recommendation was to stay with SGLang, which achieved 1,262 TFLOPS on Blackwell FP4 MoE workloads versus vLLM's 1,117 — and to focus on batch-size strategies, speculative decoding, and expert offloading rather than format changes.

Part IV: Performance Tuning and the 28% Breakthrough

With the format research confirming the NVFP4 path, the assistant turned to systematic server parameter tuning. Two key parameters were adjusted:

Part V: The Optimization Roadmap

The assistant then launched multiple research subagents to explore specific optimization approaches:

Conclusion: Lessons from the Full-Stack Journey

This chunk of work offers several enduring lessons for anyone deploying large MoE models on Blackwell hardware:

The dependency stack is fragile. A seemingly simple environment setup can consume hours when CUDA versions, PyTorch builds, and flash-attention compilation interact in unexpected ways. The assistant's methodical debugging — isolating each failure, adjusting parameters, rebuilding — is a template for handling such situations.

Roofline analysis trumps intuition. The FP8-versus-FP4 question seemed counterintuitive, but roofline analysis definitively showed that for small-batch MoE decode, memory bandwidth dominates and FP4 is optimal. The same analytical framework can guide many other optimization decisions.

Architecture is destiny. The 256-expert configuration of GLM-5 fundamentally constrains per-expert batch sizes. No quantization scheme or inference engine can change this. The real solutions involve working within this constraint — larger batches, speculative decoding, expert offloading.

Systematic research saves time. The alternative formats campaign could have been a weeks-long distraction. Instead, the subagent's structured investigation produced a clear verdict in minutes: stay with NVFP4 and SGLang, focus on tuning and optimization. This is the value of rigorous research methodology.

Small tuning gains compound. The 28% throughput improvement from parameter tuning demonstrates that significant gains are available without changing the model format or inference engine. Sometimes the best optimization is simply using the existing tools more effectively.

The journey from bare metal to optimized deployment — through driver installation, build debugging, kernel analysis, format research, parameter tuning, and optimization planning — captures the full complexity of modern ML systems engineering. Each layer reveals new constraints and opportunities, and navigating them requires both deep technical knowledge and systematic methodology.## References

[1] "The Research Mandate: Probing Alternative Paths for GLM-5 on Blackwell SM120 Hardware" — Analyzes the user's seven-question research brief that launched the alternative formats investigation.

[2] "The Deepening Search: Following Up on CUTLASS W4A16 and llama.cpp for GLM-5 on Blackwell" — Examines the assistant's targeted follow-up searches for CUTLASS mixed-precision support and llama.cpp performance benchmarks.

[3] "The Third Wave: How a Subagent's Final Search Round Reveals the Anatomy of Systematic ML Research" — Documents the final round of searches covering TensorRT-LLM support, Aphrodite integration, and Blackwell microbenchmarking.

[4] "The Architecture Detective: How an AI Assistant Researched GLM-5 Alternative Formats for Blackwell GPUs" — The comprehensive synthesis report evaluating all seven alternative format options with roofline analysis and actionable recommendations.

[5] "The First Move in a Research Campaign: Systematic Web Exploration for GLM-5 Alternative Formats" — Covers the initial wave of seven parallel searches launching the alternative formats research campaign.

[6] "The Art of the Follow-Up: How a Single Round of Parallel Searches Revealed the GLM-5 Inference Landscape on Blackwell" — Analyzes the second wave of eight targeted follow-up searches that deepened the investigation.

[7] "The Parallel Research Engine: How an AI Assistant Systematically Investigates GLM-5 Alternatives for Blackwell SM120 Hardware" — Examines the assistant's systematic, parallel search methodology across multiple research dimensions.