Chunk 2.0
In this chunk, the assistant successfully resolved the critical NaN crash during decode by using `--nsa-decode-backend trtllm` and `--nsa-prefill-backend trtllm`, which produced coherent output on the SM120 GPUs. Baseline benchmarks with 64 concurrent requests showed ~225 output tok/s and ~516 total tok/s. The assistant then tuned the server by increasing `--mem-fraction-static` to 0.92 and enabling CUDA graphs, which captured successfully without OOM, though throughput remained similar (~210–247 output tok/s). Various MoE runner backends (`flashinfer_cutlass`, `flashinfer_cutedsl`) were tested with comparable results, and the `flashinfer_trtllm` path was found to be SM100-only. The user raised the possibility of expert parallelism to improve PCIe-bound performance. The assistant analyzed memory constraints (453GB of MoE experts vs. 96GB/GPU) and concluded full replication was impossible; standard EP8 offered no advantage due to the small hidden size and similar communication volume. The user then questioned whether cross-GPU latency from the Proxmox VM environment was a bottleneck. Investigation confirmed the system is a KVM/QEMU VM with no direct GPU peer-to-peer support (NS status), forcing all cross-GPU transfers through host memory. A bandwidth test showed ~32 GB/s for large transfers but only ~1 GB/s for small messages typical of all-reduce, indicating that virtualization overhead significantly contributes to the latency-limited throughput. The main achievements include deploying GLM-5-NVFP4 with a working configuration, establishing baseline throughput metrics, and identifying virtualization-induced P2P limitations as a key performance constraint. Themes center on overcoming SM120-specific issues (NSA backend selection), tuning for throughput on PCIe-only Blackwell GPUs, and diagnosing infrastructure bottlenecks in a virtualized environment.
Deploying GLM-5-NVFP4 on Blackwell GPUs: From NaN Crashes to Virtualization Bottlenecks
Message Articles
- The Moment of Reckoning: A Systematic Debugging Chronicle for GLM-5-NVFP4 on Blackwell SM120 GPUs
- The Green Light: How a Two-Sentence User Message Delegated Critical Decision-Making in an AI-Assisted ML Deployment
- The Moment Before Breakthrough: Checking State in a High-Stakes ML Deployment
- The Moment of Truth: Testing trtllm NSA Backends for GLM-5-NVFP4 on Blackwell SM120
- The Breakthrough: How `trtllm` NSA Backends Rescued GLM-5-NVFP4 on SM120 Blackwell GPUs
- The Breakthrough: How `trtllm` NSA Backends Rescued GLM-5-NVFP4 on SM120 Blackwell GPUs
- The Breakthrough: GLM-5-NVFP4 Comes Alive on SM120 Blackwell
- The Breakthrough: How `trtllm` NSA Backends Unlocked GLM-5-NVFP4 on Blackwell SM120 GPUs
- The Calm Before the Benchmark: A Methodical Transition from Debugging to Performance Tuning
- From Crisis to Calibration: The Pivot to Performance Benchmarking in Message 220
- The First Benchmark That Never Ran: A Case Study in Tool Assumptions
- The Tokenizer Trap: Debugging Benchmark Tooling for a Reasoning Model on Blackwell GPUs
- The Benchmark That Broke: Diagnosing Reasoning Model Compatibility in SGLang's bench_serving
- Reading the Signal Through the Noise: Benchmarking GLM-5-NVFP4 After the NaN Crisis
- Establishing Baseline Throughput: The Moment GLM-5-NVFP4 Proved Stable Under Concurrent Load
- Benchmarking a 744B MoE on Blackwell GPUs: The Methodical Pursuit of Throughput
- Benchmarking GLM-5-NVFP4 on 8 Blackwell GPUs: Finding the Throughput Ceiling
- The Saturation Point: Benchmarking GLM-5-NVFP4 on Eight Blackwell GPUs
- The Tuning Crossroads: Balancing Risk and Reward in Blackwell Inference Optimization
- The Pivot Point: How a Todo List Update Captures the Transition from Benchmarking to Tuning in a Large-Scale ML Deployment
- The Pivot Point: Killing the Server to Chase 1k Tok/s
- The Graceful Shutdown That Wasn't: A Study in Process Management During ML Infrastructure Tuning
- The Clean Slate: Why a Simple Verification Command Matters in ML Infrastructure
- The Turning Point: Enabling CUDA Graphs for GLM-5-NVFP4 on Blackwell GPUs
- The Weight-Loading Check: A Pivotal Moment in Server Tuning
- The Moment of Truth: Waiting for CUDA Graphs on 8 Blackwell GPUs
- The CUDA Graph Milestone: Validating Inference Stability on SM120 with GLM-5-NVFP4
- The Moment of Validation: CUDA Graphs Go Live on GLM-5-NVFP4
- The Counterintuitive Optimization: When CUDA Graphs Make Things Worse
- The Plateau: Recognizing the PCIe Bottleneck in Blackwell MoE Inference
- The Single-Stream Latency Test: Understanding the Performance Ceiling of GLM-5-NVFP4 on Blackwell GPUs
- The 11 Tok/s Wall: Diagnosing PCIe-Bound MoE Throughput on 8× Blackwell GPUs
- The Pivot Question: When a User Asks "Would Expert-Parallel Be Faster?"
- Expert Parallel vs Tensor Parallel: A Critical Decision for PCIe-Bound MoE Inference
- The Pivot Point: Evaluating Expert Parallelism for PCIe-Bound Blackwell Inference
- Investigating Expert Parallelism for Blackwell GPUs on PCIe: A Diagnostic Deep Dive
- When Intuition Meets Memory Physics: A User's Expert Replication Proposal for GLM-5 on 8 Blackwell GPUs
- The Expert Replication Gambit: Analyzing a Memory-Bound Parallelism Decision for GLM-5-NVFP4 on Virtualized Blackwell GPUs
- Reading the Blueprint: How One Bash Command Unlocked the Architecture of GLM-5-NVFP4
- The Memory Wall: Why You Cannot Replicate 453 GB of MoE Experts Across 8 GPUs
- The Communication Calculus: Why Expert Parallelism Fails on PCIe-Bound Blackwell GPUs
- The Latency Mirage: When Expert Parallelism Meets PCIe Reality on Blackwell GPUs
- The Moment of Acceptance: When a Creative Idea Meets Reality
- The Moment of Acceptance: When a Brilliant Idea Meets Cold Math
- The Pivot: Exploring Server-Side Knobs After Ruling Out Expert Parallelism
- The Turning Point: Accepting the PCIe Bottleneck
- The Turning Point: When a Debugging Session Realized PCIe Wasn't the Bottleneck
- Diagnosing the PCIe Bottleneck: A Deep Dive into GPU Topology Verification
- The Pivot Point: Investigating PCIe Link Configuration in a Virtualized ML Deployment
- The Pivot Point: How a Single `nvidia-smi` Command Unraveled the PCIe Bottleneck Myth
- The Pivot: When PCIe Stopped Being the Bottleneck
- The Power of Three Words: Deconstructing "fix script" in an Expert Debugging Session
- The Minimal Fix: A Single Bash Command That Unlocks Performance Debugging
- The Profiling Loop That Wasn't: A Microcosm of Debugging Blackwell Inference Performance
- The Power of "Nope": A Single-Word Correction That Reshaped a Debugging Session
- The Debugging Tactic That Uncovered a Bottleneck: A Single Failed `nvidia-smi` Query
- The 100% Revelation: How a Single nvidia-smi Command Reshaped the Debugging of GLM-5-NVFP4 on 8 Blackwell GPUs
- The Pivot: When 100% GPU Utilization Rewrites the Bottleneck Story
- The 55% Power Budget Revelation: How Small Matrix Operations Became the Real Bottleneck in Blackwell Inference
- The Cryptic Pivot: How a Three-Word Message Redirected a GPU Inference Debugging Session
- Reading the Artifacts: An Assistant's Investigative Pivot in a GPU Deployment Session
- Delegating Analysis: The Strategic Use of Subagents in MoE Configuration Investigation
- The MoE Kernel Tuning Hypothesis: A Pivot in Debugging GLM-5-NVFP4 Throughput
- The Flashinfer Revelation: When Prior Art Doesn't Apply
- The Pivot Point: From Reusing Configs to Understanding FP4 MoE Code Paths
- The MoE Kernel Hunt: Diagnosing GPU Inefficiency in FP4 Decode
- The Art of the Kernel Swap: Why One Bash Command Encapsulates an Hour of GPU Debugging
- The MoE Runner Gambit: Probing FP4 Kernel Efficiency on Blackwell GPUs
- The Flashinfer TRTLLM Crash: A Diagnostic Pivot on SM120 GPUs
- The Kernel Tuning Imperative: Why a New GPU Demands Custom Configs
- The SM100-Only Trap: Diagnosing MoE Backend Failures on Consumer Blackwell GPUs
- Probing the Kernel Frontier: Investigating Autotune Mechanisms for Blackwell SM120 MoE Execution
- Discovering SM120-Specific CUTLASS MoE Kernels: A Turning Point in Blackwell GPU Optimization
- Unearthing the Autotuner: A Pivotal Discovery in Kernel Optimization for Blackwell GPUs
- The Missing Config: Discovering FlashInfer's Autotuner Gap for Blackwell RTX PRO 6000
- The Missing Config: How a Single File Explained 45% of GPU Performance
- Unlocking Blackwell Performance: The Hunt for MoE Kernel Tuning Configs in FlashInfer
- The Autotune Investigation: Peering into FlashInfer's Kernel Tuning Pipeline
- Peering Inside the Autotuner: The Quest for SM120 MoE Kernel Configs
- The Autotuner That Didn't Tune: Diagnosing Runtime MoE Kernel Optimization on Blackwell GPUs
- The Autotuning Mirage: How a Single Log Confusion Redirected a GPU Kernel Optimization Effort
- The Autotuner's Missing Config: Debugging MoE Kernel Performance on a New Blackwell GPU
- The Autotuner's Silence: Diagnosing Kernel Configuration on Blackwell GPUs
- The Signal in the Noise: Debugging Autotuner Configuration for Blackwell GPUs
- The Art of the Targeted Grep: Debugging Log Verbosity in LLM Server Tuning
- The Autotuner That Wasn't: A Pivotal Discovery in GPU Kernel Tuning
- The Autotuner Mirage: A Diagnostic Pivot in Blackwell MoE Inference
- The Last MoE Backend: Systematic Kernel Exploration on Blackwell GPUs
- The Final Backend: Testing `flashinfer_cutedsl` on Blackwell SM120 GPUs
- Testing the CuteDSL MoE Backend: A Moment of Uncertainty in the GLM-5-NVFP4 Deployment
- The Validation Moment: Confirming Correct Model Output After Debugging NaN Crashes on Blackwell GPUs