From 3.4% Efficiency to Root Cause: A System-Level Optimization Odyssey on Blackwell GPUs

Introduction

In the high-stakes world of large language model inference optimization, few moments are as sobering as computing the theoretical maximum performance of your system and discovering that reality is a mere 3.4% of that number. This is precisely the situation the assistant and user found themselves in during Segment 10 of this opencode session. The GLM-5-NVFP4 model, a massive Mixture-of-Experts architecture deployed via SGLang across eight NVIDIA RTX PRO 6000 Blackwell GPUs, was achieving approximately 10.36 tokens per second in single-stream inference — against a theoretical ceiling of 309 tok/s. The gap was not a factor of two or three; it was a factor of thirty.

What followed was one of the most intensive system-level optimization campaigns in the entire session. The assistant launched a comprehensive parallel audit via ten subagents, uncovered a litany of misconfigurations, executed a major kernel upgrade from 6.8.12 to 6.14.11, survived a post-reboot CUDA crisis caused by stale device major numbers in the LXC container's cgroup configuration, and ultimately built diagnostic tools that identified the true bottleneck: FP4 GEMM kernel overhead on the Blackwell SM120 architecture. This article synthesizes that journey, tracing the arc from shock to diagnosis to the hard-won understanding that some bottlenecks cannot be fixed by system tuning alone.

The Shock of Theoretical vs. Actual Performance

The session's turning point came when the assistant computed the theoretical maximum single-stream throughput for the GLM-5-NVFP4 model. Using the GPU's FP4 tensor core throughput of 2,228 TFLOPS and the model's per-token compute requirements, the calculation yielded 309 tok/s. The actual measured performance was 10.36 tok/s — a staggering 3.4% efficiency [1]. This was not a minor tuning gap; it was a sign that something fundamental was wrong at the system level.

The assistant's response was immediate and methodical. Rather than chasing model-level optimizations, it launched a parallel system audit via ten subagents, each tasked with examining a different aspect of the system's configuration. The audit uncovered multiple critical misconfigurations: the CPU was running under the generic acpi-cpufreq governor instead of the AMD-specific amd_pstate driver; the kernel was an outdated 6.8.12; NUMA balancing was enabled, causing unnecessary cross-socket memory migrations; deep CPU C-states were adding wake-up latency; and the PCIe MaxReadReq was stuck at 512 bytes instead of the optimal 4096 [2].

The Kernel Upgrade: A Multi-Step Orchestration

The user's directive was unambiguous: "Try new kernel and the reboot-requiring fixes, feel free to reboot that proxmox node too" ([msg 1294]). This green-lit the most invasive change possible — a kernel upgrade requiring a full system reboot. The assistant's response was a meticulously orchestrated multi-step operation spanning messages 1295 through 1313.

The first step was reconnaissance. The assistant checked the current kernel (6.8.12-9-pve), surveyed available kernels via apt-cache search proxmox-kernel-6.14, and confirmed that the latest available version was 6.14.11-5-bpo12-pve [3][4]. It also verified the boot mechanism — Proxmox uses proxmox-boot-tool with systemd-boot on UEFI systems, with kernel cmdline stored in /etc/kernel/cmdline — ensuring that any boot-time parameters would be correctly applied [5].

The kernel installation itself revealed a critical dependency chain. When the assistant installed the proxmox-kernel-6.14.11-5-bpo12-pve-signed package, the DKMS (Dynamic Kernel Module Support) auto-build for the NVIDIA driver was skipped because kernel headers for the new kernel were not installed [6]. This was a near-miss: if the assistant had not noticed the warning and proceeded directly to reboot, the system would have booted without GPU drivers, rendering all eight GPUs inaccessible. The assistant immediately installed the headers package (proxmox-headers-6.14.11-5-bpo12-pve) and manually triggered the DKMS build and install for NVIDIA driver version 590.48.01 [7][8].

With the driver built, the assistant turned to persistent configuration. The kernel cmdline was updated to include amd_pstate=active, processor.max_cstate=1, and nmi_watchdog=0 [9]. Runtime sysctl parameters — NUMA balancing disabled, scheduler autogroup disabled, swappiness reduced, dirty ratios tightened, network buffers increased — were written to /etc/sysctl.d/99-gpu-compute.conf for persistence across reboots [10]. The PCIe MaxReadReq fix, which resets on driver reload, was encoded in a systemd oneshot service (gpu-pcie-tuning.service) that fires after nvidia-persistenced.service at boot [11].

The assistant then performed a series of verifications. It checked the kernel boot order via proxmox-boot-tool kernel list, confirming that the 6.14 kernel was listed first and would be the default [12]. It mounted the EFI partitions and inspected the actual boot entry files, confirming that the new cmdline had been correctly propagated [13][14]. It verified DKMS status one final time, listing the five NVIDIA kernel modules (nvidia.ko, nvidia-uvm.ko, nvidia-drm.ko, nvidia-modeset.ko, nvidia-peermem.ko) built for the new kernel [15]. And crucially, it regenerated the initramfs for the 6.14 kernel to include the uvm_disable_hmm=1 modprobe configuration — a critical workaround for a known CUDA initialization issue with the NVIDIA UVM driver's Heterogeneous Memory Management feature [16][17].

The Reboot and the Post-Reboot Crisis

The reboot command was issued in message 1313, accompanied by a summary of the six major changes about to take effect [18]. The assistant then waited — a 150-second polling loop that checked SSH connectivity every 10 seconds, with a 5-second connection timeout and a 10-minute maximum wait [19]. The host came back cleanly, and the verification in message 1316 confirmed that every tuning parameter had taken effect: kernel 6.14.11, amd-pstate-epp driver active, only C0 and C1 C-states available, all eight GPUs detected in P0 state, MaxReadReq set to 4096, and all sysctls applied [20][21].

But when the assistant turned to the LXC container that hosted the actual ML workload, disaster struck. The container had not auto-started — the assistant's polling loop in message 1318 ran for 300 seconds without success [22]. The user intervened with a three-word correction: "need to start it manually" ([msg 1319]). The assistant checked the container status via pct list, found VMID 129 in "stopped" state, and started it with pct start 129 [23][24][25].

With the container running, the assistant attempted to run a P2P bandwidth benchmark — and it crashed immediately with a CUDA initialization error [26]. The error was CUDA_ERROR_UNKNOWN (error code 999), a different and more opaque failure than the error 3 (CUDA_ERROR_NOT_INITIALIZED) that had been resolved earlier by disabling HMM [27]. The assistant's debugging process was a masterclass in systematic troubleshooting.

The first hypothesis — that the uvm_disable_hmm modprobe parameter had not applied on the new kernel — was tested and rejected when /sys/module/nvidia_uvm/parameters/uvm_disable_hmm returned Y [28]. The assistant then confirmed that nvidia-smi worked inside the container (ruling out a complete driver failure) but that PyTorch's torch.cuda.is_available() returned False [29][30]. A direct cuInit(0) call via the CUDA Python bindings confirmed the failure at the lowest driver API level [31].

The breakthrough came when the assistant compared the host's /proc/devices output against the LXC container's cgroup configuration. The kernel upgrade had changed the device major numbers for NVIDIA drivers: nvidia-uvm had moved from major 504 to 509, and nvidia-caps from 507 to 237 [32][33]. But the container's cgroup rules — stored in /etc/pve/lxc/129.conf — still referenced the old numbers [34][35]. The cgroup v2 device controller was silently blocking the container's access to the UVM and caps devices, causing cuInit to fail with a generic unknown error [36].

The fix was straightforward once the root cause was identified. The assistant stopped the container, edited the configuration to replace the stale cgroup rules with the correct major numbers (509 for nvidia-uvm, 237 and 238 for nvidia-caps), and restarted the container [37][38][39]. The verification in message 1341 confirmed success: torch.cuda.is_available() returned True, torch.cuda.device_count() returned 8, and a tensor could be created on cuda:0 [40].

The Diagnostic Turn: Building Tools to Find the Real Bottleneck

With CUDA restored and the system fully tuned, the assistant turned to the next phase of the investigation. The P2P benchmark was re-uploaded (the original had been lost when the container's /tmp was cleared during restart) [41] and executed. The results showed healthy P2P bandwidth of approximately 50 GB/s between same-NUMA GPUs, confirming that the PCIe and NVLink infrastructure was working correctly [42][43].

But the fundamental question remained: why was single-stream throughput stuck at 10 tok/s when the theoretical maximum was 309 tok/s? The assistant had eliminated system-level misconfigurations as the primary cause. The kernel was modern, the CPU governor was optimal, C-states were restricted, PCIe was tuned, and P2P bandwidth was healthy. The bottleneck had to be elsewhere.

The user redirected the investigation toward single and dual-stream performance, and the assistant built a diagnostic tool to measure the latency of individual decode components [44]. This tool decomposed the ~95ms per-token decode time into its constituent parts: simulated BF16 GEMMs, AllReduces, and the remaining overhead. The results were striking: the simulated BF16 GEMMs and AllReduces accounted for only 8.9ms of the 95ms decode time. The remaining ~86ms — over 90% of the latency — was attributable to FP4 GEMM kernel overhead, MoE routing, and attention [45].

This was the critical finding. The bottleneck was not in the system configuration, the CPU governor, the PCIe tuning, or any of the changes made during the kernel upgrade. It was in the FP4 GEMM kernels themselves, which were inefficient on the SM120 architecture of the Blackwell GPUs. The theoretical maximum of 309 tok/s was unattainable not because of system misconfiguration, but because the CUDA kernels for FP4 matrix multiplication were not optimized for this hardware generation.

Conclusion: The Value of Systematic Elimination

The journey documented in this chunk is a testament to the power of systematic elimination in performance optimization. The assistant started with a shocking 3.4% efficiency gap and methodically worked through every layer of the system: the kernel, the CPU governor, the C-states, the PCIe configuration, the NUMA topology, the container device passthrough, and the CUDA driver stack. Each layer was tuned, verified, and either confirmed as healthy or fixed. The kernel upgrade alone required installing headers, rebuilding DKMS modules, updating boot parameters, creating systemd services, regenerating initramfs, and debugging a post-reboot CUDA failure caused by stale cgroup device major numbers.

In the end, the system was in a pristine state — every tuning parameter applied, every device accessible, every benchmark passing. But the fundamental bottleneck remained in the FP4 GEMM kernels, a problem that no amount of system tuning could solve. This is the hard truth of ML infrastructure optimization: sometimes the bottleneck is not in the infrastructure at all, but in the software stack that runs on it. The assistant's systematic approach ensured that this conclusion was reached with confidence, not guesswork — a foundation for the next phase of the optimization journey.## References

[1] The Shock of Discovery: Computing Theoretical Maximum Performance — [chunk 10.0]

[2] System Audit and Misconfiguration Discovery — [chunk 10.0]

[3] The Reconnaissance Before the Leap: Surveying Kernel Options on a Proxmox ML Node — [1]

[4] The Reconnaissance Before a Kernel Upgrade: A Methodical Search for the Right Linux Kernel — [2]

[5] The Kernel Upgrade Decision: Bridging the Efficiency Gap Between 10 tok/s and 309 tok/s — [3]

[6] The Kernel That Almost Broke the GPUs: A Case Study in Assumptions About DKMS Auto-Build — [5]

[7] The Critical Middle Step: Installing Kernel Headers for DKMS After a Proxmox Kernel Upgrade — [6]

[8] The Critical Link: Building NVIDIA DKMS Modules for a Kernel Upgrade — [7]

[9] The Kernel Cmdline That Unlocked Performance: A Deep Dive Into Boot-Time Tuning for ML Inference — [8]

[10] The Sysctl That Anchored a Reboot: Persistent Kernel Tuning for Blackwell GPU Inference — [9]

[11] Locking in Performance: The Systemd Service That Makes GPU Tuning Survive Reboot — [10]

[12] The Kernel Boot Order Check: A Small Step with Big Implications — [11]

[13] The Critical Verification: Confirming Kernel Boot Order Before a High-Stakes Reboot — [12]

[14] The Final Verification: Trusting but Verifying a Kernel Upgrade for ML Infrastructure — [14]

[15] The Pre-Flight Check: Verifying a Kernel Upgrade Before the Critical Reboot — [15]

[16] The Final Verification Before Reboot: A Methodical Check That Saved the Kernel Upgrade — [16]

[17] The Final Pre-Reboot Step: Baking the HMM Fix into the Kernel Initramfs — [17]

[18] The Point of No Return: Rebooting a Production ML Server After Kernel Surgery — [18]

[19] The Waiting Game: A Pivotal 150 Seconds in ML Infrastructure Optimization — [20]

[20] The Moment of Truth: Validating a Kernel Upgrade for 8-GPU Inference — [21]

[21] The Reboot Verification: Confirming a System Transformation in the GLM-5 Optimization Journey — [22]

[22] The Art of Waiting: A Single Bash Loop Between Kernel Upgrade and Container Recovery — [23]

[23] The Pivot Point: How Three Words Rescued a Kernel Upgrade — [24]

[24] The Diagnostic Pivot: How a Single pct list Command Revealed the Assistant's Reasoning After a Kernel Upgrade — [25]

[25] The Quiet Container Start: A Single Command That Reveals Assumption, Correction, and Infrastructure Reality — [26]

[26] The Crash After Triumph: A P2P Benchmark Failure Exposes the Fragility of Kernel Upgrades in ML Infrastructure — [30]

[27] The Hypothesis That Failed: Debugging CUDA After a Kernel Upgrade — [31]

[28] The Diagnostic Pivot: Tracing a Post-Reboot CUDA Failure in LXC — [32]

[29] The Diagnostic Pivot: Tracing a CUDA Unknown Error After Kernel Upgrade — [33]

[30] The Diagnostic Pivot: Unraveling a Post-Kernel-Upgrade CUDA Failure — [34]

[31] The Moment of Diagnosis: Tracing a CUDA_ERROR_UNKNOWN Across a Kernel Upgrade — [35]

[32] The Missing Device Major Number: Debugging CUDA Initialization After a Kernel Upgrade — [38]

[33] A Single Grep That Uncovered a Kernel Upgrade Blind Spot: Debugging CUDA Initialization in LXC Containers — [39]

[34] The Silent Mismatch: How a Kernel Upgrade Broke CUDA in an LXC Container — [40]

[35] The Stale Major Number: Diagnosing a Post-Kernel-Upgrade CUDA Failure in LXC — [41]

[36] The Stale Major Number: A Post-Kernel-Upgrade CUDA Debugging Breakthrough — [42]

[37] Reading the LXC Config: A Diagnostic Pivot in a GPU Inference Optimization Session — [43]

[38] The Cgroup Repair: How a Kernel Upgrade Broke CUDA in an LXC Container — [44]

[39] The Quiet Culmination: A Single Command That Restores GPU Access After a Kernel Upgrade — [45]

[40] CUDA Verification After Cgroup Fix — [msg 1341]

[41] The Ephemeral Filesystem Problem: A P2P Benchmark Script's Journey Through a Kernel Upgrade — [29]

[42] The P2P Benchmark That Proved the Kernel Upgrade: Validating System Tuning After a CUDA Crisis — [48]

[43] The Moment of Assessment: Interpreting P2P Benchmarks After a Kernel Upgrade — [49]

[44] Building a Diagnostic Scalpel: The Decode Latency Breakdown Tool — [57]

[45] The 86ms Gap: Diagnosing FP4 GEMM Overhead in GLM-5 Inference on Blackwell — [60]