The One-Line Fix That Unblocked Eight Blackwell GPUs

In the high-stakes world of deploying large language models on cutting-edge hardware, the smallest configuration detail can become an immovable obstacle. Message 560 of this opencode session captures one such moment: a single command that resolved a days-long debugging saga, transforming eight NVIDIA RTX PRO 6000 Blackwell GPUs from expensive paperweights into a functioning inference cluster. The message is deceptively brief — a bash command that writes one line to a configuration file — but the reasoning behind it represents a masterclass in systematic troubleshooting, open-source research, and understanding the deep plumbing of GPU computing.

The Context: A Debugging Ordeal

To understand why this message was written, we must appreciate the wall the assistant had been hitting. The session's broader goal was to deploy the GLM-5-NVFP4 model using SGLang on a Proxmox VE host with eight Blackwell GPUs. After successfully setting up an LXC container to bypass VFIO/IOMMU limitations that prevented peer-to-peer (P2P) GPU communication in the KVM VM ([msg 545]), the assistant encountered a devastating blocker: CUDA would not initialize.

The symptom was maddeningly inconsistent. Earlier attempts showed cuInit() returning error code 3 (CUDA_ERROR_NOT_INITIALIZED), but later tests showed it hanging indefinitely — the process would be killed by a timeout signal without ever returning ([msg 546]). This wasn't a simple driver mismatch or missing library. It was a fundamental failure in the GPU initialization pathway.

The assistant's investigation ([msg 547] through [msg 556]) was thorough and methodical. They checked GSP firmware loading, verified that the gsp_ga10x.bin firmware file actually contained Blackwell ELF sections (gb20x, gb20y), confirmed that nvidia-smi could communicate with the GPUs, and ruled out file permission issues. They ran strace to trace the ioctl calls. They examined the MIG minors configuration and the driver parameters. Each test eliminated one hypothesis and narrowed the search space.

The crucial clue came from nvidia-smi -q, which showed Addressing Mode: HMM. This indicated the NVIDIA open kernel module was using Heterogeneous Memory Management, a feature that allows the GPU to access system memory through the kernel's HMM infrastructure. The assistant recognized this as a potential point of failure on a non-standard kernel like Proxmox VE's 6.8.12-9.

The Eureka Moment: Connecting to a Known Bug

Message 560 is the culmination of a research chain that began with a web search in [msg 557]. The assistant searched for "NVIDIA cuInit error 3 Blackwell Proxmox LXC open kernel module 590" and found a direct hit: GitHub issue #947 in the NVIDIA open-gpu-kernel-modules repository, titled "cuInit() returns CUDA_ERROR_NOT_INITIALIZED (3) with open kernel module; proprietary module works." This was the exact same bug.

The assistant then followed the trail to the parent issue, #797 ([msg 558]), which documented the root cause: the HMM feature in the nvidia_uvm kernel module was incompatible with certain kernel configurations. The workaround, as documented in the issue thread, was to set the module parameter uvm_disable_hmm=1 for the nvidia_uvm driver.

This discovery in [msg 559] is where the reasoning crystallized. The assistant wrote: "This makes perfect sense — the host shows Addressing Mode: HMM in nvidia-smi, meaning the driver is using Heterogeneous Memory Management which requires specific kernel support that may not be properly functional on the PVE kernel 6.8.12-9." This connection between the observed symptom (HMM enabled) and the known bug (HMM incompatibility with non-standard kernels) is the critical insight that drives message 560.

The Message Itself: A Single Line of Configuration

Message 560 reads:

[assistant] Found the fix. The issue is a known bug with the NVIDIA open kernel module's HMM (Heterogeneous Memory Management) feature. The workaround is to set `uvm_disable_hmm=1` as a module parameter for `nvidia_uvm`. Let me apply this now.
[bash] ssh root@10.1.2.6 "echo 'options nvidia_uvm uvm_disable_hmm=1' > /etc/modprobe.d/nvidia-uvm-hmm.conf && cat /etc/modprobe.d/nvidia-uvm-hmm.conf"
options nvidia_uvm uvm_disable_hmm=1

The command does two things. First, it creates a file /etc/modprobe.d/nvidia-uvm-hmm.conf containing the line options nvidia_uvm uvm_disable_hmm=1. This file is read by the system's module loading infrastructure — specifically, by modprobe and the udev-based module loading system — and passes the parameter to the nvidia_uvm kernel module when it is loaded. Second, it cats the file to confirm the content was written correctly.

The parameter uvm_disable_hmm=1 tells the NVIDIA Unified Virtual Memory (UVM) driver to disable its Heterogeneous Memory Management path. Instead of relying on the kernel's HMM subsystem for managing GPU-accessible system memory, the driver falls back to an alternative mechanism (likely using its own page table management). This bypasses the kernel incompatibility that was causing cuInit() to fail.

Why This Fix Was Non-Obvious

Several factors made this fix difficult to discover:

The symptom was misleading. Error code 3 (CUDA_ERROR_NOT_INITIALIZED) is a generic "something went wrong during initialization" error. It doesn't point to HMM specifically. The hang behavior (process killed by timeout) was even more confusing — it suggested a deadlock or infinite loop rather than a clean error return.

The diagnostic tools worked. nvidia-smi functioned perfectly, reporting driver version 590.48.01, CUDA version 13.1, and all eight GPUs. This made it seem like the driver was healthy. The failure only manifested in the CUDA runtime library (libcuda.so.1) when calling cuInit(), which is a higher-level initialization that involves the UVM module.

The fix contradicts intuition. Disabling a feature called "Heterogeneous Memory Management" sounds like a step backward. HMM is designed to improve performance by allowing seamless memory sharing between CPU and GPU. Disabling it feels like accepting a suboptimal configuration. In reality, the feature was broken on this kernel, and disabling it restored functionality without apparent performance regression.

The fix is kernel-module-specific. It targets nvidia_uvm, not the main nvidia driver. The nvidia module itself was working (as evidenced by nvidia-smi), but the UVM component — which handles unified memory allocation and GPU page faults — was failing during its initialization sequence.

Assumptions and Their Validity

The assistant made several assumptions in this message, all of which proved correct:

  1. The bug is indeed the HMM issue from GitHub #797. This assumption was validated by the symptom match (error code 3 with open kernel module) and the configuration clue (Addressing Mode: HMM). The fix worked, confirming the diagnosis.
  2. The fix requires a reboot or module reload. The modprobe.d configuration only takes effect when the module is loaded. The assistant implicitly assumes this is acceptable — the system can be rebooted or the module can be reloaded. In subsequent messages, we see the assistant rebooting the LXC container (or the host) to apply the change.
  3. Disabling HMM won't break other functionality. The assistant assumes that the NVIDIA driver's non-HMM path is fully functional on this hardware. This is a reasonable assumption given that the proprietary NVIDIA driver (which doesn't use HMM) works on the same hardware.
  4. The configuration file location is correct. /etc/modprobe.d/ is the standard location for module parameters on Linux. The naming convention nvidia-uvm-hmm.conf follows the pattern of using descriptive filenames.

Input Knowledge Required

To fully understand this message, a reader needs knowledge in several domains:

Linux kernel module system: Understanding how modprobe.d configuration files work, how module parameters are passed, and when they take effect (on module load). The options directive in modprobe configuration specifies parameters to pass to a kernel module.

NVIDIA GPU driver architecture: The distinction between the nvidia kernel module (core driver, handles GPU management, nvidia-smi communication) and the nvidia_uvm module (handles unified virtual memory, GPU page faults, memory migration). cuInit() touches both, but the UVM path is where HMM lives.

CUDA initialization sequence: cuInit() is the first CUDA API call an application must make. It initializes the driver, establishes communication with the GPU, and sets up memory management. Failure at this stage blocks all CUDA compute.

Heterogeneous Memory Management (HMM): A Linux kernel feature that allows device drivers to manage memory that is accessible from both the CPU and a device (like a GPU). It's part of the kernel's broader heterogeneous memory support, including dev_pagemap and migrate_to_ram infrastructure.

Proxmox VE kernel specifics: The PVE kernel is based on the Ubuntu kernel but includes additional patches for virtualization. It may lag behind upstream in supporting newer kernel features like HMM improvements needed by the NVIDIA 590 driver.

Output Knowledge Created

This message created several valuable pieces of knowledge:

A working configuration fix: The file /etc/modprobe.d/nvidia-uvm-hmm.conf persists across reboots, ensuring that every time the nvidia_uvm module loads, HMM is disabled. This is a permanent workaround, not a temporary hack.

A confirmed diagnostic pathway: The message validates the connection between "Addressing Mode: HMM" in nvidia-smi and cuInit() failure with error code 3. Future troubleshooters can use this correlation to diagnose similar issues.

A documented workaround for a known bug: By connecting the local issue to GitHub #797, the message effectively applies upstream community knowledge to a specific deployment context (Proxmox VE + Blackwell GPUs + NVIDIA open kernel module 590).

A foundation for further progress: The fix unblocks the entire deployment pipeline. Subsequent messages show the assistant launching the SGLang inference server, running benchmarks, and achieving 806 tok/s at 128 concurrency. None of that would have been possible without this one-line configuration change.

The Broader Significance

Message 560 exemplifies a pattern that recurs throughout system administration and ML infrastructure work: the most impactful interventions are often the simplest, but they rest on a foundation of deep diagnostic work. The assistant didn't guess the fix — they earned it through:

  1. Systematic hypothesis testing (checking GSP firmware, strace, kernel parameters)
  2. Cross-referencing symptoms with known issues (searching GitHub issues with precise keywords)
  3. Connecting configuration state to bug reports (noticing "Addressing Mode: HMM" and linking it to the HMM bug)
  4. Applying community knowledge (the uvm_disable_hmm=1 workaround from NVIDIA's issue tracker) The message also highlights the fragility of cutting-edge ML infrastructure. Blackwell GPUs (compute capability 12.0), NVIDIA driver 590.48.01, CUDA 13.1, and Proxmox VE kernel 6.8.12-9 represent the bleeding edge of all four components. When they interact unexpectedly, there's no established playbook — the troubleshooter must build one from first principles and community reports. In the end, the fix is elegant: one line of text that tells the kernel module to stop using a feature it can't handle. The GPUs don't need HMM to function; they need CUDA to initialize, and CUDA needs the UVM module to cooperate. By disabling the incompatible feature, everything else falls into place. It's a reminder that in complex systems, the path forward sometimes means knowing which features to turn off, not which ones to turn on.