The Moment Everything Unblocked: Disabling HMM to Fix CUDA on Blackwell GPUs

In the middle of a grueling multi-session debugging marathon to deploy the GLM-5-NVFP4 large language model across 8 NVIDIA RTX PRO 6000 Blackwell GPUs, a single message marks the turning point. Message <msg id=562> is deceptively simple — a one-line bash command executed over SSH that removes and reloads a kernel module. But this message represents the culmination of hours of diagnostic work, the resolution of a critical blocker, and the moment the entire deployment pipeline unblocked. Understanding why this message was written and what it accomplished requires tracing the reasoning chain that led to it.

The Message Itself

The assistant executed:

ssh root@10.1.2.6 "rmmod nvidia_uvm && modprobe nvidia_uvm && lsmod | grep nvidia_uvm && cat /sys/module/nvidia_uvm/parameters/uvm_disable_hmm 2>/dev/null || echo 'param not found'"

The output confirmed success:

nvidia_uvm           2043904  0
nvidia              14753792  2 nvidia_uvm,nvidia_modeset
Y

The Y output from reading uvm_disable_hmm is the critical signal: it confirms that the Heterogeneous Memory Management (HMM) feature of the NVIDIA Unified Virtual Memory (UVM) driver has been disabled. This single parameter change was the key to making CUDA work on this Proxmox VE host with Blackwell GPUs.

The Reasoning Chain: From Hang to Fix

To understand why this message was written, one must follow the assistant's diagnostic journey through the preceding messages. The problem had manifested as a complete inability to initialize CUDA — cuInit() either hung indefinitely or returned error code 3 (CUDA_ERROR_NOT_INITIALIZED). This was a show-stopper: without CUDA initialization, no GPU compute of any kind was possible.

The assistant's investigation went through several hypotheses:

Hypothesis 1: Missing GSP firmware. The NVIDIA open kernel module requires GPU System Processor (GSP) firmware to initialize Blackwell GPUs. The assistant checked /lib/firmware/nvidia/590.48.01/ and found only gsp_ga10x.bin and gsp_tu10x.bin — files named for Ampere and Turing architectures. A search for Blackwell-specific firmware (gsp_gb*) returned nothing. This seemed plausible until the assistant ran strings on the gsp_ga10x.bin file and discovered it contained Blackwell ELF sections (kernel_gb20y.elf, kernel_gb20x.elf) embedded inside. The "ga10x" filename was merely a container label; the firmware actually supported Blackwell. Furthermore, nvidia-smi -q confirmed the GSP firmware was loaded and running at version 590.48.01. Hypothesis discarded.

Hypothesis 2: Kernel version incompatibility. The Proxmox VE host was running kernel 6.8.12-9, and the assistant considered upgrading to 6.8.12-18. But before pursuing this time-consuming path, the assistant dug deeper.

Hypothesis 3: HMM incompatibility (the correct one). While examining nvidia-smi output, the assistant noticed Addressing Mode: HMM — indicating the driver was using Heterogeneous Memory Management. A web search for "cuInit error 3 Blackwell open kernel module" led directly to NVIDIA GitHub issue #947, which was a duplicate of issue #797. The resolution documented there was clear: setting uvm_disable_hmm=1 on the nvidia_uvm module parameter works around the CUDA initialization failure.

The Decision to Apply the Fix

The assistant made a critical judgment call at this point. Rather than pursuing the kernel upgrade (which would have taken significant time and might not have fixed the issue), the assistant recognized that the HMM workaround was both faster and more likely to succeed. The reasoning was sound:

1. The GitHub issue was a direct match — same symptom, same driver version, same open kernel module.
2. The fix was low-risk — disabling HMM only affects UVM's ability to use heterogeneous memory, which is not needed for standard GPU compute workloads.
3. The fix was reversible — simply reloading the module without the parameter restores the default behavior.
4. The fix was immediate — no reboot or kernel rebuild required. Message <msg id=560> shows the assistant creating the configuration file /etc/modprobe.d/nvidia-uvm-hmm.conf with the line options nvidia_uvm uvm_disable_hmm=1. Then in <msg id=561>, the assistant stopped the LXC container that might be holding GPU device references and checked the loaded module state.

The Execution in Message 562

The command in the subject message performs four operations in a carefully orchestrated sequence:

1. rmmod nvidia_uvm: Removes the currently loaded nvidia_uvm kernel module. This is necessary because module parameters can only be set at load time — they cannot be changed on a running module. The && ensures the chain stops if removal fails.
2. modprobe nvidia_uvm: Loads the module fresh. Since the configuration file /etc/modprobe.d/nvidia-uvm-hmm.conf exists, modprobe automatically passes uvm_disable_hmm=1 as a parameter. The module initializes with HMM disabled.
3. lsmod | grep nvidia_uvm: Verifies the module is loaded and shows its memory footprint (2043904 bytes) and reference count. The output also shows nvidia module with a reference count of 2, indicating nvidia_uvm and nvidia_modeset are both depending on it — confirming the dependency chain is intact.
4. cat /sys/module/nvidia_uvm/parameters/uvm_disable_hmm: Reads the actual parameter value from the kernel's module parameter interface. The output Y confirms that HMM is disabled. The fallback || echo 'param not found' handles the case where the parameter doesn't exist (e.g., if the module version doesn't support it). The output Y is the moment of validation — it proves the parameter was accepted and is active.

Assumptions Made and Corrected

Several assumptions were made during this debugging process, some of which turned out to be incorrect:

Incorrect assumption: Missing firmware. The assistant initially assumed the absence of Blackwell-named firmware files meant the GPUs couldn't initialize. This was a reasonable assumption given the naming convention, but it was wrong — the firmware was bundled inside a container file. The assistant corrected this by running strings on the binary and checking nvidia-smi for GSP status.

Incorrect assumption: cuInit hang vs error. Earlier, cuInit() appeared to hang indefinitely (exit code 124 from a timeout). Later, it returned error code 3 quickly. The assistant correctly recognized this might be a timing-dependent behavior or a race condition in the driver initialization path, and focused on the error code 3 as the more actionable signal.

Correct assumption: HMM is the culprit. The assistant correctly connected the Addressing Mode: HMM indicator to the known GitHub issue. This was the key insight that led to the fix.

Input Knowledge Required

To understand this message, one needs:

• Kernel module management: Understanding that rmmod removes a module, modprobe loads it with parameters from config files, and that module parameters are read-only after load.
• NVIDIA driver architecture: Knowing that nvidia_uvm is the Unified Virtual Memory module responsible for GPU memory management, and that HMM is a feature for integrating with the kernel's heterogeneous memory subsystem.
• The Proxmox/LXC context: The assistant had already established that the LXC container provides bare-metal GPU access (as opposed to the VFIO-mediated access in the KVM VM), but CUDA initialization was failing even on the host. This fix was applied to the host kernel, which also benefits the container.
• The GitHub issue chain: The discovery of issue #947 and its parent #797 provided the specific workaround. Without this external knowledge, the assistant would likely have pursued the kernel upgrade path instead.

Output Knowledge Created

This message produced several important pieces of knowledge:

1. Confirmed fix: The uvm_disable_hmm=1 parameter works for NVIDIA driver 590.48.01 on Proxmox VE kernel 6.8.12-9 with Blackwell GPUs. This is a specific, actionable finding that others in similar environments can use.
2. Parameter verification method: Reading /sys/module/nvidia_uvm/parameters/uvm_disable_hmm provides a reliable way to confirm the setting is active. The output Y is unambiguous.
3. Module reload procedure: The sequence of stopping dependent processes, removing the module, and reloading it is documented and tested. This is non-trivial because nvidia_uvm has dependencies from nvidia and nvidia_modeset.
4. The fix is persistent: Because the configuration was written to /etc/modprobe.d/nvidia-uvm-hmm.conf, it survives reboots. This is important for production deployment.

The Aftermath

The very next message (<msg id=563>) shows the payoff:

cuInit: 0
devices: 8

CUDA initialized successfully and detected all 8 Blackwell GPUs. The entire deployment pipeline — which had been blocked for hours — was now unblocked. The assistant immediately proceeded to launch the SGLang inference server, install missing dependencies, upgrade transformers, and run benchmarks that eventually achieved 806 tok/s at 128 concurrent requests.

This single message, for all its brevity, represents the critical inflection point in a complex debugging session. It demonstrates the power of systematic hypothesis testing, the value of searching for known issues in open-source repositories, and the importance of understanding the kernel-driver interface when working with cutting-edge hardware. The Y in the output is not just a parameter value — it's the sound of a door unlocking.