The Kernel Module That Wouldn't Initialize: Debugging CUDA's HMM Bug on Proxmox

In the high-stakes world of GPU-accelerated machine learning infrastructure, few moments are as satisfying as the one where a stubborn, days-long hardware initialization bug finally yields to a single, well-researched configuration change. This article examines a pivotal message in an opencode coding session — message index 561 — where the assistant, after an exhaustive debugging journey, prepares to apply the fix that will finally bring CUDA to life on a Proxmox-hosted LXC container equipped with eight NVIDIA RTX PRO 6000 Blackwell GPUs.

The message itself is deceptively brief. It contains the assistant's reasoning, a single bash command, and its output. But behind this brevity lies a complex chain of investigation spanning kernel internals, GPU firmware architecture, virtualization topology, and the peculiarities of NVIDIA's open-source kernel module strategy. To understand why this message matters, we must first understand the crisis it aims to resolve.

The CUDA Initialization Crisis

The session's broader context reveals a multi-day struggle to deploy the GLM-5-NVFP4 large language model across eight Blackwell GPUs. The infrastructure team had chosen Proxmox VE as their virtualization platform, initially running the GPUs through VFIO passthrough to a KVM virtual machine. While this approach worked, it introduced severe performance bottlenecks: the virtualized PCIe topology prevented direct peer-to-peer (P2P) GPU communication, crippling the tensor-parallel inference workloads that the GLM model required.

The solution seemed elegant: switch from a KVM virtual machine to an LXC container, which shares the host kernel and thus should provide bare-metal GPU access with full P2P capabilities. The container was created, the NVIDIA driver was installed on the host, and GPU devices were bind-mounted into the container. Everything looked promising — until the moment of truth arrived and cuInit(), the fundamental CUDA runtime initialization call, simply refused to work.

The failure manifested in two distinct modes. Sometimes cuInit() would hang indefinitely, blocking until a kernel timeout killed the process. Other times it would return error code 3, defined as CUDA_ERROR_NOT_INITIALIZED, with no further explanation. The GPUs were visible to nvidia-smi, the driver version was the latest 590.48.01, and the CUDA toolkit was properly installed. Yet every attempt to initialize the CUDA runtime failed.

The Investigation: Tracing the Bug

The assistant's debugging process, visible in the messages preceding our target, demonstrates a methodical approach to kernel-level troubleshooting. The first hypothesis was missing GSP (GPU System Processor) firmware for the Blackwell architecture. The firmware directory contained only gsp_ga10x.bin and gsp_tu10x.bin — files named for the Ampere and Turing architectures, respectively. However, closer inspection revealed that the gsp_ga10x.bin file was a container format that actually included Blackwell firmware sections (kernel_gb20x.elf, kernel_gb20y.elf), and nvidia-smi confirmed that GSP firmware version 590.48.01 was loaded and operational.

The next hypothesis involved the Proxmox VE kernel version (6.8.12-9) being too old. But strace analysis showed that cuInit() was making successful ioctl calls to the NVIDIA device files — the kernel module was responding, but the userspace library was rejecting the initialization.

The breakthrough came from searching the NVIDIA open-gpu-kernel-modules issue tracker. Issue #947 described the exact same symptom: cuInit() returning error code 3 with the open kernel module while the proprietary module worked fine. The issue was closed as a duplicate of #797, which revealed the root cause: the Heterogeneous Memory Management (HMM) feature in the nvidia_uvm kernel module.

HMM is a feature in NVIDIA's open kernel module that enables the GPU to access system memory through the kernel's standard memory management infrastructure, rather than through the traditional UVM (Unified Virtual Memory) driver path. On certain kernel configurations — particularly those used by Proxmox VE — the HMM initialization path fails silently, causing cuInit() to either hang or return error code 3. The workaround, confirmed by NVIDIA's own developers, is to disable HMM by setting the module parameter uvm_disable_hmm=1 on the nvidia_uvm module.

The Subject Message: Applying the Fix

With the root cause identified, the assistant's message at index 561 represents the critical transition from investigation to remediation:

Now I need to reload the nvidia_uvm module for this to take effect. Let me stop the container first (since it might be using the GPU devices), then rmmod and modprobe.

>

`` ssh root@10.1.2.6 "pct stop 129 2>&1; sleep 2; lsmod | grep nvidia" nvidia_uvm 2043904 0 nvidia_drm 135168 0 nvidia_modeset 1683456 1 nvidia_drm nvidia 14753792 2 nvidia_uvm,nvidia_modeset ecc 40960 1 nvidia video 77824 1 nvidia_modeset ``

This message reveals several layers of technical reasoning. First, the assistant correctly recognizes that a module parameter change written to /etc/modprobe.d/ is only applied when the module is loaded. Since nvidia_uvm is already loaded (as confirmed by lsmod), it must be unloaded and reloaded for the uvm_disable_hmm=1 parameter to take effect.

Second, the assistant anticipates a practical complication: the LXC container (ID 129) may have open file descriptors to the NVIDIA device files. If the container holds references to /dev/nvidia-uvm or the GPU device files, the kernel module cannot be safely unloaded. The pct stop 129 command cleanly shuts down the container, releasing those references.

Third, the lsmod output provides a snapshot of the NVIDIA kernel module dependency chain. The nvidia_uvm module has a reference count of 0 (no other loaded module depends on it), meaning it can be unloaded independently. However, the nvidia core module has a reference count of 2, indicating that both nvidia_uvm and nvidia_modeset depend on it. This dependency graph informs the reload strategy: only nvidia_uvm needs to be reloaded, not the entire NVIDIA driver stack.

The Thinking Process Visible in the Message

The assistant's reasoning, stated explicitly in the message, reveals a careful consideration of operational safety. The phrase "Let me stop the container first (since it might be using the GPU devices)" demonstrates an understanding that kernel module operations require exclusive access to device resources. The parenthetical clarification shows the assistant explaining its reasoning for a non-obvious prerequisite step.

The decision to run lsmod | grep nvidia after stopping the container serves a dual purpose. It confirms that the container has been successfully stopped (the command completed without error), and it provides the current module state before the reload operation. This is a classic systems-administration pattern: verify the starting state before making a change, so that any unexpected behavior can be detected.

Input Knowledge Required

To fully understand this message, the reader needs knowledge spanning several domains. One must understand the Linux kernel module system — how modprobe.d configuration files work, how module parameters are passed, and how the lsmod output (module name, size, reference count, and dependency list) reveals the module dependency graph. One must understand LXC container management, including the pct command and the concept that containers share the host kernel and can hold references to host device files. One must understand NVIDIA's GPU driver architecture, specifically the separation between the core nvidia module (PCIe device management), nvidia_uvm (Unified Virtual Memory), nvidia_modeset (display mode setting), and nvidia_drm (Direct Rendering Manager). And one must understand the CUDA runtime initialization sequence, where cuInit() triggers UVM setup that may fail if HMM is broken.

Output Knowledge Created

This message produces several pieces of actionable knowledge. It confirms that the container has been successfully stopped. It documents the exact state of the NVIDIA kernel modules at the moment before the fix is applied. It establishes the dependency chain that will guide the reload procedure. And it implicitly validates the debugging conclusion — that the nvidia_uvm module with its HMM feature is the culprit — by demonstrating that the module is loaded and therefore eligible for the parameter change.

The Broader Significance

This message represents the culmination of a debugging process that touched on GPU firmware analysis, kernel module internals, virtualization architecture, and open-source driver development. The HMM bug in NVIDIA's open kernel module is a relatively obscure issue — it only manifests on specific kernel configurations and only affects the open module, not the proprietary one. The assistant's ability to trace the symptom (cuInit error 3) to the root cause (HMM incompatibility with the Proxmox kernel) through a combination of strace analysis, firmware inspection, and GitHub issue searching demonstrates the kind of cross-domain troubleshooting that modern AI-assisted systems engineering requires.

The fix itself — a single line in a modprobe configuration file — is almost anticlimactic after the extensive investigation. But that is often the nature of kernel-level debugging: the solution is simple once the problem is truly understood. The real work lies not in applying the fix, but in finding it.