The "Aha!" Moment: How a Single Kernel Module Parameter Fixed CUDA on Blackwell GPUs in an LXC Container

In the long and winding journey of deploying a GLM-5-NVFP4 inference server across eight NVIDIA RTX PRO 6000 Blackwell GPUs, there comes a moment of crystalline clarity—a message where all the threads of investigation converge, a root cause is identified, and a fix is within reach. That message is [msg 559], and it represents the turning point in a saga that had spanned multiple sessions, countless bash commands, and deep dives into NVIDIA driver internals, Proxmox VE kernel quirks, and GPU firmware loading.

The Long Road to This Moment

To understand the significance of [msg 559], we must first appreciate the context. The assistant had been working across several segments of an opencode session to set up a machine learning inference environment on a Proxmox VE host with eight Blackwell GPUs. The journey began in Segment 0 with driver installation and environment setup, moved through Segment 1-2 with SGLang deployment and NaN debugging, and arrived at Segment 3-4 where the critical bottleneck was identified: the KVM virtual machine's VFIO/IOMMU path prevented peer-to-peer (P2P) DMA between GPUs, crippling performance for tensor-parallel inference workloads.

The solution was to bypass the VM entirely by using an LXC container on the Proxmox host, which would give the GPUs bare-metal access with proper P2P topology. But this fix introduced a new and devastating problem: CUDA refused to initialize.

The Symptom: cuInit Error 3

The symptom was deceptively simple. When the assistant ran cuInit(0)—the fundamental entry point for any CUDA program—it returned error code 3, which maps to CUDA_ERROR_NOT_INITIALIZED. This is not a helpful error; it is a catch-all failure that means "something went wrong during CUDA driver initialization, and I can't tell you what." The NVIDIA driver stack had loaded successfully (nvidia-smi worked perfectly, showing all eight GPUs with compute capability 12.0), the device files were present (/dev/nvidia0 through /dev/nvidia7, plus /dev/nvidia-uvm and /dev/nvidiactl), and the CUDA userspace libraries were correctly installed. Yet cuInit stubbornly returned 3.

Earlier in the investigation, the assistant had even observed cuInit hanging indefinitely (exit code 124 from a timeout command), suggesting that the initialization was not just failing but sometimes blocking on some kernel-level operation. This made debugging particularly difficult—was it a hang, a crash, or a clean failure? The behavior seemed inconsistent, which is often a hallmark of race conditions or timeout-dependent initialization paths in kernel drivers.

Tracing the Root Cause

The assistant's investigation in the messages leading up to [msg 559] was methodical. It checked GSP (GPU System Processor) firmware loading, confirmed that the gsp_ga10x.bin firmware file actually contained Blackwell ELF sections (kernel_gb20y.elf, kernel_gb20x.elf), verified that GSP firmware version 590.48.01 was reported as loaded, ruled out missing firmware as the cause, and examined strace output to see which ioctl calls were failing.

A critical clue emerged from nvidia-smi -q: the driver reported Addressing Mode: HMM. HMM stands for Heterogeneous Memory Management, a feature of the NVIDIA open kernel module that allows the GPU to manage system memory through the kernel's memory management subsystem. HMM relies on specific kernel infrastructure—particularly mmu_notifier and migration primitives—that may not be fully functional or correctly integrated in the Proxmox VE kernel (version 6.8.12-9 at the time).

The Discovery: GitHub Issue #947

The breakthrough came when the assistant searched for the exact error pattern: "cuInit error 3 Blackwell Proxmox LXC open kernel module 590." The search returned a direct hit: GitHub issue #947 in the NVIDIA/open-gpu-kernel-modules repository, titled "cuInit() returns CUDA_ERROR_NOT_INITIALIZED (3) with open kernel module; proprietary module works."

The assistant fetched the full issue page and discovered that it was closed as a duplicate of issue #797. The key insight from these issues was that the problem was a known incompatibility between the NVIDIA open kernel module's HMM feature and certain kernel configurations. The recommended workaround was to disable HMM by setting uvm_disable_hmm=1 as a module parameter for nvidia_uvm.

Analyzing Message 559

Let us now examine the subject message itself in full:

This is exactly our bug. The key fix mentioned is: Setting uvm_disable_hmm=1 on the open kernel module sometimes works around/solves the issue.

>

This makes perfect sense — the host shows Addressing Mode: HMM in nvidia-smi, meaning the driver is using Heterogeneous Memory Management which requires specific kernel support that may not be properly functional on the PVE kernel 6.8.12-9.

>

Let me try this fix immediately!

>

[todo list updated to mark "Try uvm_disable_hmm=1 to fix cuInit error 3 (HMM issue)" as in_progress]

This message is remarkable for what it reveals about the assistant's reasoning process. First, there is the explicit connection drawn between the external knowledge (the GitHub issue) and the local observation ("the host shows Addressing Mode: HMM in nvidia-smi"). This is not a blind application of a found fix—it is a reasoned diagnosis. The assistant understands why the fix works: HMM requires kernel support that the PVE kernel lacks, so disabling HMM removes the dependency.

Second, the message shows confidence. The phrase "This is exactly our bug" is definitive, not speculative. After hours of debugging—checking firmware, strace output, kernel messages, device file permissions, and library versions—the assistant has found the single root cause that explains all the observed symptoms. The inconsistent behavior (sometimes hang, sometimes error 3) is consistent with a kernel feature that works partially or intermittently depending on timing and memory state.

Third, the todo list update is revealing. The assistant immediately promotes the HMM fix to "in_progress" status and demotes the previously planned kernel upgrade to "pending." This shows adaptive prioritization: the most promising fix is tried first, and the fallback (kernel upgrade) is kept ready if needed.

Knowledge Required to Understand This Message

To fully grasp the significance of [msg 559], one needs several layers of knowledge:

  1. CUDA driver architecture: Understanding that cuInit is the first CUDA API call that initializes the driver stack, and that it involves multiple kernel-level interactions including device file access, ioctl calls, and memory management setup.
  2. NVIDIA open kernel module vs. proprietary module: The open module (nvidia-open) has different features and limitations compared to the proprietary module. HMM is a feature specific to the open module that enables better integration with Linux kernel memory management but requires specific kernel support.
  3. Heterogeneous Memory Management (HMM): A kernel feature that allows device drivers to manage memory across CPU and GPU domains. It relies on kernel primitives like mmu_notifier and migration infrastructure that may not be present or correctly configured in all kernels.
  4. Proxmox VE kernel specifics: PVE kernels are based on Ubuntu's kernel but with additional patches for virtualization features. They may not include all the upstream kernel features that NVIDIA's open module expects, particularly in the memory management subsystem.
  5. LXC container GPU passthrough: Understanding that LXC containers share the host kernel, so kernel-level issues (like HMM incompatibility) affect both the host and containers equally. This is different from KVM virtual machines where a different kernel (the guest kernel) can be used.
  6. Module parameters for NVIDIA drivers: The nvidia_uvm module (Unified Virtual Memory) accepts parameters like uvm_disable_hmm that control its behavior. These can be set via kernel boot parameters, module configuration files, or runtime modprobe commands.

Output Knowledge Created

This message creates several important pieces of knowledge for the ongoing session:

  1. A confirmed diagnosis: The root cause of the CUDA initialization failure is identified as HMM incompatibility with the PVE kernel. This is not a guess—it is supported by both the GitHub issue (external validation) and the local observation of "Addressing Mode: HMM" in nvidia-smi (internal validation).
  2. A specific fix to apply: uvm_disable_hmm=1 as a module parameter for nvidia_uvm. The assistant knows exactly what command to run.
  3. A fallback plan: If the HMM fix doesn't work, the next step would be upgrading the PVE kernel from 6.8.12-9 to 6.8.12-18 and rebuilding the NVIDIA DKMS modules.
  4. Updated priorities: The todo list is reorganized to focus on the most promising fix first.

Assumptions and Potential Pitfalls

The assistant makes several assumptions in this message:

  1. That the GitHub issue is correct and applicable: The issue was for a different hardware/driver combination, but the assistant assumes the fix generalizes to the Blackwell/590.48.01 combination. This is a reasonable assumption given the identical symptom, but there is always a risk that the root cause is different despite the same error code.
  2. That HMM is the only incompatibility: The assistant assumes that disabling HMM will fully resolve the issue. In reality, there could be multiple kernel incompatibilities, and disabling HMM might fix cuInit but cause other problems later (e.g., with memory allocation or GPU-to-CPU data transfer).
  3. That the PVE kernel is at fault: The assistant attributes the HMM issue to "specific kernel support that may not be properly functional on the PVE kernel 6.8.12-9." This is a reasonable inference, but it is also possible that the NVIDIA open module's HMM implementation itself has a bug that manifests on any kernel, and the PVE kernel just happens to trigger it.
  4. That the fix is safe: Disabling HMM might have performance implications for Unified Virtual Memory operations. The assistant implicitly assumes that the performance cost is acceptable compared to the current state (CUDA not working at all).

The Thinking Process Visible in the Message

The message reveals a clear chain of reasoning:

  1. Observation: The GitHub issue describes exactly our symptom (cuInit error 3 with open kernel module).
  2. Connection: The issue mentions uvm_disable_hmm=1 as a workaround.
  3. Validation: We can verify this applies to our case because nvidia-smi shows "Addressing Mode: HMM."
  4. Explanation: HMM requires kernel support that the PVE kernel likely doesn't provide correctly.
  5. Action: Try the fix immediately. This is textbook diagnostic reasoning: identify the symptom, find a known match, verify the match with local evidence, understand the mechanism, and apply the fix. The assistant does not just blindly apply the workaround—it takes the time to explain why it should work, which builds confidence in the diagnosis.

The Broader Significance

Message [msg 559] is the turning point of Segment 5. Before this message, the entire LXC approach was in jeopardy—if CUDA couldn't initialize in the container, the bare-metal GPU topology advantage was worthless. After this message, the fix is applied and confirmed to work, leading to the successful deployment of the SGLang inference server and the achievement of 806 tok/s throughput at 128 concurrency.

The message also illustrates a crucial lesson in systems debugging: when a low-level API like cuInit fails with a generic error, the root cause is often not in the obvious places (missing firmware, wrong library versions, device file permissions) but in subtle kernel-level interactions between the driver and the kernel's memory management subsystem. The assistant's willingness to search for known issues in upstream repositories, rather than continuing to debug in isolation, is what ultimately broke the logjam.

Conclusion

Message [msg 559] is a masterclass in diagnostic reasoning. It connects external knowledge (a GitHub issue) with local observations (HMM addressing mode) to produce a confident diagnosis and a targeted fix. The message is concise but dense with meaning: it represents the culmination of hours of debugging, the application of community knowledge, and the reasoned adaptation of a known workaround to a specific hardware and software configuration. In the narrative of this opencode session, it is the moment when the protagonist finally sees the solution and reaches for it.