The Final Step: Rebooting a VM After a Kernel Module Swap

Message Overview

The subject message ([msg 5769]) is deceptively brief — just two lines of text and a single bash command:

DKMS built and installed the open modules. Now reboot:

>

[bash] ssh theuser@10.1.16.221 'sudo reboot' 2>&1; echo "Reboot sent"

>

Reboot sent

On its surface, this appears to be a trivial action: send a reboot command to a remote machine and report success. But this message is the culmination of a deep diagnostic chain that spanned multiple rounds of investigation, requiring an understanding of NVIDIA's bifurcated driver ecosystem, the peculiarities of Blackwell GPU architecture, and the mechanics of PCIe passthrough on Proxmox. It represents the final step in a surgical fix that resolved a frustrating failure mode where GPUs were visible to the operating system but entirely unusable.

The Diagnostic Journey: Why This Message Was Written

To understand why this message exists, one must trace the debugging trail that led to it. The user had been running an SGLang deployment serving the Kimi-K2.5 INT4 model on a Proxmox LXC container with 8 RTX PRO 6000 Blackwell GPUs. As a tangent experiment, the user wanted to pass all 8 GPUs through to a VM instead — specifically, a guest VM at 10.1.16.221 running on the Proxmox host at 10.1.2.6.

The initial investigation ([msg 5754] through [msg 5758]) revealed that the Proxmox host already had the infrastructure for GPU passthrough: IOMMU was enabled (iommu=pt), PCI mappings existed for the pro6000 devices, and another VM (VM 131, ml-pipelines) was already running with all 8 GPUs successfully passed through via VFIO. The host's GPUs were correctly bound to vfio-pci rather than the nvidia driver. Everything looked right on the host side.

But when the user logged into the guest VM and ran nvidia-smi, the result was baffling: "No devices were found" ([msg 5762]). Yet the GPUs were clearly visible in lspci — all 8 devices showed up. The /dev/nvidia* device files existed. The nvidia kernel module was loaded. The contradiction demanded an explanation.

The assistant dug deeper ([msg 5763]), checking lspci -ks to confirm the kernel driver in use was nvidia, and examining dmesg for error messages. There, the critical clue appeared:

NVRM: installed in this system requires use of the NVIDIA open kernel modules.
NVRM: GPU ... RmInitAdapter failed! (0x22:0x56:1001)

This error message is the key to the entire problem. NVIDIA's driver strategy has evolved significantly with recent hardware generations. For modern architectures, NVIDIA offers two parallel kernel module tracks: the proprietary (nvidia.ko) modules, which have been the standard for decades, and the open (nvidia-open.ko) modules, which NVIDIA began releasing as GPL-licensed source code starting with the 515-series drivers. While both modules expose the same interface to userspace libraries and applications, they differ internally in how they initialize and manage the GPU hardware.

The critical detail is that Blackwell GPUs (SM120 architecture, device ID 10de:2bb5) require the open kernel modules. The proprietary module's RmInitAdapter function — the low-level routine that initializes the GPU adapter — fails on these cards with error code 0x22:0x56:1001. The proprietary driver simply cannot initialize Blackwell GPUs at all. This is not a bug or a configuration issue; it is an intentional design decision by NVIDIA to route new architecture support exclusively through the open module path.

Input Knowledge Required

Understanding this message requires several layers of domain knowledge:

  1. DKMS (Dynamic Kernel Module Support): The mechanism by which NVIDIA kernel modules are built and installed. DKMS automatically rebuilds kernel modules when the kernel is updated, ensuring driver compatibility across kernel versions. The assistant's statement "DKMS built and installed the open modules" references this build process, which compiles the nvidia-open source code against the running kernel's headers.
  2. NVIDIA's bifurcated driver ecosystem: The coexistence of proprietary (nvidia-dkms-590) and open (nvidia-dkms-590-open) kernel module packages. Both are versioned identically (590.48.01 in this case) and share the same userspace libraries (libnvidia-gl-590, libnvidia-compute-590, etc.), but the kernel modules themselves are different codebases.
  3. GPU architecture-specific requirements: The knowledge that SM120 (Blackwell) GPUs require open kernel modules is not widely documented — it emerges from NVIDIA's release notes and community experience. The assistant had to recognize the RmInitAdapter failed error and connect it to the Blackwell requirement.
  4. PCIe passthrough and VFIO: Understanding that GPU passthrough to a VM requires the host to unbind the GPU from the nvidia driver and bind it to vfio-pci, while the guest VM loads its own nvidia driver to manage the device. The host-side setup was already correct, but the guest had the wrong kernel module flavor.
  5. The reboot requirement: After swapping kernel modules via DKMS, a reboot is mandatory because the kernel cannot unload and reload the nvidia module while it's in use (or even easily swap between proprietary and open variants). The nvidia-dkms-590-open package replaces the kernel module binary, but the running kernel still has the old proprietary module loaded. Only a reboot causes the new open module to be loaded at boot time.

The Decision Process: Why Reboot and Not Something Else

The assistant's decision to issue a reboot command was not arbitrary — it was the logical conclusion of a carefully reasoned chain. The previous message ([msg 5766]) laid out the diagnosis and proposed the fix: "Just swap the DKMS package: sudo apt install nvidia-dkms-590-open. This will replace nvidia-dkms-590 (proprietary) with the open kernel modules and rebuild via DKMS. Then reboot the VM."

The user's response ([msg 5767]) was simply "Run it." The assistant then executed the apt install command ([msg 5768]), which completed successfully. The output of that command (partially visible in the conversation) showed the package manager replacing the proprietary DKMS package with the open one and triggering a DKMS build.

Now in [msg 5769], the assistant reports the result of that build ("DKMS built and installed the open modules") and proceeds with the mandatory next step: rebooting. There was no alternative path here. Without a reboot, the old proprietary module would remain loaded in the kernel, and the GPUs would continue to fail initialization. A module reload (modprobe -r nvidia && modprobe nvidia) might have worked in theory, but in practice the nvidia module has complex dependencies (nvidia_drm, nvidia_modeset, nvidia_uvm) that make clean unloading difficult, especially when userspace processes have opened the devices. A reboot is the reliable, predictable approach.

The assistant chose to send the reboot command via SSH with sudo reboot, which is the standard way to remotely reboot a Linux system. The 2>&1 redirect ensures any stderr output (e.g., if the SSH connection drops before the reboot completes) is captured. The echo "Reboot sent" provides immediate confirmation that the command was dispatched, since the SSH session may terminate before the reboot actually completes.

Assumptions and Potential Mistakes

The assistant made several assumptions in this message:

  1. The DKMS build succeeded: The statement "DKMS built and installed the open modules" assumes the apt install command from the previous round completed without errors. The assistant could not see the full output of that command in the same round (since tool calls are synchronous and the assistant waits for all results before the next round), but the truncated output visible in [msg 5768] showed no error messages, suggesting success.
  2. The reboot will complete cleanly: There is an implicit assumption that the VM will reboot successfully and come back up with the open kernel modules loaded. The assistant does not verify this — it sends the reboot and reports "Reboot sent," but does not wait for the VM to come back online or check that the GPUs are now functional. This is a reasonable division of responsibility: the reboot is a fire-and-forget operation at this point, and the user can verify the result afterward.
  3. The VM has working console access: If the reboot hangs or the VM fails to boot, the user would need to access the VM console through Proxmox to debug. The assistant does not provide fallback instructions, assuming the VM will reboot normally.
  4. The open modules are fully compatible: While the open kernel modules support Blackwell GPUs, there could be edge cases where certain features (e.g., MIG, certain virtualization features) behave differently. The assistant assumes the swap is a drop-in replacement, which is generally true for the 590.48.01 release. One potential mistake is that the assistant did not verify the DKMS build output more thoroughly. The apt install output was truncated in the conversation data, and the assistant did not run a follow-up command like dkms status or ls /lib/modules/$(uname -r)/updates/dkms/nvidia* to confirm the open module was built and installed. However, given the constraints of the conversation (the apt command timed out or was truncated), the assistant made a reasonable judgment call to proceed.

Output Knowledge Created

This message creates several pieces of output knowledge:

  1. Confirmation of the DKMS build: The statement "DKMS built and installed the open modules" documents that the package swap completed successfully, providing closure to the diagnostic chain.
  2. The reboot command itself: The execution of sudo reboot on the VM initiates the state transition from the broken configuration (proprietary module loaded, GPUs failing RmInitAdapter) to the fixed configuration (open module loaded, GPUs functional).
  3. Confirmation of command dispatch: The "Reboot sent" output confirms that the SSH session successfully delivered the reboot command to the remote host, at least as far as the network layer is concerned.
  4. A documented workflow for Blackwell GPU passthrough: The entire sequence from diagnosis to fix creates a reusable pattern: when Blackwell GPUs are passed through to a VM and nvidia-smi reports "No devices were found" despite the devices being visible in lspci, check dmesg for RmInitAdapter failed errors, and if present, swap from nvidia-dkms-590 to nvidia-dkms-590-open and reboot.

The Thinking Process Visible in the Reasoning

The assistant's reasoning is most visible not in this message itself, but in the chain of messages leading up to it. The diagnostic process followed a classic troubleshooting pattern:

  1. Observe the symptom: nvidia-smi reports "No devices were found" ([msg 5762]).
  2. Verify the device is visible to the OS: lspci shows 8 NVIDIA devices ([msg 5761]).
  3. Verify the driver is loaded: /dev/nvidia* exists, lspci -ks shows "Kernel driver in use: nvidia" ([msg 5763]).
  4. Check driver logs for errors: dmesg | grep -i nvidia reveals the RmInitAdapter failed error ([msg 5764]).
  5. Interpret the error: The error message explicitly states "installed in this system requires use of the NVIDIA open kernel modules" ([msg 5764]).
  6. Check what's installed: dpkg -l | grep nvidia shows nvidia-dkms-590 (proprietary) is installed ([msg 5765]).
  7. Verify the fix is available: apt list --all-versions nvidia-dkms-590-open confirms the open module package exists in the repository ([msg 5765]).
  8. Propose the fix: Explain the swap, confirm it's a drop-in replacement ([msg 5766]).
  9. Execute with user approval: Run the apt install ([msg 5768]).
  10. Complete the fix: Report success and reboot ([msg 5769]). This methodical approach — moving from symptom to root cause to fix — demonstrates a systematic debugging methodology. Each step eliminates a hypothesis: the GPUs aren't missing from the PCI bus, the driver is loaded, the error is specific and documented, the fix is available and straightforward.

Broader Significance

This message, while small, illustrates several important principles of systems engineering. First, it shows how modern GPU computing has introduced new layers of complexity in driver management. The split between proprietary and open kernel modules is not just a licensing distinction — it carries real functional implications for which hardware is supported. Second, it demonstrates the importance of reading error messages carefully. The dmesg output contained the exact diagnosis; the problem was not a missing driver but the wrong flavor of driver. Third, it highlights the value of methodical troubleshooting: rather than randomly reinstalling drivers or rebooting repeatedly, the assistant followed a logical chain from symptom to cause to fix.

The message also underscores a practical reality of working with cutting-edge hardware: documentation is often sparse, and solutions emerge from combining error messages with domain knowledge about GPU architecture support policies. The knowledge that Blackwell GPUs require open kernel modules is not something one would find in a typical "getting started" guide — it's a detail that lives in NVIDIA's release notes, community forums, and the collective experience of engineers working with this hardware.

In the end, a two-line message and a reboot command represent the successful conclusion of a diagnostic journey that required understanding PCIe passthrough, DKMS mechanics, NVIDIA's driver architecture, and the specific requirements of Blackwell GPUs. The reboot itself is trivial; the knowledge required to know that this reboot was the right thing to do is anything but.