The Missing Device Major Number: Debugging CUDA Initialization After a Kernel Upgrade

In the middle of an intense optimization session for GLM-5-NVFP4 inference on eight RTX PRO 6000 Blackwell GPUs, the assistant encountered a frustrating roadblock. After performing a major kernel upgrade — from Proxmox's 6.8.12-9-pve to a custom 6.14.11-5-bpo12-pve with aggressive tuning flags like amd_pstate=active and processor.max_cstate=1 — the host machine rebooted successfully, all eight GPUs were detected, and nvidia-smi reported cleanly. But inside the LXC container that ran the actual inference workload, CUDA had stopped working. PyTorch reported a cryptic CUDA unknown error (error code 999), and the entire benchmarking pipeline was dead. Message 1333 is the turning point in this debugging story: a single, deceptively simple bash command that diagnosed the root cause and set the stage for a swift fix.

The Context: Why CUDA Broke After a Kernel Upgrade

The assistant had spent the preceding messages orchestrating a comprehensive system audit and tuning pass. The goal was to squeeze every drop of performance from the Blackwell GPUs for the GLM-5-NVFP4 model. The audit had revealed multiple systemic issues: a suboptimal CPU frequency driver (acpi-cpufreq instead of amd_pstate), an outdated kernel, enabled NUMA balancing, deep CPU C-states, and a PCIe MaxReadReq stuck at 512 bytes instead of the optimal 4096. All of these were fixed in a coordinated push that culminated in a kernel upgrade to 6.14.11 and a full reboot.

When the host came back up, everything looked perfect on the surface. The new kernel was running, the amd-pstate-epp driver was active, C-states were restricted to C0/C1, MaxReadReq was set to 4096, and all eight GPUs showed up in nvidia-smi. But the LXC container — the actual environment where SGLang and PyTorch ran — failed to initialize CUDA. The assistant tried a simple PyTorch CUDA test and got:

CUDA initialization: CUDA unknown error - this may be due to an incorrectly set up environment...

This was not the same error as before the kernel upgrade. Earlier, the team had dealt with CUDA_ERROR_NOT_INITIALIZED (error code 3), which was fixed by disabling HMM in the nvidia_uvm module via uvm_disable_hmm=1. The new error code 999 (CUDA_ERROR_UNKNOWN) was different, and it pointed to a different root cause.

The Diagnostic Leap: Checking Device Major Numbers

The assistant methodically worked through the possibilities. First, it verified that uvm_disable_hmm=Y was still set on the new kernel — it was. It checked nvidia-smi inside the container — that worked fine, listing all eight GPUs. It checked device nodes — /dev/nvidia0 through /dev/nvidia7 and /dev/nvidia-uvm were all present with what looked like correct major numbers. Everything seemed fine, yet cuInit returned error 999.

The critical insight came when the assistant examined the device nodes more carefully. Inside the container, the nvidia-uvm device had major number 509. But the assistant remembered that the LXC container's cgroup configuration — which controls which device major numbers the container is allowed to access — might have been written for the old kernel's device numbering. Kernel upgrades can change the assignment of major numbers to device drivers, especially when the driver is rebuilt via DKMS against a new kernel version.

Message 1333 captures this exact moment of diagnosis:

[assistant] Device nodes look correct. The issue might be that the nvidia-uvm major number changed with the new kernel. Let me check:
[bash] ssh root@10.1.2.6 'cat /proc/devices | grep nvidia'
195 nvidia
195 nvidia-modeset
195 nvidiactl
235 nvidia-nvswitch
236 nvidia-nvlink
237 nvidia-caps
238 nvidia-caps-imex-channels
509 nvidia-uvm

This single command revealed the truth. The host's /proc/devices showed the actual major numbers assigned by the new kernel: 509 for nvidia-uvm and 237 for nvidia-caps. The assistant already suspected these numbers might have changed because the NVIDIA driver was rebuilt via DKMS for the new kernel, and kernel module loading order or driver registration can shift the assigned major numbers.

What the Command Revealed

The output from /proc/devices is a map of all registered device drivers and their assigned major numbers. The nvidia driver family uses major number 195 for the basic GPU devices (/dev/nvidia0 through /dev/nvidia7 and /dev/nvidiactl). This number is stable because it's assigned by the NVIDIA driver itself via a register_chrdev call with a fixed major number. However, the nvidia-uvm (Unified Virtual Memory) driver and nvidia-caps (capabilities) driver use dynamic major numbers allocated by the kernel at module load time. These can change between kernel versions, especially when the driver is rebuilt.

The assistant's reasoning was precise: if the LXC container's cgroup configuration still referenced the old major numbers for nvidia-uvm and nvidia-caps, the container's cgroup would block access to those devices, causing cuInit to fail with a generic unknown error. The error code 999 (CUDA_ERROR_UNKNOWN) is often a catch-all for low-level driver communication failures, which is exactly what would happen if CUDA tried to open the UVM device and the kernel denied permission.

The Assumption and Its Validation

The assistant made a key assumption: that the cgroup configuration in the LXC container's config file (/etc/pve/lxc/129.conf) contained stale major numbers. This was a reasonable hypothesis because the container had been configured when the old kernel (6.8.12) was running, and the cgroup rules had been set up at that time. The assistant had not yet looked at the actual cgroup rules — it was working purely from the hypothesis that the major numbers might have changed.

The assumption was correct. In the very next message (msg 1334), the assistant checked the container config and found the cgroup rules:

lxc.cgroup2.devices.allow: c 195:* rwm
lxc.cgroup2.devices.allow: c 504:* rwm
lxc.cgroup2.devices.allow: c 507:* rwm

The old rules allowed major numbers 504 and 507, but the new kernel had assigned 509 to nvidia-uvm and 237 to nvidia-caps. The mismatch was the root cause of the CUDA initialization failure.

Input Knowledge Required

To understand this message, the reader needs several pieces of background knowledge. First, an understanding of Linux device major numbers: every character device in Linux is identified by a major number (driver type) and a minor number (instance). The /proc/devices file lists all registered drivers and their assigned major numbers. Second, knowledge of LXC container cgroup device access control: Proxmox LXC containers use cgroup v2 device allow rules to control which hardware devices a container can access. A rule like c 509:* rwm allows read, write, and mknod access to all devices with major number 509. If the rule doesn't match the actual major number, the container gets EPERM (permission denied) when trying to open the device. Third, familiarity with NVIDIA's driver architecture: the nvidia-uvm driver handles Unified Virtual Memory management and is essential for CUDA memory allocation; without it, cuInit fails. Fourth, the understanding that DKMS-rebuilt drivers can receive different dynamic major numbers after a kernel upgrade, since the kernel allocates dynamic majors based on load order and available slots.

Output Knowledge Created

This message produced critical diagnostic information that directly led to the fix. It confirmed that the nvidia-uvm major number had changed from 504 (under kernel 6.8.12) to 509 (under kernel 6.14.11), and that the nvidia-caps major number had changed from 507 to 237. Armed with this knowledge, the assistant was able to update the LXC container configuration in subsequent messages: it stopped the container, replaced the stale cgroup rules with the correct ones (c 509:* rwm, c 237:* rwm, and c 238:* rwm for the imex-channels device), restarted the container, and verified that CUDA initialized successfully. The fix took only a few minutes from diagnosis to resolution.

Broader Significance

This message exemplifies a class of system administration problems that are easy to overlook: configuration drift between kernel versions. When a kernel is upgraded, especially across major versions, device numbering, module parameters, and driver behavior can all change in subtle ways. The LXC container's cgroup configuration was a snapshot of the old kernel's device topology, and it silently broke after the upgrade. The assistant's debugging approach — verifying the actual device numbers against the container's permissions — is a textbook example of how to diagnose "impossible" CUDA errors that appear after system changes. The fix itself was trivial once the root cause was identified, but finding that root cause required precise reasoning about how the kernel assigns device major numbers and how LXC enforces device access control.

The message also highlights the importance of treating container configurations as versioned artifacts that may need updating after host-level changes. A kernel upgrade is not just a software update; it can change the fundamental device topology that containers depend on. The assistant's systematic approach — checking host devices, container devices, cgroup rules, and then comparing them — turned a frustrating CUDA error into a straightforward configuration fix, clearing the way for the performance benchmarking that followed.