The Stale Major Number: Diagnosing a Post-Kernel-Upgrade CUDA Failure in LXC

Introduction

In the course of optimizing an 8-GPU machine for large-scale ML inference, a critical moment arrived when a kernel upgrade from 6.8.12 to 6.14.11 on a Proxmox host broke CUDA inside an LXC container. Message [msg 1336] captures the precise diagnostic turning point: the assistant identifies that the container's cgroup device access rules contain stale major device numbers for the NVIDIA Unified Virtual Memory (UVM) and NVIDIA capabilities (nvidia-caps) devices, mismatched against the new kernel's device map. This message is a masterclass in systematic debugging under the constraints of containerized GPU environments, and it reveals how a seemingly obscure system administration detail — device major numbers — can silently sabotage an entire inference pipeline.

The Context: A Major System Tuning Effort

The message sits within a larger narrative of aggressive performance optimization. The team had just computed the theoretical maximum single-stream throughput for the GLM-5-NVFP4 model at 309 tok/s, only to measure a disappointing 10.36 tok/s — a staggering 3.4% efficiency gap. A comprehensive parallel system audit ([msg 1318] and surrounding messages) had uncovered multiple misconfigurations: the CPU was running the suboptimal acpi-cpufreq governor instead of amd_pstate, the kernel was outdated (6.8.12), NUMA balancing was enabled, deep CPU C-states were active, and the PCIe MaxReadReq was stuck at 512 bytes instead of the optimal 4096.

The team applied runtime fixes and executed a major kernel upgrade to 6.14.11 with boot parameters amd_pstate=active and processor.max_cstate=1. After a successful reboot, the host came back healthy — all 8 GPUs detected, the new amd-pstate-epp CPU driver active, C-states restricted to C0/C1, and the PCIe tuning service running. But when the LXC container (named llm-two, VMID 129) came back up, CUDA initialization failed with error code 999 (CUDA_ERROR_UNKNOWN), a different and more opaque error than the previous error 3 (cuInit not initialized) that had been seen before.

The Message: A Precise Diagnostic Leap

Message [msg 1336] opens with the assistant's confident declaration: "Found the problem." This is not a guess — it is the culmination of a focused investigation that spanned several messages. The assistant had already verified that:

lxc.cgroup2.devices.allow: c 195:* rwm
lxc.cgroup2.devices.allow: c 504:* rwm
lxc.cgroup2.devices.allow: c 507:* rwm

The cgroup rules allowed major numbers 504 and 507, but the actual devices on the new kernel were 509 (nvidia-uvm) and 237 (nvidia-caps). The mismatch was clear.

The Reasoning: Why Cgroup Device Rules Matter

The assistant's reasoning demonstrates a deep understanding of how LXC containers interact with device access. In Proxmox's LXC implementation, containers use cgroup device access control lists (cgroup2 devices.allow) to specify which device major/minor numbers the container is permitted to access. These rules are separate from bind-mount entries — even if the device node file appears in /dev/ inside the container (via lxc.mount.entry bind mounts), the cgroup must explicitly authorize the device major number for the container's processes to interact with it.

The NVIDIA driver stack requires access to several device classes:

Assumptions and Potential Mistakes

The assistant made several implicit assumptions that turned out to be correct:

  1. The device major numbers could change between kernel versions. This is not guaranteed — many drivers maintain stable major numbers across kernel updates — but NVIDIA's UVM and caps devices are dynamically allocated, making them susceptible to change. The assistant correctly hypothesized this as the root cause.
  2. The cgroup rules were the sole blocker. There was a possibility that other factors contributed — for instance, the nvidia_uvm module's HMM disable parameter not applying correctly, or a genuine kernel regression in the 6.14 driver compatibility layer. But the assistant's focused comparison of cgroup rules against actual device majors provided a clean, testable hypothesis.
  3. The bind-mount entries were correct. The assistant had already verified that the device nodes appeared inside the container with the right major numbers ([msg 1332]). This ruled out a bind-mount issue and narrowed the problem to access control. One potential oversight: the assistant did not immediately check whether the nvidia-caps device (major 237) was strictly necessary for CUDA operation. It's possible that only the UVM device (major 509) was the critical blocker, and the caps device was a secondary issue. However, fixing both was the prudent approach.

Input Knowledge Required

To fully understand this message, one needs:

Output Knowledge Created

This message produces several valuable outputs:

  1. A definitive root cause identification: The stale cgroup major numbers are confirmed as the reason for CUDA initialization failure after the kernel upgrade. This is not speculative — it is a direct comparison of configuration vs. reality.
  2. A reproducible diagnostic pattern: The method of comparing grep cgroup /etc/pve/lxc/129.conf against cat /proc/devices | grep nvidia is a reusable technique for diagnosing similar issues after any kernel upgrade on Proxmox hosts with GPU pass-through.
  3. Documentation of the kernel version's device map: The message records that kernel 6.14.11-5-bpo12-pve assigns major 509 to nvidia-uvm and major 237 to nvidia-caps. This is useful for future reference when debugging similar configurations.
  4. A clear action plan: The assistant's statement "Let me fix this" sets up the next steps: stop the container, update the cgroup rules with the correct major numbers, and restart. The subsequent message ([msg 1337]) executes this plan.

The Thinking Process

The assistant's reasoning, visible in the sequence of messages leading to [msg 1336], follows a textbook diagnostic chain:

  1. Observe symptom: CUDA initialization fails inside LXC container after kernel upgrade.
  2. Verify basics: Confirm NVIDIA driver loads on host, nvidia-smi works in container, device nodes exist.
  3. Isolate the failure mode: cuInit returns error 999, not the more common error 3.
  4. Compare configurations: Check the container's cgroup rules against the actual device major numbers on the new kernel.
  5. Identify the mismatch: Cgroup allows 504/507, reality has 509/237.
  6. Formulate hypothesis: The cgroup is blocking access to nvidia-uvm and nvidia-caps.
  7. Plan the fix: Update the cgroup rules to match the new major numbers. This is a textbook example of the "compare configuration to reality" debugging strategy. The assistant did not jump to conclusions — it systematically ruled out other possibilities (driver loading, module parameters, bind-mount correctness) before landing on the cgroup mismatch.

Broader Implications

Beyond the immediate fix, this message highlights a systemic fragility in GPU-accelerated container deployments: device major numbers are not stable across kernel upgrades. Any automated deployment system that hard-codes these numbers in container configurations will break when the kernel changes. The robust solution would be to either:

Conclusion

Message [msg 1336] is a small but pivotal moment in a much larger optimization journey. It demonstrates that even in a high-performance ML inference pipeline, the most mundane system administration details can become critical bottlenecks. The assistant's ability to connect the dots between a CUDA error code, a container configuration file, and a kernel's device major number table is a testament to the value of deep systems knowledge in AI infrastructure work. The fix itself is simple — updating three lines in a config file — but the diagnostic path to finding those three lines required a thorough understanding of Linux device management, LXC internals, and the NVIDIA driver stack.