The Silent Mismatch: How a Kernel Upgrade Broke CUDA in an LXC Container

A Single Grep That Uncovered the Root Cause

In the midst of an intense performance optimization session for GLM-5-NVFP4 inference on a dual-socket EPYC Turin machine with eight RTX PRO 6000 Blackwell GPUs, the assistant encountered a puzzling failure. After a major kernel upgrade from 6.8.12 to 6.14.11 — accompanied by a suite of system tuning changes including amd_pstate=active, processor.max_cstate=1, and PCIe MaxReadReq adjustments — the host rebooted successfully, but the LXC container running the inference workload could no longer initialize CUDA. The error was cryptic: CUDA_ERROR_UNKNOWN (error code 999). Every surface-level check passed: nvidia-smi worked, device nodes existed, the NVIDIA driver loaded. The problem was invisible to standard diagnostics. The breakthrough came in a single, deceptively simple message at index 1335:

[assistant] [bash] ssh root@10.1.2.6 'grep cgroup /etc/pve/lxc/129.conf'
lxc.cgroup2.devices.allow: c 195:* rwm
lxc.cgroup2.devices.allow: c 504:* rwm
lxc.cgroup2.devices.allow: c 507:* rwm

This message is a bash command executed on the Proxmox host that greps the LXC container configuration file for cgroup device access rules. The output reveals three cgroup2.devices.allow entries, each permitting access to character devices of a specific major number: 195 (for the nvidia family of devices — GPUs, modeset, nvidiactl), 504, and 507. On its face, this looks like a routine configuration check. But in context, it was the diagnostic linchpin that solved a multi-hour debugging session.

Why This Message Was Written

The assistant was in a debugging spiral. After the kernel upgrade and reboot, every attempt to use CUDA from within the LXC container failed. The assistant had already verified that the host could see all eight GPUs, that nvidia-smi inside the container reported driver version 590.48.01 and CUDA 13.1, and that the device nodes /dev/nvidia0 through /dev/nvidia7, /dev/nvidia-uvm, and /dev/nvidia-caps/* all existed with correct permissions. Yet PyTorch's torch.cuda.is_available() returned False, and direct cuInit() calls returned error 999.

The assistant had already checked /proc/devices on the host in the preceding message ([msg 1333]) and discovered that the new kernel had assigned different major numbers to NVIDIA devices than the old kernel. Specifically:

How Decisions Were Made

The decision to check the cgroup configuration was the result of a systematic narrowing-down process. The assistant had already eliminated several possible causes:

  1. Driver loading failure: Ruled out because nvidia-smi worked on the host and inside the container.
  2. Missing device nodes: Ruled out because ls -la /dev/nvidia* showed all expected devices.
  3. HMM (Host Memory Mapping) issue: Ruled out because uvm_disable_hmm=Y was confirmed active.
  4. Permission problems: Ruled out because device nodes had crw-rw-rw- (world-readable/writable). The remaining suspect was the cgroup device controller, which in Proxmox LXC containers restricts which device major/minor numbers the container's processes can access. The assistant's reasoning was: if the cgroup allows major 504 but the kernel now assigns major 509 to nvidia-uvm, then any process inside the container trying to open /dev/nvidia-uvm (which has major 509) would be denied by the cgroup — even though the file node itself exists and has correct permissions. This would manifest as a CUDA_ERROR_UNKNOWN because CUDA's cuInit internally opens the UVM device to set up the unified virtual memory subsystem. The assistant chose to run grep cgroup /etc/pve/lxc/129.conf rather than checking the running container's cgroup state directly (e.g., via cat /proc/self/cgroup from inside the container) because the container was currently stopped — the assistant had already stopped it earlier during troubleshooting. The persistent configuration file was the authoritative source.

Assumptions Made by the Assistant

Several assumptions underpinned this diagnostic step:

Assumption 1: The cgroup device allow list is the mechanism blocking CUDA. This assumed that Proxmox's LXC implementation enforces device access via cgroup v2 device controllers, and that the rules in /etc/pve/lxc/129.conf are the sole determinant of what devices the container can access. This is correct for unprivileged containers, but the container in question was configured as privileged (unprivileged: 0). In privileged LXC containers, cgroup device rules still apply, but the container has broader capabilities. The assistant implicitly assumed that privileged status did not bypass cgroup device restrictions — a correct assumption, as cgroup v2 device controllers apply regardless of container privilege level.

Assumption 2: The major numbers changed due to the kernel upgrade. This assumed that kernel version changes can alter the dynamic major number assignment for NVIDIA drivers. This is plausible because NVIDIA's nvidia-uvm and nvidia-caps drivers request dynamic major number allocation from the kernel, and the assigned numbers depend on the order of driver registration and the availability of unused majors. A kernel upgrade that changes the driver registration order or the set of pre-allocated majors can indeed shift these numbers.

Assumption 3: The old cgroup rules (504, 507) were correct under the old kernel. This assumed that the container was created and configured when those major numbers were valid, and that they had not been manually set to incorrect values. Given that the container had been working before the reboot, this was a safe assumption.

Assumption 4: The cgroup rules are the only thing that changed. The assistant assumed that no other aspect of the container configuration (e.g., bind-mount paths, AppArmor profiles, seccomp filters) was affected by the kernel upgrade. This was reasonable given that the container's filesystem and configuration file were on ZFS and persisted across reboots.

Mistakes or Incorrect Assumptions

While the assistant's diagnosis was ultimately correct, there were subtle aspects worth examining:

The assistant initially assumed the problem was with CUDA initialization itself, spending time testing cuInit() directly and checking PyTorch's CUDA detection. This was a reasonable path, but it delayed the realization that the issue was at the cgroup level. The breakthrough came only after checking /proc/devices on the host and noticing the major number discrepancy. A faster path might have been to check the cgroup rules immediately after confirming that device nodes existed but CUDA still failed.

The assistant assumed that nvidia-smi working inside the container meant device access was fully functional. In reality, nvidia-smi primarily uses the NVIDIA driver control device (/dev/nvidiactl, major 195) and the GPU devices (/dev/nvidia[0-7], also major 195). It does not necessarily require access to nvidia-uvm for basic queries. The UVM device is needed for CUDA's virtual memory management, which is why cuInit failed while nvidia-smi succeeded. This distinction is subtle and easy to miss.

There was an implicit assumption that the kernel upgrade would not change major numbers. This is a common oversight. Dynamic major numbers are not guaranteed to be stable across kernel versions, especially when the kernel's device registration order changes or when new kernel subsystems are introduced. The assistant had no reason to expect this change, but it highlights the importance of verifying all device-related configuration after a kernel upgrade in containerized environments.

Input Knowledge Required

To understand this message and its significance, one needs knowledge of several interconnected systems:

LXC container device passthrough: Proxmox uses LXC (Linux Containers) with cgroup v2 device controllers. The lxc.cgroup2.devices.allow entries in the container config specify which device major/minor numbers the container's processes can access. The format c 195:* rwm means "allow character devices with major number 195, any minor number, with read/write/mknod permissions."

Linux device major numbers: In Linux, device drivers register with the kernel and are assigned major numbers — either statically (for well-established drivers) or dynamically. The NVIDIA driver uses dynamic major number allocation for nvidia-uvm and nvidia-caps. These numbers can change across kernel versions or driver reloads.

CUDA initialization sequence: CUDA's cuInit() function does more than just open GPU device files. It initializes the CUDA driver API, which includes opening the UVM device (/dev/nvidia-uvm) for unified memory management. If the UVM device is inaccessible, cuInit fails with a generic error.

Proxmox container configuration: The file /etc/pve/lxc/<VMID>.conf is the persistent configuration for an LXC container on Proxmox. It includes both LXC directives (like lxc.cgroup2.devices.allow) and Proxmox-specific options. Changes to this file take effect on container restart.

The relationship between kernel upgrades and device numbers: Understanding that dynamic major numbers are not guaranteed to be stable across kernel versions is crucial. This is especially relevant for NVIDIA drivers, which register multiple device classes (nvidia, nvidia-uvm, nvidia-caps, nvidia-caps-imex-channels) with dynamically assigned majors.

Output Knowledge Created

This message produced several important pieces of knowledge:

Direct output: The three cgroup allow rules showing major numbers 195, 504, and 507. This was the concrete evidence needed to confirm the mismatch hypothesis.

Diagnostic confirmation: The message confirmed that the container's cgroup configuration had not been updated after the kernel upgrade. The presence of majors 504 and 507 — which no longer corresponded to any NVIDIA device — was the definitive proof.

Actionable next step: The output immediately told the assistant what needed to be done: update the cgroup rules to match the new major numbers (509 for nvidia-uvm, 237 for nvidia-caps, 238 for nvidia-caps-imex-channels). This led directly to the fix in the subsequent messages ([msg 1336] through [msg 1339]), where the assistant stopped the container, edited the config file, and restarted.

Generalizable lesson: The message implicitly documented a class of failure that can occur after kernel upgrades in containerized GPU environments: cgroup device rules become stale when driver major numbers change. This knowledge is valuable for anyone managing GPU-accelerated LXC containers on Proxmox or similar platforms.

Debugging methodology: The message demonstrates a systematic approach to debugging: when a subsystem fails despite all surface-level checks passing, look at the access control layer. The cgroup device controller is an often-overlooked component that can silently block device access even when file nodes exist and permissions appear correct.

The Thinking Process Visible in the Reasoning

The assistant's thinking process, visible across the sequence of messages leading up to and following this one, reveals a methodical diagnostic approach:

  1. Observe the symptom: CUDA initialization fails inside the LXC container after a kernel upgrade and reboot.
  2. Verify basic connectivity: Check that the container is running, the kernel is correct, and GPUs are detected by nvidia-smi.
  3. Eliminate common causes: Test device node existence, permissions, driver module parameters, and host-side GPU health.
  4. Form a hypothesis: The error 999 (CUDA_ERROR_UNKNOWN) is unusual. Previous CUDA errors in this session were error 3 (CUDA_ERROR_NOT_INITIALIZED). The change in error code suggests a different root cause.
  5. Gather evidence: Check /proc/devices on the host to see the actual major numbers assigned to NVIDIA devices. Discover that they differ from what the container's cgroup allows.
  6. Test the hypothesis: Run grep cgroup /etc/pve/lxc/129.conf to see what the container is configured to allow. The output confirms the mismatch.
  7. Act on the finding: Stop the container, edit the config to replace old majors with new ones, add the missing caps majors, and restart. This chain of reasoning is a textbook example of differential diagnosis in systems engineering. Each step eliminates a possible cause and narrows the search space. The key insight — that the cgroup was blocking UVM access — came from noticing that the error code changed after the kernel upgrade and connecting that to the dynamic major number assignment.

Conclusion

The message at index 1335 is a masterclass in diagnostic precision. On its surface, it is a trivial grep command. In context, it is the moment of clarity after a long debugging session — the single command that transforms confusion into understanding. It reveals a failure mode that is both subtle and consequential: kernel upgrades can silently break GPU acceleration in containerized environments by changing device major numbers that are baked into cgroup configuration files. The fix is simple once the root cause is identified, but finding that root cause requires deep knowledge of Linux device management, LXC internals, CUDA initialization, and the Proxmox virtualization stack.

This message also illustrates a broader truth about systems engineering: the most impactful diagnostic commands are often the simplest. A one-line grep, executed at the right moment with the right hypothesis, can save hours of fruitless debugging. The assistant's ability to connect the dots between a changed error code, a kernel upgrade, and a cgroup configuration file is what separates effective troubleshooting from aimless tinkering.