The Quiet Culmination: A Single Command That Restores GPU Access After a Kernel Upgrade

The Message

[assistant] Now start the container and test: [bash] ssh root@10.1.2.6 'pct start 129 2>&1 && echo "Container started"' Container started

At first glance, this message ([msg 1340]) appears trivial — a simple Proxmox command to start an LXC container, followed by the expected confirmation. A reader skimming the conversation might dismiss it as a routine administrative action, barely worth a second thought. But this brevity is deceptive. The command is the final, quiet resolution of a multi-hour debugging saga that involved a major kernel upgrade, a subtle device-major-number mismatch, and a CUDA initialization failure that had rendered eight high-end NVIDIA GPUs inaccessible to the machine learning workload running inside the container. To understand why this message was written, one must trace the chain of events that led to it — a chain that reveals the hidden fragility of GPU passthrough in virtualized environments and the kind of systematic diagnostic thinking required to restore it.

The Context: A Major Kernel Upgrade

The story begins with a comprehensive system optimization effort. The assistant and user had been working for hours to tune a Proxmox host running Ubuntu 24.04 with eight NVIDIA RTX PRO 6000 Blackwell GPUs. The goal was to maximize inference throughput for the GLM-5-NVFP4 model using SGLang. A parallel audit (see [msg 1310] through [msg 1317]) had revealed multiple system-level misconfigurations: the CPU was using the acpi-cpufreq driver instead of the more performant amd_pstate, the kernel was outdated at 6.8.12, NUMA balancing was enabled, deep CPU C-states were active, and PCIe MaxReadReq was stuck at a suboptimal 512 bytes instead of 4096.

The team applied runtime fixes and executed a major kernel upgrade to version 6.14.11-5-bpo12-pve with aggressive tuning parameters: amd_pstate=active, processor.max_cstate=1, and nmi_watchdog=0. The reboot was successful, and the host came back with all eight GPUs detected, the new kernel running, and all sysctls properly applied. Everything appeared to be working perfectly.

The Unexpected Failure: CUDA Goes Silent

But when the assistant attempted to run a P2P bandwidth benchmark inside the LXC container (see [msg 1325]), the script crashed with a CUDA initialization error. This was deeply puzzling. The host itself had no issues — nvidia-smi worked fine, all eight GPUs were visible, and the NVIDIA driver had loaded without errors. Yet inside the container, PyTorch reported "CUDA unknown error" and cuInit returned error code 999 (CUDA_ERROR_UNKNOWN), a different error from the earlier error 3 (CUDA_ERROR_NOT_INITIALIZED) that had been resolved earlier in the session.

The assistant's initial assumption, documented in [msg 1326], was that the uvm_disable_hmm modprobe option might not have applied on the new kernel. This was a reasonable hypothesis — the HMM (Heterogeneous Memory Management) feature in the nvidia_uvm module had caused CUDA initialization failures earlier in the session, and a modprobe configuration file had been created to disable it. However, checking /sys/module/nvidia_uvm/parameters/uvm_disable_hmm confirmed that the parameter was correctly set to Y. The NVIDIA driver had loaded cleanly on the host.

This is where the debugging process demonstrates a crucial skill: when a hypothesis fails, the assistant does not double down but instead pivots to gather more information. It checked whether nvidia-smi worked inside the container (it did — see [msg 1327]), then attempted a simpler PyTorch CUDA test (which failed — see [msg 1328]), and finally used the low-level CUDA Python bindings to isolate the failure to cuInit itself (see [msg 1329]). The error code 999, CUDA_ERROR_UNKNOWN, was a strong signal that something was wrong at the kernel-driver boundary rather than in the application layer.

The Root Cause: Stale Device Major Numbers

The breakthrough came in [msg 1333] when the assistant checked the host's /proc/devices and compared it against the LXC container's cgroup configuration. The kernel upgrade had changed the major device numbers for NVIDIA's device nodes. Specifically:

nvidia-uvm had moved from major number 504 to 509
nvidia-caps had moved from major number 507 to 237
A new device, nvidia-caps-imex-channels, had appeared at major 238 But the LXC container configuration (/etc/pve/lxc/129.conf) still contained the old cgroup rules:

lxc.cgroup2.devices.allow: c 195:* rwm
lxc.cgroup2.devices.allow: c 504:* rwm
lxc.cgroup2.devices.allow: c 507:* rwm

The cgroup v2 device controller in the Linux kernel enforces access control based on device major and minor numbers. When a process inside the container attempts to open a device file, the cgroup controller checks whether the device's major number is in the allowed list. Because the old rules referenced major numbers 504 and 507 — which no longer corresponded to any NVIDIA device — the container's CUDA runtime was denied access to the UVM and caps devices. The GPU compute nodes (/dev/nvidia0 through /dev/nvidia7) were accessible because their major number (195) hadn't changed, but without access to the UVM device, CUDA's unified memory management and virtual memory subsystem could not initialize.

This is a subtle and easy-to-miss failure mode. The device files themselves appeared correctly inside the container — ls -la /dev/nvidia-uvm showed a valid device node with major 509 — but the cgroup controller silently blocked access. The error manifested as a generic "CUDA unknown error" because the CUDA driver's cuInit routine attempts to open and communicate with the UVM device as part of its initialization sequence, and when that open fails due to cgroup filtering, the error propagates as an opaque failure.

The Fix: Editing the Container Configuration

Once the root cause was identified, the fix was straightforward but required care. The assistant stopped the container using pct stop 129 (see [msg 1337]), then edited the configuration file to replace the stale cgroup rules with the correct ones (see [msg 1339]):

lxc.cgroup2.devices.allow: c 509:* rwm
lxc.cgroup2.devices.allow: c 237:* rwm
lxc.cgroup2.devices.allow: c 238:* rwm

The assistant also added a rule for major 238 (nvidia-caps-imex-channels), which was a new device that hadn't existed under the old kernel. This was a prudent addition — even if the current workload didn't require it, future CUDA operations might.

The Subject Message: Starting the Container

This brings us to the subject message ([msg 1340]). After the configuration was updated, the assistant issued the command pct start 129 to restart the container with the corrected cgroup rules. The output "Container started" confirmed that Proxmox had successfully launched the container.

The message is remarkable for what it doesn't say. There is no triumphant declaration, no detailed explanation of the debugging journey, no summary of the root cause. It is purely operational: "Now start the container and test." The assistant's focus has already shifted to the next step — verification. The debugging is complete; the fix is applied; now it's time to confirm that CUDA works.

This terseness reflects a mature engineering mindset. The assistant does not pause to celebrate or document the fix inline because the conversation itself is the documentation. Every step of the diagnostic process — the failed benchmark, the hypothesis about HMM, the systematic elimination of possibilities, the comparison of device major numbers, the discovery of the cgroup mismatch — is recorded in the preceding messages. The subject message is the logical conclusion of that narrative, not a standalone event.

The Verification

The next message ([msg 1341]) confirms the success:

CUDA available: True
Device count: 8
Tensor on GPU: tensor([-0.0796, -2.2615,  0.0244], device='cuda:0')
CUDA OK!

Eight GPUs are accessible. A tensor can be created on cuda:0. The container is fully operational on the new kernel, and the massive optimization effort — the kernel upgrade, the CPU governor change, the C-state restriction, the PCIe tuning — can now be benchmarked and evaluated.

Input Knowledge Required

To fully understand this message, a reader needs several pieces of background knowledge:

LXC container architecture: Understanding that LXC containers share the host kernel but have their own device namespace, and that access to host devices is controlled by bind mounts and cgroup device rules.
CUDA initialization sequence: Knowing that CUDA's cuInit requires access to multiple device nodes beyond just the compute GPUs (/dev/nvidia0 etc.), particularly the UVM (/dev/nvidia-uvm) device for unified memory management.
Linux device major numbers: Understanding that device major numbers are assigned dynamically by the kernel and can change between kernel versions or driver versions, and that cgroup v2 device access control uses these major numbers.
Proxmox container management: Familiarity with pct start, pct stop, and the LXC configuration file format under /etc/pve/lxc/.
The earlier session history: Knowing that the HMM issue had been a previous blocker, that the kernel was being upgraded from 6.8.12 to 6.14.11, and that the system had eight NVIDIA Blackwell GPUs configured for ML inference.

Output Knowledge Created

This message, in conjunction with the surrounding context, creates several important pieces of knowledge:

A documented failure mode: Kernel upgrades on Proxmox hosts with NVIDIA GPU passthrough can change device major numbers, breaking CUDA inside LXC containers even when the host appears healthy.
A diagnostic pattern: When CUDA fails inside a container but nvidia-smi works, check the cgroup device rules against the actual /proc/devices major numbers.
A remediation procedure: Stop the container, update the cgroup rules in the LXC config, and restart. The fix is non-disruptive to the host.
A preventive insight: Container configurations should be audited after kernel upgrades to ensure device major numbers remain valid. This is especially important for NVIDIA devices, where the UVM and caps major numbers can shift.

Assumptions and Potential Mistakes

The assistant made several assumptions during this debugging process, most of which were correct:

The assumption that the HMM fix was the likely culprit (in [msg 1326]) was reasonable but turned out to be incorrect. The assistant did not linger on this wrong hypothesis; instead, it gathered more data and pivoted quickly.
The assumption that the device files themselves were correctly exposed was validated by the ls -la output inside the container, which showed the correct major numbers. This ruled out a bind-mount issue and narrowed the search to cgroup filtering.
The assumption that the cgroup rules were the problem was confirmed by comparing the allowed major numbers against /proc/devices. This was the key insight. One potential oversight: the assistant did not initially check whether the cgroup rules had changed after the kernel upgrade. The debugging process took several messages (from [msg 1325] to [msg 1336]) to reach this conclusion. In retrospect, checking the cgroup configuration immediately after the CUDA failure would have been faster. However, this is understandable given that the earlier HMM fix had been the solution to a previous CUDA initialization problem, and it was natural to suspect a recurrence.

The Broader Significance

This message, for all its brevity, illustrates a fundamental truth about systems engineering: the most critical operations often look the most mundane. A single pct start command, issued after hours of diagnostic work, represents the moment when all the pieces click back into place. The container starts, the GPUs become accessible, and the optimization work can proceed.

The debugging process also demonstrates the importance of understanding the full stack — from the kernel's device major number assignment, through the cgroup v2 controller, through the NVIDIA driver's initialization sequence, to PyTorch's CUDA abstraction layer. A failure at any level can manifest as a cryptic error at a higher level, and only systematic investigation can trace it back to the root cause.

For anyone managing GPU-accelerated workloads in LXC containers on Proxmox, this episode serves as a valuable case study. Kernel upgrades are routine, but their side effects on device passthrough can be subtle and easily missed. The next time CUDA fails inexplicably after a kernel update, checking the cgroup device rules against /proc/devices should be one of the first diagnostic steps — not the last.