The Moment CUDA Came Back: A Post-Kernel-Upgrade Debugging Victory

import torch
print("CUDA available:", torch.cuda.is_available())
print("Device count:", torch.cuda.device_count())
x = torch.randn(10, device="cuda:0")
print("Tensor on GPU:", x[:3])
print("CUDA OK!")

These six lines of Python, executed inside an LXC container running on a freshly upgraded Linux kernel, represent the culmination of a tense debugging session. The output — CUDA available: True, Device count: 8, Tensor on GPU: tensor([-0.0796, -2.2615, 0.0244], device='cuda:0') — is the all-clear signal after a kernel upgrade nearly broke GPU compute access for an 8-GPU machine running the GLM-5-NVFP4 large language model. This message ([msg 1341]) is the resolution of a crisis that began when a carefully planned kernel upgrade from 6.8.12 to 6.14.11 caused CUDA to fail with an opaque error code inside the Proxmox LXC container.

The Context: A Major Kernel Upgrade

To understand why this simple CUDA test matters, we need to step back. The assistant had been engaged in an intensive optimization campaign for the GLM-5-NVFP4 model inference on a machine with 8 NVIDIA RTX PRO 6000 Blackwell GPUs. A comprehensive system audit ([msg 1310] through [msg 1317]) had revealed multiple performance bottlenecks: a suboptimal CPU frequency driver (acpi-cpufreq instead of amd_pstate), an outdated kernel (6.8.12), enabled NUMA balancing, deep CPU C-states, and a PCIe MaxReadReq stuck at 512 bytes instead of 4096.

The fix required a major kernel upgrade to 6.14.11-5-bpo12-pve with custom boot parameters: amd_pstate=active, processor.max_cstate=1, and nmi_watchdog=0. The NVIDIA DKMS modules were rebuilt for the new kernel, the initramfs was updated, and the host was rebooted. The host came back perfectly ([msg 1316]): the new kernel was running, amd-pstate-epp driver was active, C-states were limited to C0/C1, all 8 GPUs were detected, and the PCIe tuning service had applied MaxReadReq=4096.

The Crisis: CUDA Initialization Fails

But when the assistant tried to run a P2P bandwidth benchmark inside the LXC container ([msg 1325]), the script crashed with a CUDA initialization error. The container had all 8 GPUs visible to nvidia-smi ([msg 1327]), but PyTorch could not initialize CUDA ([msg 1328]):

UserWarning: CUDA initialization: CUDA unknown error - this may be due to an 
incorrectly set up environment, e.g. changing env variable CUDA_VISIBLE_DEVICES 
after program start. Setting the available devices to be zero.

A direct cuInit call via the CUDA Python bindings returned error code 999 — CUDA_ERROR_UNKNOWN ([msg 1329]). This was a different error from the "error 3" (not initialized) that had been seen earlier in the session, making it a new and unfamiliar problem.

The Diagnostic Process

The assistant's debugging process reveals a systematic approach to diagnosing CUDA-in-LXC failures. The initial hypothesis was that the uvm_disable_hmm=1 modprobe option (required to disable NVIDIA's Heterogeneous Memory Management on this kernel) hadn't applied to the new kernel. This was quickly ruled out: the host confirmed uvm_disable_hmm=Y was active ([msg 1326]).

Next, the assistant checked for GPU errors in the host kernel log — none found. The host itself could run CUDA without issues. The device nodes inside the container looked correct: /dev/nvidia0 through /dev/nvidia7 with major number 195, and /dev/nvidia-uvm with major number 509 ([msg 1332]).

The breakthrough came when the assistant compared the LXC container's cgroup device access rules against the actual device major numbers on the new kernel ([msg 1333] through [msg 1336]). The container config (/etc/pve/lxc/129.conf) contained:

lxc.cgroup2.devices.allow: c 195:* rwm
lxc.cgroup2.devices.allow: c 504:* rwm
lxc.cgroup2.devices.allow: c 507:* rwm

But the new kernel's device major numbers had changed:

195 nvidia
237 nvidia-caps
238 nvidia-caps-imex-channels
509 nvidia-uvm

The cgroup rules allowed major numbers 504 and 507 — but the actual devices were at 509 (nvidia-uvm) and 237 (nvidia-caps). This mismatch meant the container's cgroup was blocking access to the UVM device, which is essential for CUDA memory management. The container could see the GPU devices (major 195 was correctly allowed) but could not interact with the UVM driver, causing cuInit to fail with the generic error 999.## The Fix: Updating Stale Cgroup Rules

The fix was straightforward once the root cause was identified. The assistant stopped the container ([msg 1337]), edited the configuration to remove the stale rules (504 and 507) and add the correct ones (509 for nvidia-uvm, 237 for nvidia-caps, and 238 for nvidia-caps-imex-channels) ([msg 1339]). The container was then restarted ([msg 1340]).

The critical insight here is that device major numbers are not guaranteed to be stable across kernel versions. The NVIDIA driver registers its device interfaces dynamically, and the kernel assigns major numbers based on allocation order and available ranges. When the kernel jumped from 6.8.12 to 6.14.11 — a span of roughly 16 kernel versions — the internal device registration order or numbering scheme shifted. The old cgroup rules, written for the 6.8 kernel, became silently wrong. This is a classic "works on my machine" pitfall that becomes invisible until a reboot: the container appeared healthy (all GPU devices were present, nvidia-smi worked), but the cgroup was quietly blocking the UVM channel that CUDA's runtime library needs to allocate and manage GPU memory.

The Verification: Message 1341

Message [msg 1341] is the verification step. After fixing the cgroup rules and restarting the container, the assistant waited 5 seconds (to ensure the container was fully booted) and then ran the simplest possible CUDA smoke test: import PyTorch, check if CUDA is available, count devices, allocate a tensor on GPU 0, and print its first three elements.

The output is unequivocal:

CUDA available: True
Device count: 8
Tensor on GPU: tensor([-0.0796, -2.2615,  0.0244], device='cuda:0')
CUDA OK!

Every line carries meaning. CUDA available: True confirms that cuInit succeeded — the UVM device is now accessible. Device count: 8 confirms all GPUs are visible to the CUDA runtime, not just to nvidia-smi. The tensor allocation on cuda:0 proves that memory allocation and kernel execution work end-to-end. The specific tensor values (-0.0796, -2.2615, 0.0244) are random noise from torch.randn, but their very existence is the proof that the entire GPU compute stack is functional.

Why This Message Matters

This message is a turning point in the broader session. The entire optimization campaign — kernel tuning, CPU governor changes, C-state limiting, PCIe MaxReadReq adjustment, NUMA balancing disable — was predicated on being able to run CUDA workloads. If CUDA remained broken after the kernel upgrade, all those optimizations would have been worthless. The assistant had invested hours in the kernel upgrade and system tuning; a failure here would have meant rolling back the kernel or debugging a much more complex NVIDIA driver compatibility issue.

The message also demonstrates a principle of debugging complex systems: when a high-level tool (PyTorch) fails with a generic error, drill down to the lowest level (cgroup device permissions) and verify the infrastructure layer. The assistant didn't try reinstalling PyTorch, rebuilding the CUDA toolkit, or blaming the NVIDIA driver. Instead, it checked the host kernel logs, verified device nodes, compared cgroup rules against actual device majors, and identified the exact mismatch. This systematic narrowing of the problem space — from "CUDA unknown error" to "cgroup blocks UVM major number 509" — is a textbook example of layered debugging.

Assumptions and Knowledge Required

To understand this message fully, one needs knowledge of several systems: LXC containerization (cgroup device access rules, the lxc.cgroup2.devices.allow syntax), Linux device major numbers (how kernel modules register character devices), NVIDIA's driver architecture (the separation between /dev/nvidia* for GPU control and /dev/nvidia-uvm for unified virtual memory management), and CUDA's initialization sequence (which requires access to both the GPU device and the UVM device).

The assistant made a key assumption that proved correct: that the device major numbers might have changed between kernel versions. This assumption was grounded in the observation that nvidia-smi worked (meaning the GPU control devices were accessible) but cuInit failed (meaning the UVM device was blocked). The mismatch between the working nvidia-smi and failing CUDA was the crucial signal that pointed to the cgroup rules rather than a deeper driver issue.

Output Knowledge Created

This message creates verified knowledge that the system is fully operational after the kernel upgrade. It confirms that:

  1. The kernel 6.14.11 upgrade is successful and compatible with NVIDIA driver 590.48.01
  2. The uvm_disable_hmm=1 modprobe option works correctly on the new kernel
  3. The cgroup fix (replacing major numbers 504/507 with 509/237/238) is correct
  4. All 8 GPUs are accessible from within the LXC container
  5. PyTorch can allocate tensors and execute GPU kernels This verification unlocks the next phase of work: running the P2P bandwidth benchmarks to confirm that the kernel tuning actually improved GPU-to-GPU communication performance, and ultimately benchmarking the GLM-5-NVFP4 model inference throughput on the optimized system.

Conclusion

Message [msg 1341] is a small output — just a few lines of Python output — but it represents the successful resolution of a complex, multi-layered debugging problem. A kernel upgrade intended to improve inference performance inadvertently broke CUDA access due to stale cgroup device rules. The assistant's systematic diagnosis and precise fix restored functionality, turning a potential disaster into a routine configuration update. The three tensor values [-0.0796, -2.2615, 0.0244] are not just random numbers; they are the digital all-clear that the system is ready for the next challenge.