The Cgroup Repair: How a Kernel Upgrade Broke CUDA in an LXC Container

Introduction

In the complex ecosystem of high-performance GPU computing, a kernel upgrade is never a trivial affair. When the assistant upgraded the Proxmox host from kernel 6.8.12 to 6.14.11—a leap that brought amd_pstate=active CPU frequency scaling, reduced C-state latency, and other critical performance improvements for the EPYC Turin processor—the reboot introduced an unexpected casualty: CUDA initialization inside the LXC container running the GLM-5-NVFP4 inference workload. Message 1339 captures the precise moment when the assistant diagnosed and repaired the root cause: stale cgroup device major numbers that no longer matched the new kernel's device allocation.

This article examines that single message in depth, exploring the reasoning that led to it, the assumptions that were tested and discarded along the way, and the systems knowledge required to understand why a kernel upgrade could silently break GPU passthrough to a container.

The Context: A High-Stakes Performance Tuning Session

The broader session was a deep optimization campaign for inference throughput on two NVIDIA RTX PRO 6000 Blackwell GPUs (later expanded to eight). The team had already achieved remarkable results—pushing throughput from ~880 to ~3,740 tok/s through FlashInfer CUTLASS MoE autotuning—but a massive efficiency gap remained. The theoretical maximum single-stream performance was calculated at 309 tok/s, yet actual throughput was a mere 10.36 tok/s, representing only 3.4% efficiency.

To close this gap, the assistant launched a comprehensive system audit via ten parallel agents. The audit uncovered multiple misconfigurations: a suboptimal CPU governor (acpi-cpufreq instead of amd_pstate), enabled NUMA balancing, deep CPU C-states, and a PCIe MaxReadReq stuck at 512 bytes instead of 4096. The prescribed fix was a major kernel upgrade to 6.14.11 with aggressive tuning parameters (amd_pstate=active, processor.max_cstate=1, nmi_watchdog=0).

The upgrade was executed, the host rebooted, and everything appeared healthy—until the assistant tried to run a P2P bandwidth benchmark inside the LXC container and was met with a CUDA initialization failure.

The Debugging Trail: Seven Steps to the Root Cause

The subject message (msg 1339) is the culmination of a systematic debugging chain that spans messages 1325 through 1338. Understanding why this message was written requires tracing that chain.

Step 1: The initial failure. At msg 1325, the P2P benchmark script crashed with a traceback during measure_p2p(). The error was truncated in the output, but it was clearly a CUDA initialization failure.

Step 2: First hypothesis—HMM parameter. The assistant initially suspected the uvm_disable_hmm=1 modprobe parameter might not have applied to the new kernel. This was a reasonable guess: the HMM (Host Memory Management) feature in the NVIDIA UVM driver had caused issues before, and a kernel upgrade could theoretically reset module parameters. However, checking /sys/module/nvidia_uvm/parameters/uvm_disable_hmm confirmed it was set to Y.

Step 3: Narrowing the scope. The assistant checked nvidia-smi inside the container—it worked fine, listing all 8 GPUs. This ruled out a basic driver or device enumeration problem. But when PyTorch tried to initialize CUDA, it failed with a "CUDA unknown error" warning. This was a crucial clue: nvidia-smi uses the NVIDIA Management Library (NVML) which communicates through a different path than CUDA's runtime library. NVML worked, but CUDA did not.

Step 4: Raw CUDA initialization. Using the Python CUDA bindings, the assistant called cuInit(0) directly, which returned error code 999—CUDA_ERROR_UNKNOWN. This was different from the earlier "error 3" (CUDA_ERROR_NOT_INITIALIZED) seen in previous sessions, suggesting a new class of problem.

Step 5: Checking the host. The assistant verified the host itself could run CUDA (it could) and checked dmesg for GPU errors (none found). This confirmed the problem was container-specific.

Step 6: Device node inspection. Both the host and container had correct /dev/nvidia* device nodes. The major numbers appeared consistent at first glance.

Step 7: The breakthrough. The assistant checked /proc/devices on the host and found that the new kernel had assigned major number 509 to nvidia-uvm and 237 to nvidia-caps. Then, inspecting the LXC container configuration at /etc/pve/lxc/129.conf revealed the mismatch: the cgroup device allow rules still referenced major numbers 504 and 507—leftovers from the old kernel.

This was the root cause. The LXC container's cgroup was blocking access to the UVM and caps devices because the cgroup whitelist specified the wrong major numbers. CUDA's runtime library needs access to /dev/nvidia-uvm (for Unified Virtual Memory management) and /dev/nvidia-caps (for GPU capabilities). When the cgroup denies access to these devices, cuInit fails with a generic unknown error because the driver can't complete its initialization sequence.

The Subject Message: A Precise Surgical Fix

Message 1339 is the fix itself. The assistant executes a single bash command on the Proxmox host:

ssh root@10.1.2.6 'sed -i "/lxc.cgroup2.devices.allow: c 504:\* rwm/d; /lxc.cgroup2.devices.allow: c 507:\* rwm/d" /etc/pve/lxc/129.conf && echo "Removed old rules" && cat >> /etc/pve/lxc/129.conf << EOF
lxc.cgroup2.devices.allow: c 509:* rwm
lxc.cgroup2.devices.allow: c 237:* rwm
lxc.cgroup2.devices.allow: c 238:* rwm
EOF
echo "Added new rules" && echo "" && grep cgroup /etc/pve/lxc/129.conf'

This command does three things:

  1. Removes the two stale cgroup rules (major numbers 504 and 507) using sed with deletion patterns.
  2. Appends three new rules: one for nvidia-uvm (major 509), one for nvidia-caps (major 237), and one for nvidia-caps-imex-channels (major 238). The c prefix indicates character devices, * allows all minor numbers, and rwm grants read, write, and mknod permissions.
  3. Verifies the updated configuration by grepping the cgroup lines. The inclusion of major 238 (nvidia-caps-imex-channels) is a nice touch—it shows the assistant was being thorough. The IMEX (In-Memory Export) channels are used for inter-process GPU memory sharing, and while not strictly required for basic CUDA, having them available prevents future issues.

Why This Message Matters

This message is a masterclass in systematic debugging and understanding the Linux device model. Several aspects are worth highlighting:

The Assumption That Almost Led Astray

The assistant's initial hypothesis—that the HMM parameter hadn't applied—was a reasonable but incorrect assumption. It was based on past experience (the HMM feature had caused issues before) and the natural tendency to suspect the most recently changed configuration. What saved time was the willingness to verify the assumption quickly and move on when it proved wrong.

The Critical Distinction Between NVML and CUDA

One of the most important diagnostic insights was recognizing that nvidia-smi working inside the container did not mean CUDA would work. NVML and CUDA use different device access paths. NVML primarily uses /dev/nvidiactl and the NVIDIA control device, while CUDA's runtime library needs access to /dev/nvidia-uvm for memory management. This distinction is subtle and often overlooked, but it was key to isolating the problem.

Understanding LXC Cgroup Device Whitelisting

The Proxmox LXC container uses cgroup v2 device controllers to restrict which device nodes the container can access. The lxc.cgroup2.devices.allow directive specifies device major:minor ranges that are permitted. When a kernel upgrade changes the major number assignment for a driver (which can happen when modules are loaded in a different order or new drivers are added), the cgroup rules become stale. This is a known fragility in LXC GPU passthrough that isn't always documented.

The Input Knowledge Required

To understand and write this fix, one needs:

The Output Knowledge Created

This message produces several valuable outputs:

  1. A repaired LXC container configuration that correctly maps to the new kernel's device layout
  2. A documented verification step (the grep output confirms the fix)
  3. An implicit lesson for future kernel upgrades: always verify cgroup device major numbers match the running kernel
  4. A demonstration of the fix pattern for others facing similar issues

The Thinking Process Visible in the Reasoning

The assistant's reasoning, visible across the preceding messages, follows a classic differential diagnosis pattern:

  1. Hypothesis generation: "Maybe the HMM parameter didn't apply" → tested and rejected.
  2. Symptom refinement: "nvidia-smi works but PyTorch CUDA doesn't" → narrows the problem to CUDA-specific initialization.
  3. Direct probing: "Let me call cuInit directly" → confirms it's a CUDA-level failure, not a PyTorch issue.
  4. Boundary testing: "Does CUDA work on the host?" → establishes the problem is container-specific.
  5. Resource inspection: "What devices are available in the container?" → checks device nodes.
  6. Configuration audit: "What does /proc/devices show?" → discovers the new major numbers.
  7. Cross-reference: "What does the LXC config allow?" → finds the mismatch.
  8. Fix and verify: Applies the correction and confirms the result. This is textbook troubleshooting: each step eliminates a class of possible causes and generates a more specific hypothesis for the next round.

Conclusion

Message 1339 is deceptively simple—a single bash command that edits a configuration file. But behind that command lies a chain of reasoning that traversed the entire NVIDIA software stack, from NVML to CUDA runtime to kernel driver to LXC cgroup controller. It demonstrates that in complex GPU computing environments, a kernel upgrade can break things in unexpected ways, and that understanding the Linux device model is essential for diagnosing these failures.

The fix itself is elegant: remove the stale rules, add the correct ones, verify. No container rebuild, no driver reinstall, no filesystem manipulation—just a configuration update that tells the kernel's cgroup controller to allow access to the right devices. In less than a minute, the assistant restored full CUDA functionality to the container, enabling the inference benchmarks to resume.

For anyone working with GPU passthrough in containerized environments, this message serves as a valuable case study: when CUDA fails after a kernel upgrade, always check your cgroup device major numbers first.