The Missing Device Node: Diagnosing a Post-Reboot CUDA Initialization Failure in an LXC Container

Introduction

In the middle of a high-stakes benchmarking session for speculative decoding on eight NVIDIA RTX PRO 6000 Blackwell GPUs, a host reboot shattered the environment. The assistant had been methodically running throughput benchmarks for the Kimi K2.6 model when the machine went down for networking maintenance. When it came back, CUDA was broken. Message [msg 11306] captures a pivotal moment in the debugging effort: the assistant has narrowed the failure to a fundamental CUDA initialization problem and is executing a carefully designed set of diagnostic commands to pinpoint the root cause. This message is the turning point where a vague "CUDA unknown error" begins to crystallize into a concrete, fixable infrastructure issue.

The Context: A Fragile Container Environment

The assistant is operating on CT200, a Proxmox LXC container hosting eight RTX PRO 6000 Blackwell GPUs. This is not a bare-metal machine — it is a lightweight container with GPU passthrough, where the NVIDIA driver runs on the host and the container accesses GPU functionality through device nodes (/dev/nvidia0, /dev/nvidia-uvm, etc.) and kernel interfaces (/proc/driver/nvidia/, /sys/module/nvidia/). This architecture is powerful but brittle: the container's GPU access depends on the host's driver state and on proper configuration of cgroup device permissions.

The session leading up to [msg 11306] is a textbook example of layered debugging. The assistant began by observing that the SGLang server crashed with "Failed to get device capability: CUDA unknown error" ([msg 11298]). It then verified that nvidia-smi works — the driver is loaded, all eight GPUs are visible, and their compute mode is Default. But every attempt to initialize CUDA from Python failed, whether using PyTorch 2.11.0+cu130 or 2.11.0+cu128. The raw CUDA driver API call cuInit(0) returned error code 999 (CUDA_ERROR_UNKNOWN), while cuDriverGetVersion succeeded and reported version 13020 (CUDA 13.2). This is a deeply strange failure mode: the driver API can report its version but cannot create a CUDA context.

By [msg 11305], the assistant had correctly identified that this is a system-level issue, not a CUDA runtime version mismatch. It also discovered that the container cannot reload the nvidia_uvm kernel module because the module files don't exist inside the container's filesystem — they belong to the host's kernel. The assistant was running out of options that could be exercised from inside the container.

The Message: A Deliberate Diagnostic Probe

Message [msg 11306] opens with the assistant's reasoning, which synthesizes everything learned so far:

The cuDriverGetVersion succeeds and reports version 13020 (CUDA 13.2), but cuInit returns 999 (CUDA_ERROR_UNKNOWN). The GPUs are in P8 (idle) state, compute mode is Default, ECC is enabled.

>

This is a fundamental GPU initialization issue. The driver API can report the version but can't actually initialize the CUDA context. This happens in containers when: 1. The GPU firmware needs to be re-initialized by the host 2. The container was started before the GPU driver was fully initialized on the host 3. Critical /proc or /sys paths aren't exposed to the container

This reasoning is significant because it represents a hypothesis refinement. Earlier messages had explored version mismatches, GPU resets, and persistence mode — all reasonable guesses that turned out to be wrong. Here, the assistant correctly pivots to container-specific failure modes. The three hypotheses are well-grounded in the known behavior of NVIDIA's driver stack in virtualized environments:

  1. GPU firmware re-initialization: After a host reboot, GPU firmware (including ECC initialization and memory training) may not have completed. The driver can report device presence but CUDA context creation requires fully initialized hardware state.
  2. Timing of container startup: If the container started before the host's NVIDIA driver finished loading, the device nodes might exist but the driver's internal state machine might not be ready for CUDA calls.
  3. Missing /proc or /sys paths: CUDA's initialization path reads driver state from /proc/driver/nvidia/ and /sys/module/nvidia/. If these are not exposed to the container (or if specific files are missing), cuInit can fail. The assistant then designs a diagnostic command that tests all three hypotheses simultaneously. The command checks: - The contents of /proc/driver/nvidia/ (capabilities, gpus, params, patches, registry, version, etc.) - The NVRM version string - The list of GPU PCI addresses - The /sys/module/nvidia/ directory - The existence of /dev/nvidia-modeset (a device node for display modeset operations, major 195, minor 254) - A strace of cuInit to capture the exact system call that fails The inclusion of strace is the most important decision in this message. After several rounds of indirect diagnosis (checking library paths, trying different venvs, attempting module reloads), the assistant finally decides to trace the actual system calls made during CUDA initialization. This is the diagnostic equivalent of turning on a flashlight in a dark room — it will reveal exactly which file open or ioctl operation fails, rather than relying on the opaque "unknown error" message.

The Output: Clues Emerge

The command output reveals several important facts:

First, /proc/driver/nvidia/ is fully accessible and contains all expected entries. The NVRM version is confirmed as 595.71.05, and all eight GPUs are listed with their PCI addresses (0000:01:00.0 through 0000:f1:00.0). The /sys/module/nvidia/ directory is also present with the expected subdirectories (coresize, drivers, holders, initsize, initstate). This rules out hypothesis #3 — the proc and sys interfaces are working.

Second, /dev/nvidia-modeset is missing:

ls: cannot access '/dev/nvidia-modeset': No such file or directory

This is a potentially significant finding. The nvidia-modeset device node is created by the nvidia_modeset kernel module and is used for display mode-setting operations. While not strictly required for CUDA compute workloads (CUDA can function without it), its absence is unusual and suggests that the container's device passthrough configuration may be incomplete. The assistant notes this as a clue but does not yet know if it is the root cause.

Third, the strace output is captured to /tmp/cuda_strace.log, but only a filtered grep is shown in the message output. The full strace will be examined in the next message ([msg 11307]), where the assistant finds the smoking gun:

openat(AT_FDCWD, "/dev/nvidia-uvm", O_RDWR|O_CLOEXEC) = -1 EPERM (Operation not permitted)

The strace output is truncated in [msg 11306], but the assistant's decision to run it is what ultimately solves the mystery.

Assumptions and Their Validity

The assistant makes several assumptions in this message, some correct and some not:

Correct assumption: The issue is container-specific and relates to device access. The assistant correctly reasons that since nvidia-smi works but cuInit fails, the problem must be in the CUDA context creation path, which involves opening device nodes and communicating with the kernel driver.

Partially correct assumption: The missing /dev/nvidia-modeset might be the cause. While its absence is a symptom of incomplete device passthrough, it is not the primary blocker — CUDA can initialize without it. The real blocker is /dev/nvidia-uvm, which has different device major number (511 vs 195).

Incorrect assumption: The GPU firmware might need re-initialization. This hypothesis is reasonable but turns out to be wrong. The GPUs are in P8 (idle) state with Default compute mode and ECC enabled — they are fully initialized. The failure is purely at the container access-control layer.

Incorrect assumption: The /proc and /sys paths might be missing. This hypothesis is also reasonable for LXC containers, which can restrict filesystem access. But in this case, the container has full access to these interfaces. The test conclusively rules this out.

Input Knowledge Required

To fully understand [msg 11306], a reader needs knowledge spanning several domains:

  1. CUDA driver architecture: The distinction between the user-mode CUDA driver (libcuda.so), the kernel-mode NVIDIA driver (nvidia.ko), and the CUDA runtime (libcudart.so). The fact that cuDriverGetVersion works but cuInit fails indicates that the user-mode driver can load but cannot establish a channel to the kernel driver.
  2. LXC container device passthrough: LXC containers access host devices through cgroup device permissions. Devices are identified by major and minor numbers (e.g., major 195 for NVIDIA compute devices, major 511 for nvidia-uvm). Missing or incorrect cgroup entries can block device access even when the device node exists with permissive file permissions.
  3. NVIDIA device node hierarchy: The NVIDIA driver creates multiple device nodes: /dev/nvidia0–/dev/nvidiaN (one per GPU, major 195), /dev/nvidiactl (control device, major 195, minor 255), /dev/nvidia-uvm (unified virtual memory, major 511), /dev/nvidia-modeset (modeset, major 195, minor 254), and /dev/nvidia-uvm-tools (UVM tools, major 511). Each serves a different purpose in the CUDA stack.
  4. strace interpretation: The strace command traces system calls. The assistant uses it to capture open, openat, and ioctl calls, filtering for NVIDIA-related operations. The ability to interpret strace output is critical to understanding the debugging process.
  5. Proxmox/PVE container management: The container runs on Proxmox VE with kernel 6.14.11-9-bpo12-pve. Proxmox uses cgroup v2 for device access control, which differs from cgroup v1 in how device permissions are managed.

Output Knowledge Created

This message produces several pieces of actionable knowledge:

  1. Confirmed working paths: /proc/driver/nvidia/ and /sys/module/nvidia/ are fully accessible, ruling out hypothesis #3. This saves the assistant from pursuing a dead end.
  2. Missing device node: /dev/nvidia-modeset is absent, which is a clue pointing toward incomplete device passthrough. While not the root cause, it narrows the search space.
  3. Strace log: The captured strace output (to be analyzed in [msg 11307]) contains the definitive evidence: EPERM on /dev/nvidia-uvm. This transforms the problem from "CUDA unknown error" to a specific, fixable issue: the container's cgroup v2 policy blocks device major 511.
  4. Container identity confirmed: The assistant confirms this is a Proxmox LXC container with GPU passthrough, which determines the fix path (host-side configuration change rather than container-side workaround).

The Thinking Process: A Methodological Case Study

The reasoning in [msg 11306] exemplifies systematic debugging under uncertainty. The assistant:

  1. Synthesizes available evidence: cuInit returns 999, cuDriverGetVersion succeeds, GPUs are idle and in Default mode.
  2. Generates plausible hypotheses: Three container-specific failure modes are enumerated, each grounded in known NVIDIA/LXC behavior.
  3. Designs a discriminating test: The bash command simultaneously checks all three hypotheses, plus adds a strace for open-ended investigation. This is efficient — one command can rule out multiple possibilities.
  4. Interprets results carefully: The missing /dev/nvidia-modeset is noted but not over-interpreted. The assistant does not jump to conclusions; it waits for the strace output to provide definitive evidence.
  5. Recognizes the limits of container-side action: The assistant understands that if the issue is host-side (cgroup permissions or GPU firmware), it cannot fix it from inside the container. This leads to the eventual escalation to the user in [msg 11308].

Conclusion

Message [msg 11306] is a masterclass in diagnostic reasoning. It sits at the inflection point of a debugging session that began with a cryptic "CUDA unknown error" and ends with a precise diagnosis: the LXC container's cgroup v2 policy blocks access to /dev/nvidia-uvm (major 511). The assistant's decision to run strace on cuInit is the key insight — it transforms an opaque error into a concrete permission problem. The message also demonstrates the importance of understanding the full stack, from GPU firmware to container device policies, when debugging accelerated computing environments. The missing device node is not just a technical detail; it is a reminder that in virtualized GPU environments, the boundary between host and container is where the most subtle failures live.