The cuInit 999: Tracing a CUDA Initialization Failure Through Diagnostic Reasoning

Introduction

In any machine learning engineering workflow, few moments are as jarring as the one where a perfectly functioning GPU environment suddenly refuses to initialize CUDA. This is precisely the situation facing the assistant in message [msg 11302] of an extended benchmarking session on an 8× RTX PRO 6000 Blackwell GPU system. The conversation leading up to this point had been a methodical, data-driven investigation of speculative decoding performance—benchmarking Qwen3.6-27B with DFlash and DDTree, deploying Kimi K2.6, and tuning PCIe/NUMA/NCCL configurations. Then, after a host reboot for networking maintenance, the entire GPU stack collapsed into a cryptic error: "CUDA unknown error."

Message [msg 11302] is the diagnostic pivot point. It is not the message that fixes the problem—that comes several messages later when the assistant discovers the cgroup v2 device permission issue and the user provides host access to correct it ([msg 11309], [msg 11310], [msg 11311]). Rather, this message is where the assistant systematically eliminates plausible hypotheses, narrows the problem from a vague "unknown error" to a specific driver-level failure, and sets the stage for the strace-based investigation that ultimately reveals the root cause. It is a masterclass in diagnostic reasoning under uncertainty.

The Context: A Reboot Breaks Everything

To understand why this message was written, we must first understand the context. The assistant had been running a comprehensive benchmark suite for Qwen3.6-27B with DFlash and DDTree speculative decoding on the CT200 machine—a Proxmox LXC container with 8 NVIDIA RTX PRO 6000 Blackwell GPUs. The benchmarks had been progressing well, with DDTree budget 15 showing a 6.5× speedup over autoregressive decoding on a single GPU. Then came the host reboot.

In [msg 11297], the assistant attempted to resume the TP1 (single-GPU) benchmarks and was greeted with:

!! SERVICE FAILED:
Failed to get device capability: CUDA unknown error - this may be due to an incorrectly set up environment, e.g. changing env variable CUDA_VISIBLE_DEVICES after program start.

This triggered an urgent diagnostic sequence spanning messages [msg 11298] through [msg 11308]. By the time we reach [msg 11302], the assistant has already verified that nvidia-smi works, that all 8 GPU device nodes exist in /dev/, that the NVIDIA kernel modules are loaded, and that the driver reports CUDA version 13.2. Yet every attempt to initialize CUDA from within the container fails.

The Message: A Systematic Diagnostic Sweep

Message [msg 11302] opens with a concise summary of what is known:

All 8 GPU device nodes exist, nvidia-smi works, driver is loaded. The CUDA runtime initialization is failing though.

This framing is important. The assistant has already ruled out the most obvious failure modes: missing device files, unloaded driver, or a completely dead GPU. The problem is more subtle—the driver interface is alive, but the CUDA runtime cannot establish a working context.

The assistant then articulates its leading hypothesis: a version mismatch between PyTorch's bundled CUDA runtime and the installed driver. The driver is 595.71.05 (CUDA 13.2), while the virtual environment has torch 2.11.0+cu130 (CUDA 13.0). The system also has CUDA 12.8 installed. The reasoning is careful: CUDA runtime versions are generally backward-compatible with newer drivers, so a version mismatch should not cause a complete initialization failure. This is correct—a version mismatch would typically produce error code 35 (CUDA_ERROR_INSUFFICIENT_DRIVER), not the mysterious error 999.

The assistant pivots to checking library paths and device permissions. It notices that /dev/nvidia-uvm-tools has no permissions at all (----------), which is suspicious. It also inspects the CUDA runtime library directories in the virtual environment:

libcheckpoint.so
libcublas.so.13
libcublasLt.so.13
libcudadevrt.a
libcudart.so.13
__init__.py
libcudart.so.12

The presence of both libcudart.so.13 (CUDA 13.0) and libcudart.so.12 (CUDA 12.8) hints at the complex library stacking that can occur in ML environments. The assistant then runs a diagnostic Python script with an explicit LD_LIBRARY_PATH to rule out library resolution issues. The script attempts torch.cuda.init() and, upon failure, falls back to a raw cuInit(0) call via ctypes:

import torch
try:
    torch.cuda.init()
    print(f"CUDA init OK, device count: {torch.cuda.device_count()}")
except Exception as e:
    print(f"CUDA init FAILED: {e}")
    import ctypes
    try:
        lib = ctypes.CDLL("libcuda.so")
        ret = lib.cuInit(0)
        print(f"cuInit returned: {ret}")
    except Exception as e2:
        print(f"cuInit failed: {e2}")

The output confirms the failure, but critically, it shows that cuInit(0) returns error code 999. This is the key diagnostic finding of the entire message. Error 999 is CUDA_ERROR_UNKNOWN—a catch-all for driver-level failures that don't fit into more specific error categories. This tells the assistant that the problem is not a library version mismatch, not a missing symbol, and not a configuration issue in PyTorch. It is a fundamental failure of the NVIDIA driver to initialize a CUDA context on the GPU.

The Reasoning Process: What the Assistant Gets Right

The assistant's reasoning in this message reveals several important cognitive moves:

First, it correctly distinguishes between driver-level and runtime-level errors. The fact that nvidia-smi works proves the kernel-mode driver is functional. The fact that cuInit returns 999 proves the user-mode CUDA driver library (libcuda.so) cannot establish communication with the kernel driver. This is a specific failure mode that points to either a firmware issue, a device permission problem, or a container passthrough misconfiguration.

Second, it correctly rejects the version mismatch hypothesis. The assistant notes that CUDA runtime 13.0 should work with driver 13.2, which is true—CUDA runtime libraries are designed to be forward-compatible with newer drivers. Error 999 is not the error you get from a version mismatch (that would be error 35 or error 802). By ruling this out, the assistant avoids going down a rabbit hole of reinstalling CUDA toolkits.

Third, it identifies the device permission anomaly. The assistant notices that /dev/nvidia-uvm-tools has no permissions set. While this turns out not to be the root cause (the real issue is the cgroup policy blocking /dev/nvidia-uvm), it is a legitimate observation that shows the assistant is paying attention to the full system state. The fix attempted (chmod 666 /dev/nvidia-uvm-tools) is harmless and reasonable.

Fourth, the assistant makes an important connection between the error code and the container environment. The reasoning states: "The error code 999 during CUDA initialization typically points to incomplete GPU firmware or ECC memory initialization." This is a plausible explanation for a post-reboot scenario where the GPU firmware might not have fully initialized. The assistant also correctly notes that "since persistence mode was just turned on, the GPUs might need time to fully come up."

Assumptions and Their Validity

The message operates under several assumptions, most of which are reasonable:

  1. CUDA runtime backward compatibility: The assistant assumes that CUDA 13.0 runtime should work with a CUDA 13.2 driver. This is correct—CUDA maintains backward compatibility, and newer drivers support older runtime libraries.
  2. The nvidia-uvm-tools device matters: The assistant assumes that fixing permissions on /dev/nvidia-uvm-tools might help. In reality, the critical device is /dev/nvidia-uvm (major 511), which is the one actually blocked by cgroup policy. The uvm-tools device is auxiliary. This assumption is incorrect but harmless—fixing the wrong device doesn't fix the problem, but it doesn't make things worse either.
  3. LD_LIBRARY_PATH might be the issue: The assistant tries setting an explicit LD_LIBRARY_PATH to ensure the correct CUDA runtime libraries are loaded. This is a reasonable diagnostic step, but the output shows it doesn't help. The failure is at a lower level than library resolution.
  4. The GPUs might need time to initialize after persistence mode: This is a valid concern. After enabling persistence mode with nvidia-smi -pm 1, the GPUs do need some time to fully initialize their firmware state. However, in this case, the root cause is different.

Input Knowledge Required

To fully understand this message, the reader needs knowledge of:

Output Knowledge Created

This message produces several important pieces of knowledge:

  1. Confirmation that cuInit returns 999: This is the critical diagnostic finding. It rules out software configuration issues and points to a driver-level or device-access problem.
  2. Verification that library paths are correct: The CUDA runtime libraries are present and accessible in the virtual environment. The failure is not due to missing or incompatible libraries.
  3. Evidence that the nvidia-uvm-tools device has broken permissions: While not the root cause, this is a real system anomaly that could cause other issues and is worth fixing.
  4. A narrowed hypothesis space: After this message, the assistant can confidently rule out: - Missing GPU device files - Unloaded kernel driver - CUDA runtime version mismatch - Incorrect LD_LIBRARY_PATH - PyTorch-specific configuration issues The remaining possibilities are: GPU firmware initialization failure, cgroup device permission blocking, or a container passthrough misconfiguration.

The Path Forward

Message [msg 11302] sets the stage for the breakthrough that comes in [msg 11307], where strace reveals the smoking gun:

openat(AT_FDCWD, "/dev/nvidia-uvm", O_RDWR|O_CLOEXEC) = -1 EPERM (Operation not permitted)

The EPERM error on /dev/nvidia-uvm is the direct consequence of the cgroup v2 device policy. The container configuration allows devices with major numbers 195 (nvidia), 509, 226, and 234—but not 511 (nvidia-uvm). This is a classic container passthrough issue: the host's NVIDIA driver was updated (to version 595.71.05, CUDA 13.2), and the new driver version changed the major number for nvidia-uvm from 509 to 511. The LXC configuration was never updated to reflect this change.

The fix, applied in [msg 11311], is a single line added to /etc/pve/lxc/200.conf:

lxc.cgroup2.devices.allow: c 511:* rwm

This is a deceptively simple resolution to a problem that required seven messages of careful diagnostic work to identify. The assistant's systematic approach—checking device files, testing CUDA at multiple API levels, running strace to catch the failing system call—is what made this possible.

Conclusion

Message [msg 11302] exemplifies the kind of rigorous diagnostic reasoning that is essential when working with complex GPU infrastructure. The assistant correctly identifies that a cuInit return code of 999 indicates a driver-level failure, rules out plausible but incorrect hypotheses (version mismatch, library path issues), and narrows the investigation to device access and container passthrough problems. While the message does not itself contain the fix, it is the critical turning point where the problem is correctly framed, setting the stage for the strace-based discovery of the cgroup permission issue.

For anyone working with NVIDIA GPUs in containerized environments, this message and the surrounding conversation serve as a valuable case study. The lesson is clear: when CUDA fails with a cryptic "unknown error" after a host reboot, look first at the device nodes—not at the software stack. The driver might be loaded, nvidia-smi might work, and all the CUDA libraries might be in place, but if the container cannot open /dev/nvidia-uvm, nothing else will work either.