Peeling the CUDA Onion: A Deep Diagnostic Dive into Driver Initialization Failure

Introduction

In the intricate dance of deploying large language models across multiple GPUs, few things are as frustrating as a silent, opaque failure at the lowest level of the software stack. Message 519 in this opencode session captures a pivotal moment in a debugging saga that had already spanned multiple architectural approaches: from KVM virtual machines with VFIO passthrough, to LXC containers designed to bypass virtualization overhead, and finally to a confrontation with the raw NVIDIA driver stack itself. The message is a masterclass in systematic diagnostic reasoning — a deliberate, layer-by-layer peeling of abstractions to isolate the root cause of a CUDA initialization failure that threatened to derail an entire deployment.

The session's broader context is ambitious: deploying the GLM-5-NVFP4 model across 8 NVIDIA RTX PRO 6000 Blackwell GPUs using SGLang, with tensor parallelism requiring efficient peer-to-peer (P2P) GPU communication. Earlier segments had revealed that the KVM virtual machine environment introduced a PHB (PCIe Host Bridge) topology, preventing direct GPU-to-GPU P2P DMA transfers. The LXC container approach was supposed to fix this by giving the container direct access to the host's PCIe topology, and indeed nvidia-smi topo -m inside the container showed the correct bare-metal NODE and SYS topology. But a new, more fundamental problem emerged: CUDA itself refused to initialize.

The Message: A Surgical Diagnostic Strike

The subject message (msg id=519) opens with a stark assessment: "CUDA runtime itself can't initialize. This is a deeper issue." This single sentence reveals the assistant's reasoning process — it has already eliminated higher-level explanations (PyTorch version mismatch, library path issues, Python environment problems) and is now zeroing in on the CUDA driver stack itself. The phrase "deeper issue" signals a shift from application-level debugging to kernel-level investigation.

The assistant then formulates a hypothesis: "Let me check if it's a cgroup restriction or something else." This is a reasonable guess. In LXC containers, device access is mediated through cgroup device controllers, and it's plausible that the container runtime might be blocking CUDA's ability to initialize even though the device nodes (/dev/nvidia0, /dev/nvidiactl, /dev/nvidia-uvm) are visible and accessible. Earlier in the session (msg id=512), the assistant had already verified that these device files could be opened with O_RDWR, but CUDA initialization involves more than just file access — it requires interaction with the kernel module, memory mapping, and potentially other system resources that cgroups might restrict.

The Diagnostic Strategy: Stripping Away Layers

What makes this message remarkable is its diagnostic methodology. Rather than continuing to debug through Python, PyTorch, or even the CUDA runtime API (cudaSetDevice, cudaGetDeviceCount), the assistant drops down to the absolute lowest level of interaction with the GPU: directly loading libcuda.so via dlopen and calling cuInit — the most fundamental CUDA driver API function — from a bare C program compiled with gcc.

This is a deliberate and elegant debugging strategy. Consider the layers that have been stripped away:

  1. Python interpreter — eliminated by using C
  2. PyTorch — eliminated by calling CUDA driver API directly
  3. CUDA runtime API — eliminated by using cuInit (driver API) instead of cudaSetDevice (runtime API)
  4. nvcc / CUDA toolkit — eliminated by compiling with gcc and dynamically linking libcuda.so
  5. Container runtime — partially eliminated by running inside the container but using the most minimal possible program The C program written in the message is minimal but thorough. It loads libcuda.so dynamically, resolves four key symbols (cuInit, cuDeviceGetCount, cuDeviceGet, cuDeviceGetName), calls cuInit(0), and then conditionally queries device information if initialization succeeds. The use of dlopen with RTLD_NOW ensures that any missing symbols or library dependencies will be caught immediately. The program also limits device enumeration to the first two GPUs to keep output manageable. This approach embodies a core debugging principle: reduce the system to its simplest possible form that still reproduces the failure. If cuInit fails from a bare C program, the problem cannot be in PyTorch, Python, or any higher-level library — it must be in the driver or kernel module layer.

The Result: A Confirmed but Unchanged Failure

The output is unambiguous: cuInit(0) = 3. Error code 3 is CUDA_ERROR_NOT_INITIALIZED, which the CUDA documentation describes as occurring when "the CUDA driver has not been initialized" — a somewhat circular description that actually means the driver's internal state machine has rejected initialization for some reason. This is the same error code that appeared in earlier diagnostics (msg id=510, 511), confirming that the problem is consistent across multiple invocation methods.

The dmesg output is equally telling — or rather, not telling. The only kernel messages visible are vxlan: non-ECT network messages, with no NVIDIA-related errors, warnings, or status messages at all. This is significant: when the NVIDIA kernel module encounters problems during initialization (such as GSP firmware loading failures, PCIe BAR allocation issues, or interrupt setup problems), it typically logs detailed error messages to the kernel ring buffer. The absence of such messages suggests either that the kernel module itself is functioning correctly and the failure occurs in userspace, or that the kernel module's error reporting is suppressed or redirected.

Input Knowledge Required

To fully understand this message, several pieces of domain knowledge are necessary:

CUDA Driver Stack Architecture: The CUDA software stack has multiple layers. At the bottom is the kernel module (nvidia.ko, nvidia-uvm.ko) which manages GPU hardware and exposes a device interface. Above that is the userspace CUDA driver library (libcuda.so), which communicates with the kernel module through ioctl calls. Above that sits the CUDA runtime API (libcudart.so), and above that, frameworks like PyTorch. The cuInit function is the entry point into the userspace driver — it initializes the driver's internal state, opens communication channels to the kernel module, and sets up the CUDA context.

CUDA Error Codes: Error code 3 (CUDA_ERROR_NOT_INITIALIZED) is distinct from error code 999 (CUDA_ERROR_UNKNOWN) or error code 100 (CUDA_ERROR_NO_DEVICE). It specifically indicates that the driver API's initialization sequence failed, which can happen for reasons including: incompatible kernel module version, missing GSP firmware, PCIe configuration issues, or resource allocation failures.

LXC Container Mechanics: LXC containers share the host kernel but have isolated userspace. Device access is controlled through bind-mounts of /dev entries and cgroup device controllers. The fact that device files are accessible (verified in msg id=512) doesn't guarantee that all ioctl operations will succeed — some operations may require additional capabilities or access to /proc/driver/nvidia files.

Blackwell GPU Architecture: The NVIDIA RTX PRO 6000 Blackwell GPUs use compute capability SM 12.0, which is a very new architecture. Driver support for new architectures often requires specific firmware files (GSP firmware) that may not be present in all driver versions or may not be compatible with all kernel versions.

Output Knowledge Created

This message produces several critical pieces of knowledge:

  1. Confirmation of the failure mode: cuInit(0) = 3 is consistently reproducible from the lowest possible level of software interaction. This rules out any hypothesis that blames higher-level software (PyTorch, Python, container configuration).
  2. Elimination of cgroup restriction hypothesis: The C program runs inside the same container with the same cgroup restrictions as PyTorch, yet produces the same error. If cgroups were the issue, the C program would also fail — which it does, but the error code is the same, suggesting the root cause is not cgroup-specific but driver-internal.
  3. Absence of kernel-level diagnostics: The dmesg output shows no NVIDIA-related errors, which is itself a diagnostic signal. It suggests either that the kernel module is not detecting any problem (meaning the issue is in the userspace driver's interaction with the kernel module), or that error reporting is not functioning.
  4. A reusable diagnostic tool: The C program written in this message becomes a permanent diagnostic asset. It can be re-run after any configuration change to test whether the fundamental CUDA driver layer is working, without needing to wait for PyTorch or SGLang to initialize.

The Thinking Process: A Window into Systematic Debugging

The reasoning visible in this message reveals a disciplined debugging methodology. The assistant has been working through a hierarchy of hypotheses, each more fundamental than the last:

Hypothesis 1 (msg id=508-509): PyTorch version mismatch — PyTorch 2.10.0 might not support driver 590 / CUDA 13.1. Test: Install PyTorch 2.9.1. Result: Same failure.

Hypothesis 2 (msg id=510): CUDA runtime API incompatibility — perhaps cudaSetDevice has a bug. Test: Call cuInit directly from Python via ctypes. Result: cuInit returns 3.

Hypothesis 3 (msg id=511-512): Device file access restrictions — cgroups or permissions might block access. Test: Check /proc/driver/nvidia accessibility and try opening device files. Result: Files are accessible, /proc/driver/nvidia is populated.

Hypothesis 4 (msg id=513-516): Missing device nodes — perhaps /dev/nvidia-modeset is required. Test: Check for its existence on host and container. Result: Not present on host either, ruled out.

Hypothesis 5 (msg id=517-518): CUDA toolkit not installed on host — perhaps the container needs host-level CUDA. Test: Try compiling and running a CUDA program on the host. Result: CUDA toolkit isn't installed on host; test inside container shows same initialization error.

Hypothesis 6 (msg id=519, current): cgroup restriction or kernel module issue — perhaps the container runtime blocks CUDA initialization at a lower level. Test: Write a minimal C program that loads libcuda.so via dlopen and calls cuInit, bypassing all higher-level software. Result: Same error, no kernel errors in dmesg.

Each hypothesis is tested with the simplest possible experiment that can confirm or refute it. When a hypothesis is refuted, the assistant doesn't discard the knowledge but moves one layer deeper. This is textbook fault isolation — the "divide and conquer" approach applied to a layered software stack.

Assumptions and Their Validity

The message operates under several assumptions, most of which are sound:

Assumption: The NVIDIA kernel module is loaded and functioning. This is supported by the fact that nvidia-smi works and shows all 8 GPUs, and /proc/driver/nvidia is populated with version information and GPU capabilities. However, the assumption that "functioning" means "fully compatible with Blackwell GPUs" may be unwarranted — the driver might load but fail to properly initialize GSP firmware for the new architecture.

Assumption: cuInit returning 3 is the root cause, not a symptom. This is a reasonable working assumption, but it's worth noting that cuInit failure could itself be a symptom of a deeper issue (e.g., PCIe configuration, IOMMU group isolation, or firmware loading) that the kernel module doesn't report through dmesg.

Assumption: The C program approach is sufficient to isolate the issue. This is correct in principle — if cuInit fails from a bare C program, the problem is definitely in the driver or below. However, the program doesn't test every possible initialization path — for example, it doesn't test with CUDA_VISIBLE_DEVICES set, or with different values for the cuInit flags parameter (though the flags parameter is currently unused in all CUDA versions).

Assumption: Absence of dmesg errors means no kernel-level errors. This is the most questionable assumption. The NVIDIA kernel module may suppress errors in certain configurations, or errors may be logged to a different facility (e.g., nvlog or systemd journal). Additionally, the dmesg buffer may have been overwritten by subsequent messages — the vxlan messages suggest network activity is continuously generating kernel messages.

The Broader Significance

This message represents a critical juncture in the debugging process. The LXC approach was supposed to be the elegant solution to the P2P bottleneck — bypassing the VFIO/IOMMU virtualization layer entirely by running the ML workload in a container that shares the host kernel. The topology verification (nvidia-smi topo -m showing NODE instead of PHB) confirmed that the approach was architecturally sound. But the CUDA initialization failure revealed that sharing the host kernel comes with its own set of constraints: the host's NVIDIA driver must be compatible with both the kernel version and the GPU architecture.

The message also illustrates a fundamental tension in GPU virtualization: KVM/VFIO passthrough gives the guest full control over its own driver stack (including the ability to install any compatible driver version), at the cost of PCIe topology abstraction and P2P limitations. LXC containers preserve the true PCIe topology but inherit the host's driver stack, which may not support the latest GPU architectures. There is no free lunch.

Conclusion

Message 519 is a textbook example of systematic diagnostic reasoning in systems engineering. Faced with a persistent CUDA initialization failure that had survived multiple attempted fixes, the assistant methodically stripped away layers of abstraction — from PyTorch to the CUDA runtime to the raw driver API — until only the most fundamental interaction remained. The result (cuInit(0) = 3) confirmed that the problem lived in the driver or kernel layer, and the absence of kernel error messages provided a crucial clue about the nature of the failure.

This message doesn't solve the problem, but it transforms it. What was previously a vague "CUDA doesn't work in the LXC container" becomes a precise "the NVIDIA userspace driver library cannot initialize when talking to the kernel module, and the kernel module is not reporting any errors." That precision is invaluable — it tells the investigator exactly where to look next (GSP firmware files, kernel module version compatibility, PCIe configuration) and exactly what not to waste time on (Python environments, container configurations, PyTorch versions).

In the end, this message exemplifies the most valuable skill in systems debugging: knowing which layer to blame, and having the tools to prove it.