The Diagnostic Dead-End: Tracing a cuInit Error 3 on Blackwell GPUs

In the long and winding journey to deploy the GLM-5-NVFP4 model across eight NVIDIA RTX PRO 6000 Blackwell GPUs, few moments are as instructive as message 555 in this coding session. At first glance, it appears to be a routine diagnostic exchange — the assistant notices something in system output, formulates a hypothesis, and dispatches two bash commands to test it. But beneath this surface lies a rich tapestry of reasoning, assumption-testing, and the quiet drama of debugging at the frontier of AI infrastructure. This message captures the moment when one promising line of investigation closes, forcing the assistant to pivot toward the root cause that will eventually unlock the entire deployment.

The Context: A Stubborn CUDA Initialization Failure

To understand message 555, we must first appreciate the wall the assistant has been running into. For several preceding messages (see [msg 541] through [msg 554]), the core problem has been that cuInit() — the fundamental CUDA runtime initialization call — either hangs indefinitely or returns error code 3 (CUDA_ERROR_NOT_INITIALIZED). This failure blocks every downstream operation: PyTorch cannot detect GPUs, inference servers cannot launch, and the entire deployment grinds to a halt.

The assistant has already explored several avenues. It checked for GSP (GPU System Processor) firmware and found that while the gsp_ga10x.bin file contains Blackwell ELF sections inside it, the GSP firmware is actually loaded and reporting version 590.48.01 via nvidia-smi. It ran strace and discovered that cuInit completes without hanging but returns error 3. It verified that device files exist (/dev/nvidia0 through /dev/nvidia7, /dev/nvidiactl, /dev/nvidia-uvm). The stage is set for a new hypothesis.

The Message: Reasoning Aloud at the Terminal

Message 555 opens with the assistant thinking through an observation:

I see something very interesting — the mig-minors shows gpu0 through gpu14 — that's 15 GPUs listed in MIG minors, but there are only 8 physical GPUs. Wait, nvidia-smi -q timed out. The MIG entries are just templates (15 GPU entries 15 GI entries 8 CI entries each). That's normal.

This passage is a window into the assistant's cognitive process. It notices an apparent anomaly — 15 GPU entries in the MIG minors file when only 8 physical GPUs exist. For a brief moment, this seems like a lead. But then the assistant catches itself: the MIG (Multi-Instance GPU) minor numbering scheme creates template entries for all possible GPU instances, compute instances, and channels. The math checks out — 15 GPU entries times 15 GPU Instance entries times 8 Compute Instance entries each produces the large number of entries. The observation is dismissed as normal behavior, and the assistant explicitly refocuses:

Let me focus on the key issue: cuInit returns error 3. Let me check if there's a CUDA userspace library mismatch — the host Python is using Debian's system Python which may have a different CUDA userspace than what we installed.

This is the core hypothesis of the message. The assistant suspects that the system Python (running on the Proxmox host) might be picking up a different version of libcuda.so than the one installed with the NVIDIA driver. This is a classic problem in Linux systems: library resolution via ldconfig can sometimes point to stale or incorrect versions, especially when multiple CUDA toolkits or driver installations coexist.

The Investigation: Two Bash Commands in Parallel

To test this hypothesis, the assistant dispatches two bash commands in the same round (they appear together in the message, meaning they run in parallel):

Command 1 — Library path inspection:

ssh root@10.1.2.6 "ldconfig -p 2>/dev/null | grep libcuda; echo '---'; ls -la /usr/lib/x86_64-linux-gnu/libcuda* 2>/dev/null; echo '---'; ls -la /usr/lib64/libcuda* 2>/dev/null"

This command does three things:

  1. Queries ldconfig's cache to see which libcuda files are registered and where they resolve to
  2. Lists the actual files in the standard 64-bit library directory
  3. Checks an alternative library directory (/usr/lib64) that some distributions use Command 2 — Driver version verification:
ssh root@10.1.2.6 "nvidia-smi -q 2>/dev/null | head -30"

This runs nvidia-smi -q (query mode) to get detailed driver information, confirming the driver version, CUDA version, and that all 8 GPUs are properly attached.

The results, which arrive in the next message, are clear:

The Assumptions Under Test

This message reveals several assumptions the assistant is operating under:

Assumption 1: A library mismatch could cause cuInit error 3. This is reasonable — if the CUDA userspace library is older than the kernel driver, or if a different CUDA toolkit's libraries are being loaded first, initialization can fail with error 3. The assistant is following standard CUDA debugging procedure.

Assumption 2: The system Python's library resolution is the problem. The assistant specifically mentions "the host Python is using Debian's system Python which may have a different CUDA userspace than what we installed." This assumes that the CUDA toolkit installation (which happened earlier in the session) might have placed libraries in a non-standard location that the system Python doesn't see.

Assumption 3: nvidia-smi -q will complete successfully. Given that earlier nvidia-smi -q commands timed out (as noted in the message), there's a risk this one would too. But it doesn't — it returns cleanly, which itself is useful information.

Input Knowledge Required

To fully understand this message, a reader needs:

  1. CUDA initialization semantics: Understanding that cuInit(0) is the entry point for CUDA, and error code 3 means CUDA_ERROR_NOT_INITIALIZED — the runtime could not initialize, typically due to driver/userspace mismatch, permissions, or hardware issues.
  2. Linux library resolution: How ldconfig maintains a cache of shared library locations, and how the dynamic linker resolves libcuda.so.1 at runtime. The difference between symlinks and actual .so files.
  3. NVIDIA driver architecture: The split between the kernel module (nvidia.ko, nvidia-uvm.ko) and the userspace library (libcuda.so). Both must match in version.
  4. MIG (Multi-Instance GPU): NVIDIA's partitioning technology that creates multiple GPU instances from one physical GPU. The minor numbering scheme creates many entries in /proc/driver/nvidia-caps/mig-minors.
  5. Proxmox VE environment: The assistant is working on a Proxmox host with LXC containers, where GPU passthrough adds complexity.
  6. Blackwell architecture: The RTX PRO 6000 Blackwell GPUs are NVIDIA's newest architecture (compute capability 12.0), which may have unique driver requirements.

Output Knowledge Created

The message produces several concrete pieces of knowledge:

  1. The CUDA userspace library is correctly installed: ldconfig shows libcuda.so.1 at /lib/x86_64-linux-gnu/libcuda.so.1, which is the standard location installed by the NVIDIA driver runfile. The version is 590.48.01, matching the kernel driver.
  2. No library mismatch exists: The symlinks point to the correct versioned files. There is no stale or incorrect library resolution.
  3. The driver is fully operational: nvidia-smi -q confirms Driver 590.48.01, CUDA 13.1, and all 8 GPUs attached. The driver layer itself is healthy.
  4. The hypothesis is disproven: The library mismatch theory cannot explain the cuInit error 3. The assistant must look elsewhere.

The Significance: A Necessary Dead-End

This message is what debugging looks like in practice: forming hypotheses, testing them, and ruling them out. The library mismatch hypothesis was reasonable — it's a common source of CUDA initialization failures. But the evidence shows it's not the cause here.

The real value of this message lies in what it eliminates. By confirming that the userspace library is correct and the driver is healthy, the assistant narrows the search space. The cuInit error 3 must be caused by something else — something in the kernel module layer, specifically the nvidia-uvm module. This sets the stage for the breakthrough that follows in subsequent messages, where the assistant discovers that the Heterogeneous Memory Management (HMM) feature in the open-source NVIDIA kernel module is incompatible with the Proxmox VE kernel, and that setting uvm_disable_hmm=1 resolves the issue entirely.

The assistant's reasoning also demonstrates an important debugging virtue: the ability to quickly dismiss a false lead. The MIG minors observation could have sent the assistant down a rabbit hole of MIG configuration issues. Instead, the assistant recognized it as normal template behavior within seconds and refocused on the actual problem. This kind of mental discipline — knowing when something is a red herring — is essential in complex system debugging.

Conclusion

Message 555 is a snapshot of the scientific method applied to systems engineering. It begins with an observation (MIG minors showing 15 GPUs), corrects a potential misinterpretation, formulates a testable hypothesis (library mismatch), and executes two diagnostic commands to gather evidence. The hypothesis is disproven, but the process yields valuable negative knowledge: the CUDA userspace is correct, the driver is healthy, and the problem lies elsewhere. In the broader narrative of deploying GLM-5-NVFP4 on eight Blackwell GPUs, this message represents the closing of one door and the quiet turning toward the door that will eventually open.