The Silent Hang: Diagnosing a CUDA Initialization Deadlock on Blackwell GPUs

Introduction

In any complex systems engineering effort, the most frustrating bugs are often the quietest ones. A crash produces a stack trace. An error code gives you something to search for. But a hang — a process that simply stops responding and must be killed by a timeout — offers no explanation at all. It is the system's way of saying "I'm thinking about it," except it never finishes thinking.

This is precisely the situation faced in message [msg 547] of an opencode coding session aimed at deploying the GLM-5-NVFP4 model — a 744-billion-parameter Mixture-of-Experts language model — across eight NVIDIA RTX PRO 6000 Blackwell Server Edition GPUs. The session had already overcome numerous obstacles: installing NVIDIA drivers on a Proxmox VE host, configuring an LXC container with GPU passthrough, and verifying that the container saw the true bare-metal GPU topology (NODE/SYS) rather than the crippled PHB topology imposed by KVM virtualization. But a single stubborn problem blocked all further progress: CUDA refused to initialize.

What makes message [msg 547] particularly interesting is that it documents a change in the failure mode. Earlier attempts had produced cuInit() returning error code 3 (CUDA_ERROR_NOT_INITIALIZED). Now, something had shifted, and cuInit() was hanging indefinitely — a more severe and more puzzling symptom. This message captures the moment when the assistant recognized that shift, formulated a new hypothesis about its cause, and began gathering evidence to test it.

The Context: A Blockade at the Gates of CUDA

To understand why message [msg 547] was written, we must first understand the stakes. The broader session (documented across segments 0 through 5 of the conversation) had been working toward a single goal: deploying the GLM-5-NVFP4 model using the SGLang inference server on a machine with eight Blackwell GPUs, achieving throughput targets of over 1,000 tokens per second total and over 100 tokens per second for single-stream requests.

The previous approach had used a KVM virtual machine, which achieved approximately 485 tokens per second at peak — respectable, but far from the target. The bottleneck was identified as PCIe Peer-to-Peer (P2P) latency between GPUs, caused by VFIO/IOMMU virtualization overhead. Each GPU lived in its own IOMMU group, preventing direct GPU-to-GPU DMA transfers. The solution was to abandon the KVM VM and instead use an LXC container, which shares the host kernel and can access hardware directly without VFIO intervention.

This migration had been largely successful. The LXC container saw the real GPU topology: GPUs within the same NUMA node reported NODE connectivity (direct P2P), while cross-NUMA GPUs reported SYS (through the system interconnect). This was a major improvement over the KVM VM, where all GPUs showed only PHB (PCIe Host Bridge) connectivity.

But there was a catch. While nvidia-smi worked perfectly — reporting all eight GPUs, their temperatures, power draw, and compute capability 12.0 — any attempt to use the CUDA runtime or driver API failed. The cuInit() function, the very first call any CUDA program makes, either returned error code 3 or, as the session progressed, simply hung forever.

The Message: A Diagnostic Pivot

Message [msg 547] opens with a critical observation:

So torch imports fine, but cuInit() hangs on the host (killed by timeout, exit code 124) — it doesn't even return an error code, it just blocks forever. This is different from what was recorded before (error code 3).

This sentence encapsulates the diagnostic pivot. The assistant had been operating under the assumption that cuInit() returned error code 3 — a well-documented error meaning "CUDA driver not initialized." That error, while blocking, is at least informative. It tells you something about the state of the driver or the GPU firmware. But a hang is different. A hang suggests that cuInit() entered some internal loop or wait state from which it never emerged. It suggests the problem is not that CUDA can't initialize, but that it's stuck trying to initialize and waiting for something that never arrives.

The assistant's immediate hypothesis is stated plainly: "Let me check if GSP firmware loading is the issue — cuInit may be trying to load GSP firmware and getting stuck."

This hypothesis is grounded in knowledge of NVIDIA's GPU architecture. Modern NVIDIA GPUs include a GPU System Processor (GSP) — a RISC-V coprocessor that handles tasks like firmware management, power management, and display control. The open-source NVIDIA kernel module (the "open kernel module," as opposed to the proprietary one) relies on the GSP for certain initialization tasks. If the GSP firmware is missing, incompatible, or fails to load, cuInit() might block indefinitely waiting for a response from the GSP.

The Investigation: Gathering Evidence

The assistant then executes three bash commands in parallel, each targeting a different source of diagnostic information.

The first command checks the GSP firmware files and searches kernel logs for any mention of GSP or NVIDIA firmware loading:

ssh root@10.1.2.6 "ls -la /lib/firmware/nvidia/590.48.01/ 2>/dev/null; echo '---'; dmesg | grep -i gsp 2>/dev/null; echo '---'; dmesg | grep -i 'nvidia.*firmware\|nvidia.*ucode\|nvidia.*init' 2>/dev/null | tail -10"

The results are telling. The firmware directory contains only two files: gsp_ga10x.bin (for Ampere/GA10x GPUs) and gsp_tu10x.bin (for Turing/TU10x GPUs). There is no gsp_gb20x.bin or any other Blackwell-specific firmware file. The dmesg output contains no GSP-related messages at all — not even an error message saying "failed to load GSP firmware." The driver loads, the NVLink core initializes, but the GSP firmware is never mentioned. It's as if the driver doesn't even try to load it.

The second command examines the GPU's information file in /proc/driver/nvidia:

ssh root@10.1.2.6 "cat /proc/driver/nvidia/gpus/0000:01:00.0/information 2>/dev/null; echo '---'; cat /proc/driver/nvidia/gpus/0000:01:00.0/registry 2>/dev/null | head -20"

This confirms the GPU model, firmware version (590.48.01), and that the GPU is not excluded. The registry is empty — Binary: "" — which is normal for an unconfigured GPU.

The third command tests cuInit() inside the LXC container, confirming the same hang behavior:

ssh root@10.1.230.174 "timeout 5 /root/ml-env/bin/python3 -c '
import ctypes, signal
signal.alarm(4)
try:
    lib = ctypes.CDLL(\"libcuda.so.1\")
    print(\"loaded libcuda\")
    r = lib.cuInit(0)
    print(\"cuInit returned:\", r)
except Exception as e:
    print(\"error:\", e)
' 2>&1; echo 'exit code:' \$?"

The exit code is 124 — the process was killed by the timeout command. The loaded libcuda print statement never appears, suggesting that even loading the CUDA library might be hanging, or that the print is buffered and never flushed before the hang occurs. Either way, the container exhibits the same behavior as the host, confirming that the issue is not specific to the container's environment but is a property of the driver and kernel combination.

Assumptions and Their Implications

The assistant makes several assumptions in this message, most of which are reasonable given the available evidence.

Assumption 1: The hang is caused by GSP firmware loading. This is the central hypothesis. The assistant connects the absence of Blackwell GSP firmware files with the hang behavior, inferring that cuInit() is waiting for the GSP to respond and the GSP never does because its firmware was never loaded. This is a plausible inference, but it's not the only possible explanation. Other potential causes include a kernel incompatibility (the PVE kernel 6.8.12-9 may lack patches needed for Blackwell), a driver bug specific to the open kernel module on SM120 hardware, or a conflict with IOMMU settings on the host.

Assumption 2: The change from error code 3 to a hang is significant. The assistant treats this as a meaningful diagnostic clue, and rightly so. However, it's worth noting that the earlier tests that produced error code 3 may have been using a slightly different environment — perhaps a different CUDA toolkit version, or a different state of the driver after module reloads. The change could reflect a genuine shift in the driver's behavior, or it could be an artifact of testing methodology.

Assumption 3: The GSP firmware should be loaded during cuInit(). This is correct for modern NVIDIA architectures. The GSP is initialized when the CUDA driver initializes, and if the firmware isn't available, initialization will fail. However, the exact failure mode (hang vs. error) may depend on the specific driver version and GPU architecture.

The Knowledge Landscape

To fully understand this message, one needs input knowledge spanning several domains:

CUDA Driver Architecture: Understanding that cuInit() is the entry point for all CUDA operations, and that it triggers a chain of initialization events including kernel module communication, GPU discovery, firmware loading, and memory management setup.

NVIDIA GSP Firmware: Knowledge that modern NVIDIA GPUs include a RISC-V-based GPU System Processor that runs firmware loaded by the kernel module, and that different GPU architectures require different firmware binaries (e.g., gsp_ga10x.bin for Ampere, gsp_tu10x.bin for Turing).

Linux Kernel and DKMS: Understanding that the NVIDIA kernel module is compiled against specific kernel headers via DKMS, and that kernel upgrades require rebuilding the module. The PVE kernel is a custom Proxmox kernel based on the 6.8 mainline, and may not include all the patches that the NVIDIA open module expects.

Proxmox VE Virtualization: Knowledge that LXC containers share the host kernel and can access hardware directly, unlike KVM VMs which go through VFIO. This is why the topology improves but also why kernel compatibility issues from the host affect the container.

Blackwell Architecture (SM120): Understanding that Blackwell is NVIDIA's newest GPU architecture (compute capability 12.0) and that driver support for it is still maturing. The 590.48.01 driver is relatively recent, and the open kernel module's support for Blackwell may have bugs or missing features.

The output knowledge created by this message is a refined diagnosis: the problem is not a simple initialization error but a hang, likely related to GSP firmware that doesn't exist for Blackwell in this driver version. This narrows the search space and points toward specific solutions: upgrading the kernel (which might include updated firmware loading paths), finding or extracting Blackwell GSP firmware, or switching to a different driver version that includes proper Blackwell support.

The Thinking Process

The assistant's reasoning in this message follows a clear diagnostic pattern:

  1. Observe the symptom: cuInit() hangs (exit code 124 from timeout) rather than returning an error code.
  2. Form a hypothesis: The hang might be caused by GSP firmware loading — the driver is waiting for the GSP to initialize, but the firmware isn't available for Blackwell GPUs.
  3. Gather evidence: Check the firmware directory (no Blackwell firmware), check kernel logs (no GSP messages at all), check GPU info (firmware version matches driver), and confirm the hang is consistent across host and container.
  4. Refine the diagnosis: The absence of GSP firmware combined with the absence of any GSP-related log messages suggests the driver isn't even attempting to load GSP firmware for these GPUs. This could mean the driver doesn't recognize the GPUs as needing GSP firmware (a driver bug), or that the GSP firmware loading path is broken on this kernel version. The assistant's thinking is methodical and evidence-based. It doesn't jump to conclusions but instead uses each command to test a specific aspect of the hypothesis. The parallel execution of the three bash commands shows an efficient approach to data gathering — each command targets a different information source (firmware filesystem, kernel logs, GPU device info, and container behavior).

Conclusion

Message [msg 547] represents a critical turning point in the debugging process. By recognizing that the failure mode had changed from an error to a hang, and by systematically investigating the GSP firmware hypothesis, the assistant laid the groundwork for the eventual solution. In the following messages, the assistant would discover that the Heterogeneous Memory Management (HMM) feature in the nvidia_uvm module was the actual culprit — disabling it with uvm_disable_hmm=1 would resolve the CUDA initialization issue entirely.

But at this moment in the conversation, the assistant is still in the thick of the investigation, following the evidence where it leads. The message captures the essence of disciplined debugging: observe carefully, hypothesize precisely, and test relentlessly. It's a small but instructive window into how complex systems problems are diagnosed when the usual tools — error codes, stack traces, log messages — are silent.