Diagnosing a Missing GPU: A Forensic Deep Dive into Kernel-Level Device Availability

"ls: cannot access '/proc/driver/nvidia/gpus/0000:CE:00.0': No such file or directory"

In the midst of a protracted troubleshooting session spanning dozens of messages, the assistant issues a single, deceptively simple bash command that cuts to the heart of a stubborn deployment failure. Message [msg 11103] is a diagnostic pivot point—a moment where the assistant stops guessing about software-level issues and reaches down to the kernel driver layer to ask a fundamental question: is the GPU even visible to the system?

The Message

The assistant runs:

ssh -o ConnectTimeout=10 root@10.1.230.172 "fuser -v /dev/nvidia1 /dev/nvidiactl /dev/nvidia-uvm 2>&1 || true; ls -l /proc/driver/nvidia/gpus/0000:CE:00.0"

And receives:

                     USER        PID ACCESS COMMAND
/dev/nvidia1:        root     kernel mount /dev/nvidia1
/dev/nvidiactl:      root     kernel mount /dev/nvidiactl
/dev/nvidia-uvm:     root     kernel mount /dev/nvidia-uvm
ls: cannot access '/proc/driver/nvidia/gpus/0000:CE:00.0': No such file or directory

The Context: A Service That Won't Stay Up

To understand why this message was written, we must trace the chain of failures that preceded it. The assistant had been attempting to deploy a native SGLang DFlash service with DDTree speculative decoding on a machine designated CT129, equipped with two NVIDIA RTX A6000 GPUs. The service had been crashing repeatedly. Earlier messages show the assistant restarting the service, clearing stale Python bytecode, and even attempting a GPU reset via nvidia-smi --gpu-reset—which failed with "Resetting GPU ... is not supported."

The critical clue emerged in [msg 11102], where the assistant ran a PyTorch CUDA diagnostics script. That script revealed a stark asymmetry: GPU 0 was functional (tensor([0.], device='cuda:0')), but GPU 1 returned a CUDA error: "CUDA-capable device(s) is/are busy or unavailable" (error code cudaErrorDevicesUnavailable). This error is not a typical "process is using the GPU" error; it originates from the CUDA runtime API and indicates that the driver believes the device is in a state where it cannot be acquired—possibly due to a prior crash, a stuck context, or a hardware-level fault.

Why This Message Was Written: The Reasoning and Motivation

The assistant's reasoning at this point is unstated in the message itself (the "## Agent Reasoning" header is followed by no explicit text), but the choice of commands reveals a clear diagnostic strategy. The assistant is pursuing two complementary lines of inquiry simultaneously:

Process-level investigation via fuser: The fuser -v command identifies which processes are holding open file descriptors to the NVIDIA device files (/dev/nvidia1, /dev/nvidiactl, /dev/nvidia-uvm). If a zombie process or orphaned CUDA context were clinging to GPU 1, fuser would reveal its PID. This is the standard first step in diagnosing a "GPU busy" error.
Kernel driver-level investigation via /proc/driver/nvidia/gpus/: The second command checks for the existence of the kernel driver's device entry for the specific PCI bus address of GPU 1 (0000:CE:00.0). This path is created by the NVIDIA kernel driver when it successfully initializes a GPU. If this file is missing, it means the kernel driver itself does not recognize the GPU—a far more fundamental problem than a stuck process. The assistant's decision to run both commands in a single SSH call (rather than sequentially with separate reasoning steps) reflects an efficient, parallel investigative approach. The || true appended to the fuser command is a defensive measure: fuser returns a non-zero exit code if no processes are found, and the || true prevents the entire SSH command from failing due to that exit code, ensuring the ls command still runs.

The Output: A Revelation

The output delivers a powerful negative result. The fuser command shows that the NVIDIA device files are held only by kernel mounts—there are no user-space processes occupying GPU 1. This rules out the "stuck process" hypothesis. But the second command is the real bombshell: /proc/driver/nvidia/gpus/0000:CE:00.0 does not exist. The kernel driver has no record of GPU 1 at that PCI bus address.

This is a profound finding. It means the issue is not software-level (a stuck CUDA context, a zombie Python process, or a misconfigured SGLang service) but hardware or driver-level. The NVIDIA kernel module either failed to initialize GPU 1 during boot, or the GPU suffered a catastrophic error (such as a Triton compiler crash that triggered a GPU reset or a TDR—Timeout Detection and Recovery) that caused the driver to permanently lose contact with the device.

Assumptions and Potential Mistakes

The assistant makes several implicit assumptions in this message:

The bus address 0000:CE:00.0 is correct for GPU 1. This assumption is well-founded, as earlier nvidia-smi output (in the context messages) showed two GPUs, and the PCI bus topology can be cross-referenced. However, if the bus address were misidentified, the ls failure could be a false negative. The assistant had previously seen GPU 1 fail in PyTorch, so the address is likely correct.
The absence of the /proc/driver/nvidia/gpus/ entry indicates a kernel driver problem. This is a sound inference, but it's worth noting that the path format can vary across NVIDIA driver versions. The assistant is using the standard path convention for the 580-series driver, which is correct.
The fuser output showing "kernel mount" means no user processes are using the GPU. This is accurate: fuser reports the PID and access type, and "kernel mount" indicates the kernel module itself holds the reference, not a user-space process. One subtle limitation of this diagnostic: fuser -v checks open file descriptors on the device files, but a CUDA context can persist in the driver state even after the owning process has exited, especially if the GPU experienced an error. The kernel mount entries suggest the driver module is loaded, but the missing /proc/driver/nvidia/gpus/ entry suggests the driver's per-GPU initialization failed.

Input Knowledge Required

To fully understand this message, the reader needs:

Familiarity with Linux device files: /dev/nvidia0, /dev/nvidia1, /dev/nvidiactl, and /dev/nvidia-uvm are the standard device nodes created by the NVIDIA driver. /dev/nvidia1 corresponds to the second GPU.
Knowledge of the NVIDIA driver's procfs interface: /proc/driver/nvidia/gpus/ contains per-GPU directories created by the kernel module during initialization. Each directory is named by the GPU's PCI bus address in domain:bus:device.function format.
Understanding of PCI bus addressing: 0000:CE:00.0 breaks down as domain 0000, bus CE (206 decimal), device 00, function 0. This address uniquely identifies the GPU on the PCIe bus.
Context from the preceding failure cascade: The reader must know that GPU 0 works, GPU 1 fails with cudaErrorDevicesUnavailable, and that GPU resets are unsupported on this hardware (A6000 cards often don't support --gpu-reset without proper driver configuration).
The deployment goal: The assistant is trying to run SGLang with TP2 (tensor parallelism across 2 GPUs), which requires both GPUs to be functional. A single-GPU failure makes TP2 impossible.

Output Knowledge Created

This message produces several critical pieces of knowledge:

GPU 1 is not just "busy"—it is invisible to the kernel driver. This is a far more severe diagnosis than a stuck process. It means the NVIDIA kernel module did not successfully initialize GPU 1, or the GPU suffered a fault that caused the driver to release its device entry.
No user-space process is holding GPU 1. The fuser output confirms that the device files are only held by kernel mounts, eliminating the need to hunt for orphaned Python processes or zombie CUDA contexts.
The problem is at the driver or hardware layer. The missing procfs entry points to causes such as: a GPU that failed to initialize during boot, a PCIe enumeration issue, a GPU that experienced an uncorrectable error and was put into an error state by the driver, or a physical hardware fault.
The deployment on CT129 is likely infeasible without a reboot or hardware intervention. Since nvidia-smi --gpu-reset is unsupported and the driver does not recognize the GPU, the only remaining software options are unloading and reloading the NVIDIA kernel module (risky on a production system) or rebooting the machine. A hardware reseat or replacement may be necessary.

The Thinking Process: What the Assistant's Silence Reveals

The assistant's reasoning section is empty in this message—just the "## Agent Reasoning" header with no text. This absence is itself revealing. In earlier messages, the assistant often wrote extensive reasoning paragraphs. The silence here suggests a shift in cognitive mode: from exploratory reasoning to focused execution. The assistant has formed a hypothesis (GPU 1 is lost at the driver level) and is executing a targeted diagnostic to confirm it. The reasoning is embedded in the choice of commands rather than in explicit prose.

The parallel structure of the command—checking both process-level (fuser) and kernel-level (/proc/driver/nvidia/gpus/) state—reveals a systematic diagnostic methodology. The assistant is not guessing; it is systematically ruling out possible causes from the most likely (stuck process) to the most fundamental (driver initialization failure).

Impact and Aftermath

This message marks the effective end of the CT129 deployment attempt. The assistant pivots in subsequent messages to CT200, an 8× RTX PRO 6000 Blackwell machine, where a fresh environment is built from scratch. The lesson learned from CT129—that a GPU can become invisible to the driver after a Triton crash—informs the assistant's approach on CT200, where more careful environment isolation and service management are employed.

In the broader narrative of the coding session, [msg 11103] is a turning point. It transforms a frustrating, opaque failure ("the service crashes, but I don't know why") into a clear, actionable diagnosis ("GPU 1 is gone at the driver level; this machine cannot run TP2 SGLang"). That clarity, even though it forces a platform change, is valuable—it prevents further wasted effort on a fundamentally broken host.