The Moment of Silence: When Four Blackwell GPUs Vanished After an IOMMU Identity Domain Switch

Introduction

In the high-stakes world of large-scale ML inference deployment, few things are as jarring as the command that returns "No devices were found." This is precisely what happened at message [msg 6266] in a prolonged session dedicated to deploying and optimizing NVIDIA Blackwell GPUs for serving large language models. The assistant had just completed what appeared to be a flawless sequence of unbinding four NVIDIA RTX PRO 6000 Blackwell GPUs from their driver, switching their IOMMU group types to identity mode, and rebinding them to the nvidia driver. Yet when the verification command ran, the response was stark: "No devices were found." This single line of output signaled the collapse of a carefully planned optimization and set off a multi-step debugging odyssey that would ultimately reveal a fundamental incompatibility between Blackwell GPUs and per-group IOMMU identity domains.

The Message

The subject message is deceptively brief:

nvidia-persistenced not a separate service on this host (probably managed differently). Let me check nvidia-smi:

>

``bash ssh root@[REDACTED] 'nvidia-smi --query-gpu=index,gpu_bus_id,name,memory.total --format=csv,noheader 2>&1' No devices were found ``

On its surface, this appears to be a routine verification step. The assistant had just finished rebinding all four GPUs to the nvidia driver, and a minor hiccup occurred when restarting nvidia-persistenced failed because the service unit didn't exist on this host. The assistant dismisses this with "probably managed differently" — a reasonable assumption, as different Linux distributions and configurations handle NVIDIA persistence through various mechanisms (some use systemd, others use init scripts, and some integrate it directly into the nvidia driver module). The real concern, however, is the output of nvidia-smi: "No devices were found."

The Context: A Plan to Restore P2P DMA

To understand why this message carries such weight, we must trace back through the preceding messages. The system in question is a Proxmox host running Ubuntu 24.04 with 8 NVIDIA RTX PRO 6000 Blackwell GPUs, split between an LXC container (4 GPUs for ML inference) and a SEV-SNP VM (4 GPUs for confidential computing). The GPUs in the LXC container had been operating with NCCL_P2P_DISABLE=1 because Peer-to-Peer DMA transfers were failing under the system's IOMMU configuration — the host runs with amd_iommu=on and mem_encrypt=on for SEV-SNP support, which puts all PCI devices under DMA translation.

The performance cost of disabling P2P is significant. For tensor-parallel inference across 4 GPUs, every all-reduce operation must go through system memory (SHM copies) instead of direct GPU-to-GPU transfers, which are 5-10x faster. The assistant had researched a promising solution: per-IOMMU-group identity domains, a kernel feature available since Linux 5.11 that allows individual IOMMU groups to be switched from translation mode (DMA-FQ) to passthrough mode (identity). The plan was to switch only the four GPU groups used by the LXC container to identity mode, leaving the four VFIO groups in translation mode for the SEV-SNP VM. This would give the ML workloads direct P2P access while keeping the VM's confidential computing guarantees intact.

The execution had gone smoothly up to this point. The assistant stopped the SGLang service ([msg 6257]), killed zombie processes ([msg 6259]), verified GPUs were free ([msg 6260]), unbinded all four GPUs from the nvidia driver ([msg 6263]), successfully wrote identity to each group's type file ([msg 6263]), and rebinded them to nvidia ([msg 6265]). Every step reported success. The bind operation returned "bound to nvidia OK" for all four GPUs. Then came the verification.

The Assumptions That Broke

This message reveals several assumptions that, in hindsight, were flawed:

Assumption 1: The nvidia driver bind operation implies a working GPU. The assistant assumed that a successful write to /sys/bus/pci/drivers/nvidia/bind meant the GPU had been properly initialized by the driver. In reality, the bind operation merely associates the PCI device with the driver — it does not guarantee that the GPU's firmware (GSP) initialized correctly. The GPU can be "bound" but non-functional.

Assumption 2: IOMMU identity mode is transparent to device initialization. The assistant assumed that switching a GPU's IOMMU group to identity mode would be invisible to the device — that the GPU would simply see physical addresses directly instead of translated IOVAs. This assumption turned out to be catastrophically wrong for Blackwell GPUs, as later investigation would reveal.

Assumption 3: The nvidia-persistenced failure was a minor, unrelated issue. The assistant dismissed the nvidia-persistenced service failure as a configuration difference. While this was technically correct (the service didn't exist on this host), the dismissal meant the assistant didn't immediately connect it to the GPU initialization failure. In reality, the GPUs were in a far worse state than a missing persistence service could explain.

Assumption 4: The unbind/rebind cycle is a safe operation for Blackwell GPUs. The assistant had previously performed driver unbind/rebind operations on these GPUs without issue. However, those operations were done under DMA-FQ (translation) mode. The combination of switching to identity mode while performing a driver unbind/rebind created a unique failure scenario.

Input Knowledge Required

To fully understand this message, one needs:

  1. IOMMU concepts: Understanding that IOMMU translates device DMA addresses to system physical addresses, and that "identity" mode bypasses this translation, giving devices direct memory access.
  2. Linux PCI device model: Knowledge of how PCI devices are bound to drivers via sysfs (/sys/bus/pci/drivers/<driver>/bind), and that a successful bind doesn't guarantee successful device initialization.
  3. NVIDIA driver architecture: Familiarity with nvidia-smi as the primary diagnostic tool, and understanding that "No devices were found" means the nvidia driver's kernel module loaded but the GPU initialization (RmInitAdapter) failed.
  4. Blackwell GPU specifics: Knowledge that Blackwell GPUs have a GSP (GPU System Processor) that requires specific firmware initialization sequences, and that this initialization depends on DMA mappings set up by the kernel.
  5. The system topology: Understanding that the 4 GPUs in question are on NUMA0, bound to the nvidia driver, while 4 other GPUs on NUMA1 are on vfio-pci for VM passthrough.

Output Knowledge Created

This message creates several critical pieces of knowledge:

  1. The identity domain switch broke GPU initialization. The stark "No devices were found" output is the first evidence that something went fundamentally wrong. It transforms the plan from "verify and celebrate" to "diagnose and recover."
  2. The bind operation's success is not sufficient. The assistant learns that a successful PCI driver bind does not guarantee a functional device. This distinction becomes crucial for future troubleshooting.
  3. A new debugging trace begins. The assistant immediately pivots to checking dmesg and /dev/nvidia* devices in the next message ([msg 6267]), starting a diagnostic chain that will uncover the GSP firmware initialization failure.
  4. The plan needs revision. The P2P restoration via identity domains is now in question. The assistant doesn't yet know the full extent of the problem, but the seed of doubt is planted.

The Thinking Process

The assistant's reasoning in this message is visible in its structure. First, it addresses the nvidia-persistenced failure with a rationalization: "not a separate service on this host (probably managed differently)." This is a reasonable inference — on Proxmox hosts with custom configurations, systemd services often differ from standard Ubuntu installations. The assistant doesn't dwell on this because the real verification is nvidia-smi.

The "Let me check nvidia-smi" transition is significant. The assistant is moving from a minor concern (persistence service) to the primary verification (GPU visibility). The confidence is still high at this point — the bind operations all succeeded, the identity domains were set, and the only hiccup was a non-critical service. The expectation is clearly that nvidia-smi will show 4 healthy GPUs.

The output "No devices were found" is therefore a shock. The assistant doesn't show an emotional reaction (it's an AI), but the brevity of the message — just the command and its output — speaks volumes. There's no analysis yet, no "this means X" conclusion. The assistant is in data-gathering mode, presenting the raw failure to be interpreted. The next message ([msg 6267]) will begin the actual diagnosis by checking device files and kernel logs.

The Deeper Significance

This message is a turning point in the segment. Before it, the trajectory was optimistic: research had identified a promising solution, the execution plan was sound, and the initial steps completed without error. After it, the trajectory shifts to damage control and investigation. The assistant will spend the next dozen messages trying various reset techniques — PCI remove/rescan, Function Level Reset (FLR), Secondary Bus Reset (SBR), and combinations thereof — before ultimately discovering that Blackwell's GSP firmware requires DMA translation mode during initialization and that identity mode is fundamentally incompatible with these GPUs.

The message also illustrates a crucial principle in systems engineering: a successful operation at one layer does not guarantee success at a higher layer. The PCI bind succeeded at the bus/driver layer, but the GPU firmware initialization failed at the device layer. This layering mismatch is a common source of subtle bugs, and catching it requires exactly the kind of verification that the assistant performed here.

Conclusion

Message [msg 6266] captures the precise moment when a well-planned optimization derails. In just two lines of output — "nvidia-persistenced not a separate service on this host" and "No devices were found" — it encapsulates the gap between expectation and reality that drives all debugging. The assistant had followed every step correctly, the kernel had accepted every command, and yet the GPUs were invisible. This message is the pivot point between the confident execution of a plan and the uncertain work of recovery. It is a reminder that in complex systems, success is never guaranteed until the final verification runs, and that "No devices were found" is often the beginning of a much deeper investigation.