The Vanishing Group: A Pivotal Discovery in the Blackwell IOMMU Saga

Introduction

In the course of deploying a high-performance inference server across eight NVIDIA RTX PRO 6000 Blackwell GPUs, a critical hardware-software boundary was encountered. Message [msg 6304] captures the precise moment when a promising approach to restoring peer-to-peer (P2P) DMA connectivity — setting IOMMU identity domains — collided with the fundamental architecture of the Linux kernel's IOMMU subsystem. The message is deceptively short: a few lines of reflection followed by a single bash command and its output. But within that brevity lies the crystallization of a key insight that would ultimately lead to the conclusion that per-group IOMMU identity domains are fundamentally incompatible with Blackwell GPUs.

The Context: Why Identity Domains Matter

To understand the significance of this message, one must appreciate the problem being solved. The system had eight RTX PRO 6000 Blackwell GPUs split between two NUMA domains. Four GPUs on NUMA0 were bound to the NVIDIA driver for serving a Qwen3.5-122B-A10B model with tensor parallelism across all four. The other four GPUs on NUMA1 were assigned to VFIO for a SEV-SNP confidential computing VM. The NUMA0 GPUs needed P2P DMA to communicate efficiently during tensor-parallel inference, but the combination of AMD EPYC's IOMMU with SEV-SNP enabled was causing P2P DMA transactions to fault.

The standard fix for this on AMD systems is to set the IOMMU domain type to identity for the GPU groups, which bypasses DMA translation and allows direct peer-to-peer access. The assistant had been attempting to do exactly this — but every attempt to set identity mode resulted in the NVIDIA driver failing to initialize the GPUs, with the Blackwell Firmware Security Processor (FSP) reporting initialization errors.

The Message: A Moment of Reflection and Experimentation

The assistant begins the message by summarizing the state of play:

GPUs restored. All 4 working, VFIO GPUs unaffected. Now I know the challenge: the nvidia driver on Blackwell can't survive unbind/rebind without a hardware-level power cycle (SBR only works from a clean PCI-removed state).

This opening line reveals the hard-won knowledge from the preceding messages ([msg 6291] through [msg 6303]). The assistant had spent considerable effort attempting various reset strategies — Secondary Bus Reset (SBR), fundamental reset via gpu-admin-tools, D3cold power state transitions — all of which failed to fully reset the Blackwell GPUs' FSP firmware state. The FSP, a protected security processor embedded in the GPU, maintains its state across software-initiated resets and can only be cleared by a true power cycle (D3cold or AC power removal).

The assistant then articulates a hypothesis:

The key insight is: the SBR only works if we remove the GPU from the OS first, SBR the bridge, then rescan. When nvidia auto-probes during rescan, the IOMMU domain is whatever was default at that point.

This is the crucial reasoning step. The assistant realizes that the timing of when the IOMMU domain type is set relative to when the NVIDIA driver probes the device is the critical variable. If the IOMMU group could be switched to identity mode between the PCI removal and the rescan — when the device doesn't exist yet but the group structure might still be alive — then the NVIDIA driver would probe the GPU under identity translation and everything would work.

So the trick is: set the group to identity domain while the group still exists but the device is being removed or re-enumerated. But the group is destroyed when the device is removed...

The assistant catches themselves mid-thought. The IOMMU group is a kernel object that exists only as long as at least one device in the group is present. When all devices are removed, the group is destroyed. This means there is no temporal window in which to switch the domain type — the group vanishes the moment the last device is removed.

Actually wait — let me re-read the earlier output. When I did remove + SBR + rescan, the groups still existed with the same numbers. Let me verify if the group sysfs persists across remove/rescan:

The assistant then executes a targeted test:

ssh root@10.1.2.6 'gpu="0000:01:00.0"; group=$(basename $(readlink /sys/bus/pci/devices/$gpu/iommu_group)); echo "Before remove: group=$group, type=$(cat /sys/kernel/iommu_groups/$group/type)"; echo "$gpu" > /sys/bus/pci/drivers/nvidia/unbind; echo 1 > /sys/bus/pci/devices/$gpu/remove; sleep 1; echo "After remove: group $group exists? $(test -d /sys/kernel/iommu_groups/$group && echo YES || echo NO)"'

The output is definitive:

Before remove: group=28, type=DMA-FQ
After remove: group 28 exists? NO

The IOMMU group is destroyed when the device is removed. There is no gap. The approach is dead.

The Thinking Process: A Detective's Reasoning

What makes this message remarkable is the visible reasoning process. The assistant walks through a logical chain:

  1. Observation: The NVIDIA driver on Blackwell cannot survive unbind/rebind without a hardware power cycle. SBR only works from a clean PCI-removed state where the GPU has been freshly enumerated.
  2. Deduction: When the GPU is removed from the OS and the PCI bridge is reset, the IOMMU domain type at the moment of re-enumeration determines whether identity mode is active. If we could set identity mode before the NVIDIA driver probes the re-enumerated device, the driver would initialize successfully.
  3. Hypothesis: There might be a temporal window between PCI removal and group destruction where the IOMMU group still exists but has no devices, allowing the type to be changed.
  4. Test: Remove a GPU and check whether the IOMMU group persists.
  5. Conclusion: The group is destroyed immediately upon device removal. No window exists. This is classic scientific method applied to systems debugging: observe, hypothesize, test, conclude. The assistant doesn't just try random approaches — they reason about the underlying mechanism and design a targeted experiment to validate or falsify their hypothesis.

Assumptions and Their Validity

The assistant makes several assumptions in this message:

Assumption 1: The IOMMU group might persist after device removal. This turns out to be false, but it was a reasonable hypothesis. In some kernel subsystems, group-like structures can outlive their members. The test conclusively disproves this.

Assumption 2: Setting identity mode before the NVIDIA driver probes would fix the FSP initialization failure. This assumption is not tested in this message but would later be proven false in a different way — the Blackwell FSP fundamentally requires DMA translation mode during its boot sequence, and identity mode breaks it regardless of timing ([chunk 41.0]).

Assumption 3: The group number (28) would be the same after rescan. The assistant notes that earlier attempts showed the same group numbers persisting across remove/rescan cycles, suggesting the IOMMU group numbering is deterministic based on hardware topology.

Input Knowledge Required

To fully understand this message, the reader needs knowledge of:

Output Knowledge Created

This message produces several valuable pieces of knowledge:

  1. Confirmed behavior: IOMMU groups are destroyed when their last member device is removed. The group sysfs directory (/sys/kernel/iommu_groups/N) does not persist.
  2. Negative result: There is no temporal window between PCI removal and IOMMU group destruction in which to change the domain type. This rules out one class of approaches.
  3. Refined understanding: The assistant now understands that the NVIDIA driver's inability to survive unbind/rebind on Blackwell is not just a driver quirk but is tied to the FSP's protected state, which survives SBR and requires a true power cycle to reset.
  4. Methodology validation: The approach of designing targeted experiments (remove a single GPU, check group existence) proves effective for quickly testing hypotheses without disrupting the entire system.

The Broader Significance

This message represents a turning point in the session. After this, the assistant would pivot to a different approach — using a modprobe install hook to set identity domains before the NVIDIA driver loads at boot time. That approach would also fail, but for a different reason: the Blackwell FSP boot sequence itself requires DMA translation mode, and identity mode causes the FSP to fail with error code 0x177 ([chunk 41.0]).

The ultimate conclusion — that per-group IOMMU identity domains are fundamentally incompatible with Blackwell GPUs — is foreshadowed in this message. The assistant is systematically exploring the space of possible approaches and eliminating them one by one. Message [msg 6304] eliminates the "timing trick" approach, forcing the search into new territory.

What makes this message compelling is the raw problem-solving process it reveals. The assistant doesn't just issue commands; they reason about kernel internals, formulate hypotheses, and test them with surgical precision. The "actually wait" self-correction in the middle of the message is a hallmark of genuine debugging — the moment when a clever idea reveals its own flaw before it's even fully articulated.

Conclusion

Message [msg 6304] is a small but pivotal moment in a complex systems debugging journey. It captures the instant when a promising hypothesis is tested and falsified, forcing a pivot to new approaches. The message demonstrates the value of precise reasoning, targeted experimentation, and the willingness to follow evidence wherever it leads — even when it closes off a seemingly elegant solution. In the end, the vanishing IOMMU group would not be the final obstacle; the Blackwell FSP's requirement for DMA translation mode would prove to be the harder constraint. But this message marks the point where the assistant learned that there is no shortcut through the kernel's IOMMU subsystem — the group is destroyed when the device leaves, and there is no gap to exploit.