The Race Against the Modalias: A Blackwell IOMMU Puzzle

In the high-stakes world of deploying large language models across multiple NVIDIA Blackwell GPUs, few things are as frustrating as discovering that a hardware-level optimization you've spent hours chasing is fundamentally impossible. Message 6384 of this opencode session captures exactly such a moment — a turning point where the assistant, having painstakingly rebuilt the GPU stack after a reboot, realizes that the nvidia driver's auto-loading mechanism is sabotaging its attempt to enable peer-to-peer DMA through IOMMU identity domains. The message is a masterclass in systems-level debugging, combining deep knowledge of Linux PCI subsystem mechanics, NVIDIA GPU firmware behavior, and the intricate timing dependencies that govern hardware initialization.

The Context: A Quest for P2P DMA

To understand why message 6384 was written, we need to step back. The broader session involves deploying a Qwen3.5-122B-A10B BF16 model across four NVIDIA RTX PRO 6000 Blackwell GPUs using SGLang, a high-performance inference engine. The system has eight GPUs total, split across two NUMA domains: four GPUs on NUMA0 are dedicated to the SGLang service, while four GPUs on NUMA1 are passed through via VFIO to a confidential computing VM using SEV-SNP.

The problem is that GPU peer-to-peer (P2P) DMA — the ability for GPUs to directly access each other's memory without going through the CPU — is broken. The root cause is the IOMMU (Input/Output Memory Management Unit), which is in full translation mode for security reasons (SEV-SNP requires it). In translation mode, the IOMMU remaps all DMA addresses, and the NVIDIA driver's P2P mechanism relies on specific IOMMU mappings that aren't being set up correctly. The symptom is that NCCL_P2P_DISABLE=1 must be set to prevent hangs and crashes.

The assistant has been pursuing a clever workaround: setting individual IOMMU groups to "identity" mode, which bypasses translation for specific PCI devices while leaving the rest of the system protected. This would allow the NUMA0 GPUs to have direct P2P DMA while the NUMA1 GPUs remain under full translation for the SEV-SNP VM. Previous messages (6374–6383) document the struggle: the assistant built a systemd service to set identity domains before the nvidia driver loads, tested various reset sequences involving Secondary Bus Reset (SBR), and discovered that the Blackwell GPU's Firmware Security Processor (FSP) dies when the GPU is unbound and rebound from the nvidia driver.

The Discovery: Modalias as Saboteur

Message 6384 opens with a moment of clarity. The assistant has just executed a PCI rescan sequence (in message 6383) that successfully brought all four NUMA0 GPUs back to life with the nvidia driver loaded. The GPUs work — nvidia-smi reports all four RTX PRO 6000 Blackwell cards with their full 97,887 MiB of memory. But there's a catch:

GPUs work! But the IOMMU groups are DMA-FQ because the nvidia module was loaded from modalias trigger during the PCI rescan, and the new groups defaulted to DMA-FQ.

This is the critical observation. The IOMMU groups are in DMA-FQ (DMA with Fine-Grained translation) mode rather than identity mode. The assistant immediately identifies the culprit: the kernel's modalias mechanism. When PCI devices appear on the bus during a rescan, the kernel's udev system automatically triggers modprobe nvidia because the PCI device ID (vendor 0x10de, device 0x2bb5) matches the nvidia driver's modalias entry. This happens before any user-space script can intervene to set the IOMMU group type.

The assistant's reasoning here reveals deep understanding of the Linux PCI hotplug chain. The sequence is: (1) PCI rescan discovers devices, (2) kernel creates IOMMU groups with default type (DMA-FQ), (3) kernel fires modalias uevent, (4) udev runs modprobe, (5) nvidia module loads and binds to the devices. Steps 2–5 happen in rapid succession, all before any systemd service or custom script can react. The window for setting identity domains is effectively zero.

The Proposed Solution: A Multi-Stage Dance

The assistant formulates a new approach, articulated in the message's reasoning section:

The solution: we need to set the IOMMU group default domain BEFORE the device is re-enumerated, or we need to prevent nvidia from being triggered by modalias and manually load it after setting identity.

The first option — setting the domain before enumeration — is impossible because IOMMU groups don't exist until the device appears on the bus. So the assistant pivots to the second option: a carefully orchestrated sequence that blocks the nvidia driver from auto-loading, rescans the PCI bus, sets identity domains while the GPUs are driverless, and then manually loads nvidia.

The bash script embedded in the message implements this in ten stages:

  1. Unbind and unload nvidia: Remove any existing nvidia driver bindings and unload all nvidia kernel modules (nvidia, nvidia_modeset, nvidia_uvm, nvidia_drm).
  2. Remove GPUs from PCI bus: Write 1 to each GPU's remove sysfs file, which logically disconnects the device from the PCI hierarchy.
  3. Secondary Bus Reset (SBR): For each parent PCI bridge, set the SBR bit in the bridge control register, wait, then clear it. This issues a hardware reset to everything downstream of the bridge, including the GPUs. The 15-second wait is critical — the Blackwell FSP needs time to reinitialize after reset.
  4. Block driver autoload: Two actions here. First, remove the VFIO-PCI driver's new_id binding so it doesn't grab the GPUs. Second, create a modprobe configuration file (/etc/modprobe.d/tmp-nvidia-block.conf) containing install nvidia /bin/false, which replaces the normal module loading with a command that does nothing.
  5. Rescan PCI bus: Trigger a full PCI bus rescan. Because nvidia is blocked, the GPUs appear with no driver bound.
  6. Verify GPUs unbound: Check that all four GPUs exist on the bus with driver=none.
  7. Set identity domains: For each GPU's IOMMU group, write identity to the group's type file. This is the critical step — it must happen while no driver is bound to the GPU.
  8. Remove block, load nvidia: Delete the modprobe block file and load nvidia. The assistant includes a fallback: if modprobe nvidia fails (because the module might need devices present at load time), manually trigger probe by writing each GPU's BDF to /sys/bus/pci/drivers_probe.
  9. Re-register VFIO-PCI: Restore the VFIO-PCI new_id binding so the NUMA1 GPUs (which should not be touched) remain on VFIO-PCI.
  10. Final verification: Run nvidia-smi and check each GPU's driver, IOMMU group, and type.

Assumptions Embedded in the Approach

The assistant makes several assumptions in this message, some explicit and some implicit:

That blocking modprobe prevents nvidia from binding during rescan. This is correct in theory — the install nvidia /bin/false trick replaces the module loading command with a no-op. However, it assumes that nvidia is only loaded via the modprobe mechanism and not through some other kernel path (e.g., direct driver matching during device enumeration).

That identity domains can be set on a driverless device. The IOMMU group type file is writable regardless of whether a driver is bound. The assistant assumes that once set to identity, the type will persist when nvidia later binds. This is a reasonable assumption but untested at this point.

That manually triggering drivers_probe works equivalently to modalias-triggered loading. The kernel's drivers_probe interface iterates over registered drivers and attempts to match them to the specified device. This should work identically to modalias-triggered probing, but the assistant hedges with a fallback comment.

That the Blackwell FSP will initialize correctly when nvidia binds to a GPU that was reset via SBR. Previous attempts (messages 6381–6382) showed that SBR alone doesn't reset the FSP — the error 0xbadf4100 and "WPR2 already up" messages indicated the firmware state persisted. But message 6383 demonstrated that when nvidia is loaded during rescan (rather than bound to an already-present device), the initialization succeeds. The assistant is betting that this timing dependency is the key.

That the 120-second timeout is sufficient. The bash tool has a 120-second timeout, and the script includes multiple sleeps (2s + 15s + 5s + 5s + 5s + 3s = 35s minimum, plus time for module loading and device probing). The timeout is tight but should be adequate — unless something hangs.

What Goes Wrong: The Timeout

The message ends with the bash tool being terminated after exceeding the 120-second timeout. The last visible output is:

=== 1. Unbind + unload nvidia ===

The script didn't even get past the first step before something hung. The unbind operations — writing each GPU's BDF to /sys/bus/pci/drivers/nvidia/unbind — apparently blocked. This is a known issue: unbinding a GPU from the nvidia driver while the GPU's firmware is in an active state can cause the unbind operation to hang indefinitely, especially on Blackwell where the FSP communication channel may be in a bad state.

The timeout is itself a significant outcome. It tells us that the GPUs are in a state where even unbinding from nvidia is non-functional. This is consistent with the earlier observation (message 6382) that the FSP is stuck with "WPR2 already up" and "Cannot initialize GSP firmware RM." The nvidia driver's unbind path apparently tries to communicate with the GSP/FSP to shut it down gracefully, and if the firmware is hung, this communication never completes.

Input Knowledge Required

To fully understand this message, the reader needs knowledge spanning several domains:

Linux PCI subsystem internals: Understanding of BDF (Bus:Device.Function) addresses, PCI rescan, driver binding/unbinding via sysfs, and the drivers_probe interface.

IOMMU architecture: Knowledge of IOMMU groups, domain types (DMA-FQ vs. identity), and how the IOMMU affects DMA transactions. The distinction between translation mode and identity mode is central.

NVIDIA GPU driver architecture: Familiarity with the nvidia kernel module stack (nvidia, nvidia_modeset, nvidia_uvm, nvidia_drm), the GSP (GPU System Processor) and FSP (Firmware Security Processor) on Blackwell GPUs, and how the driver initializes GPU firmware during probe.

Modalias and udev: Understanding of how Linux automatically loads drivers when new PCI devices appear, via the modalias mechanism and udev's drivers_probe triggers.

Blackwell-specific quirks: The fact that Blackwell's FSP requires specific DMA mappings during initialization, that SBR doesn't fully reset the firmware state, and that unbind/rebind cycles can permanently corrupt the GSP session.

System administration on Proxmox/SEV-SNP hosts: The context of GPU splitting between an LXC container and a confidential VM, VFIO-PCI passthrough, and the constraints imposed by SEV-SNP's IOMMU requirements.

Output Knowledge Created

This message, despite its timeout, creates significant knowledge:

The modalias problem is confirmed: The assistant has definitively identified why previous attempts failed — the nvidia driver auto-loads during PCI rescan before identity domains can be set. This is a timing constraint that cannot be worked around without blocking modprobe.

The unbind hang is a new data point: The 120-second timeout on the unbind operation reveals that the nvidia driver's unbind path can hang when the FSP is in a corrupted state. This is valuable diagnostic information for anyone else working with Blackwell GPUs.

The ten-stage sequence is a documented approach: Even though it timed out, the script represents a well-reasoned attempt that future efforts can build upon. The structure — unbind, remove, SBR, block, rescan, set identity, load, verify — is a logical template.

The fundamental constraint is clearer: The message makes explicit that the window for setting identity domains is between device enumeration and driver binding, and that this window is effectively zero under normal operation. Any solution must either (a) prevent the driver from binding automatically, or (b) set the domain type before enumeration (which is impossible since groups don't exist yet).

The Thinking Process: A Window into Debugging

The assistant's reasoning in this message is particularly instructive. The opening line — "GPUs work! But the IOMMU groups are DMA-FQ" — captures the mixed emotions of a partial success. The GPUs are alive, which is a victory after the FSP corruption issues of previous messages, but the IOMMU configuration is wrong.

The assistant then performs a rapid diagnosis: "the nvidia module was loaded from modalias trigger during the PCI rescan, and the new groups defaulted to DMA-FQ." This connects two observations — that nvidia loaded automatically and that the groups are in the wrong mode — into a causal chain. The modalias trigger is the root cause.

The formulation of the solution shows the assistant working through constraints: "set the IOMMU group default domain BEFORE the device is re-enumerated, or prevent nvidia from being triggered by modalias." The first option is quickly dismissed as impossible, leading to the second. The ten-stage script is then constructed as a concrete implementation of the second option.

What's notable is the hedging in the script. The assistant includes a comment: "If modprobe failed because nvidia is already unloaded and needs devices first, try triggering probe manually." This shows awareness that the module loading path might not work as expected, and includes a fallback. Similarly, the script checks for driver=none before attempting drivers_probe, showing careful state management.

The Broader Significance

Message 6384 sits at a critical juncture in the segment. The chunk summary reveals that ultimately, the identity domain approach is abandoned — the Blackwell FSP requires DMA translation mode during initialization, making identity domains fundamentally incompatible. The assistant reverts the modprobe hook and reboots, restoring the working DMA-FQ configuration.

But this message is where the final attempt is made. It's the last roll of the dice before the assistant accepts that P2P DMA via identity domains is impossible on Blackwell. The timeout is almost poetic — the system itself refuses to cooperate, hanging on the unbind operation as if to say "you can't undo what's already been done."

For the reader, this message offers a rare view into the debugging process at the intersection of GPU firmware, kernel driver mechanics, and IOMMU virtualization. It shows how even a well-reasoned approach can fail due to unforeseen timing constraints, and how the most valuable output of a failed experiment is often the precise documentation of why it failed. The modalias problem, the unbind hang, and the ten-stage sequence are all contributions to the collective understanding of Blackwell GPU deployment under IOMMU translation — knowledge that will serve anyone who follows this path.