The Autoloading Trap: Debugging Blackwell GPU Initialization Under IOMMU Identity Domains

In the high-stakes world of large-scale ML inference deployment, few things are as frustrating as a hardware initialization problem that refuses to yield. The message at [msg 6291] captures a pivotal moment in an intense debugging session: the assistant has been fighting for hours to enable Peer-to-Peer (P2P) DMA between NVIDIA Blackwell RTX PRO 6000 GPUs by switching their IOMMU groups from the default DMA-FQ (DMA with Fast-path Queuing) translation mode to identity mode (which bypasses DMA translation entirely). The goal is straightforward—P2P DMA allows GPUs to exchange data directly across the PCIe bus without bouncing through host memory, a critical performance optimization for tensor-parallel inference serving. But Blackwell's Firmware Security Processor (FSP) has proven extraordinarily resistant to cooperating.

This message is a study in iterative debugging under tight hardware constraints, where each failed attempt reveals a new layer of complexity in the Linux kernel's PCI subsystem, the IOMMU, and NVIDIA's proprietary driver stack. It is the kind of message that looks like a simple bash script at first glance but contains deep insight into the race conditions and timing dependencies that govern hardware initialization.

The Problem: Blackwell's Fragile GSP State

To understand why this message matters, we must first understand what the assistant is fighting against. The NVIDIA Blackwell architecture introduces a Firmware Security Processor (FSP), also referred to as the GPU System Processor (GSP), which handles low-level GPU initialization, security, and firmware management. The GSP firmware is loaded during the nvidia driver's probe sequence and, crucially, maintains state in GPU memory that persists across driver unbind/bind cycles. If the driver is unloaded and reloaded without a full hardware reset—specifically a Secondary Bus Reset (SBR) on the parent PCIe bridge—the GSP enters a corrupted state and fails with error code 0x177.

The assistant discovered this in earlier messages ([msg 6282], [msg 6287]) while attempting to set IOMMU identity domains at runtime. The standard approach—unbind from nvidia, set identity, rebind to nvidia—consistently produced RmInitAdapter failed! (0x62:0xffff:2142) errors in dmesg, indicating the GSP could not initialize. The only reliable recovery was to physically remove the GPU from the PCI bus, perform an SBR on its parent bridge (which resets the GSP firmware), and rescan.

This led to a refined procedure: unload all nvidia kernel modules, remove the GPUs from PCI, SBR all bridges, rescan the bus, set identity domains while the GPUs are driverless, then reload the nvidia modules. The SBR clears the GSP state, and loading the driver fresh onto a clean GPU should work—provided the identity domain is set before the driver probes the device.

Partial Success and a New Problem

The message immediately preceding this one ([msg 6290]) attempted exactly this procedure. The assistant unloaded nvidia modules, removed GPUs, performed SBRs, rescanned, and then checked the state. The result was perplexing: two GPUs (0000:01:00.0 and 0000:71:00.0) had been grabbed by the nvidia driver during the rescan, while two others (0000:11:00.0 and 0000:61:00.0) remained driverless. The assistant had successfully set identity on the two driverless GPUs, but the two that were already bound to nvidia were stuck with DMA-FQ.

How could this happen? The assistant had explicitly run rmmod nvidia and confirmed "nvidia fully unloaded." Yet during the PCI rescan, something reloaded the nvidia module. This is the critical observation that drives message [msg 6291].

The Message: Identifying the Autoloading Race

The message opens with the assistant's reasoning:

Interesting — partial success! Two GPUs (01:00.0 and 71:00.0) got grabbed by nvidia during the rescan before I could set identity. Two others (11:00.0 and 61:00.0) were left without a driver and I set identity on them. It seems the nvidia module was still in memory or got loaded from somewhere.

>

Wait — I see 2 GPUs work in nvidia-smi. The issue is that modprobe nvidia was removed but during PCI rescan, some subsystem loaded it. Let me look more carefully — rmmod nvidia was successful but PCI rescan triggered autoloading.

This moment of realization is the heart of the message. The assistant correctly identifies that PCI rescan is triggering automatic module loading via the kernel's kmod infrastructure. When the PCI subsystem discovers a device with a known vendor/device ID (in this case, NVIDIA's PCI ID 10de:2bb5 for the Blackwell RTX PRO 6000), it consults the module alias database and automatically loads the matching driver module. This happens asynchronously during the rescan—the echo 1 > /sys/bus/pci/rescan command triggers a cascade of events that include driver probing, and by the time the script checks the state, the nvidia module is already back in memory and has claimed the GPUs.

The assistant's proposed solution is elegant:

I need to blacklist nvidia temporarily during the rescan window.

This is the key innovation in this message. Rather than fighting the kernel's autoloading mechanism, the assistant decides to preempt it by creating a temporary modprobe blacklist file before the rescan, then removing it after the identity domains are set and before manually loading the nvidia modules.

The Script: A Carefully Orchestrated Sequence

The bash script that follows is a masterpiece of hardware initialization choreography. Let me quote it in full:

ssh root@10.1.2.6 'set -e

echo "=== Current state ==="
nvidia-smi --query-gpu=index,gpu_bus_id --format=csv,noheader 2>&1
lsmod | grep nvidia

echo ""
echo "=== Will need full restart. First unbind all nvidia GPUs ==="
GPUS="0000:01:00.0 0000:11:00.0 0000:61:00.0 0000:71:00.0"
for gpu in $GPUS; do
    echo "$gpu" > /sys/bus/pci/drivers/nvidia/unbind 2>/dev/null || true
done
sleep 1

echo ""
echo "=== Unload nvidia modules ==="
rmmod nvidia_drm 2>/dev/null || true
rmmod nvidia_uvm 2>/dev/null || true
rmmod nvidia_modeset 2>/dev/null || true
rmmod nvidia 2>/dev/null || true
lsmod | grep nvidia && echo "WARN: nvidia still loaded" || echo "nvidia fully unloaded"

echo ""
echo "=== Create temp blacklist to prevent autoload ==="
echo "blacklist nvidia" > /etc/modprobe.d/temp-nvidia-blacklist.conf
echo "blacklist nvidia_drm" >> /etc/modprobe.d/temp-nvidia-blacklist.conf  
echo "blacklist nvidia_modeset" >> /etc/modprobe.d/temp-nvidia-blacklist.conf
echo "blacklist nvidia_uvm" >> /etc/modprobe.d/temp-nvidia-blacklist.conf

echo ""
echo "=== Remove GPUs ==="
for gpu in $GPUS; do
    echo 1 > /sys/bus/pci/devices/$gpu/remove 2>/dev/null || true
done
sleep 2

echo ""
echo "=== SBR on bridges ==="
for bridge in 0000:00:01.1 0000:10:01.1 0000:60:01.1 0000:70:01.1; do
    bridge_ctl=$(setpci -s $bridge BRIDGE_CONTROL)
    new_val=$(printf "%04x" $((0x$bridge_ctl | 0x40)))
    setpci -s $bridge BRIDGE_CONTROL=$new_val
    sleep 0.5
    setpci -s $bridge BRIDGE_CONTROL=$bridge_ctl
done
echo "  SBR complete, waiting 10s..."
sleep 10

echo ""
echo "=== Prevent vfio-pci from grabbing ==="
echo "10de 2bb5" > /sys/bus/pci/drivers/vfio-pci/remove_id 2>/dev/null || true

echo ""
echo "=== Rescan ==="
echo 1 > /sys/bus/pci/rescan
sleep 5

echo ""
echo "=== Check GPU state (should all be driver=none) ==="
for gpu in $GPUS; do
    if [ -e /sys/bus/pci/devices/$gpu ]; then
        driver=$(basename $(readlink /sys/bus/pci/devices/$gpu/driver) 2>/dev/null || echo "none")
        echo "  $gpu: driver=$driver"
    else
        echo "  $gpu: NOT FOUND"
    fi
done
lsmod | grep nvidia && echo "WARN: nvidia loaded!" || echo "nvidia NOT loaded (good)"'

The script is structured as a sequence of distinct phases, each with appropriate waits:

  1. State snapshot: Check what's currently running and which GPUs are visible.
  2. Unbind: Release GPUs from the nvidia driver.
  3. Module unload: Remove all four nvidia kernel modules in reverse dependency order (nvidia_drm → nvidia_uvm → nvidia_modeset → nvidia).
  4. Blacklist creation: Write a temporary modprobe configuration that blacklists all nvidia modules.
  5. PCI removal: Remove the GPU devices from the PCI bus entirely.
  6. SBR: Perform Secondary Bus Reset on each parent bridge, holding the reset bit for 500ms.
  7. SBR wait: A 10-second pause to ensure the GSP firmware fully resets.
  8. vfio-pci cleanup: Remove the new_id registration that would cause vfio-pci to claim the GPUs.
  9. Rescan: Trigger PCI bus rescan—now safe because nvidia is blacklisted.
  10. Verification: Confirm all GPUs are present with no driver bound and nvidia module is not loaded. The attention to timing is notable. The SBR uses a 500ms hold time (longer than the 200ms used in earlier attempts), and the post-SBR wait is 10 seconds. These delays reflect the assistant's growing understanding that the GSP firmware reset is not instantaneous and that racing against hardware state transitions is a losing game.

Assumptions and Potential Pitfalls

The message makes several assumptions worth examining:

Assumption 1: The temporary blacklist will be respected during PCI rescan. Modprobe blacklist files prevent modprobe from loading a module automatically, but they do not prevent direct insmod or loading initiated by the kernel's device-probing infrastructure in all cases. The blacklist mechanism works at the modprobe level—if the kernel's PCI subsystem uses a different path to load the module (e.g., through request_module which calls modprobe), the blacklist should work. But there are edge cases where modules can be loaded via kmod directly, bypassing the blacklist.

Assumption 2: The SBR with 500ms hold time is sufficient to clear GSP state. Earlier attempts used 200ms, and the assistant has empirically determined that longer is better. But the actual reset time required by the Blackwell FSP is undocumented; 500ms is a heuristic.

Assumption 3: vfio-pci's remove_id will prevent it from claiming the GPUs. The remove_id operation removes a dynamic device ID registration, but if vfio-pci is already bound to the device, unbinding must happen first. The script handles this by unbinding from nvidia (which may leave the device available for vfio-pci during rescan if new_id is still active).

Assumption 4: The blacklist file will not interfere with later manual module loading. After identity domains are set, the assistant plans to remove the blacklist and manually modprobe nvidia. This should work because the blacklist only prevents automatic loading; explicit modprobe commands are unaffected.

Input Knowledge Required

To fully understand this message, one needs knowledge of:

Output Knowledge Created

This message produces several important pieces of knowledge:

  1. The autoloading race condition: PCI rescan triggers kernel module autoloading, which can race against IOMMU domain configuration. This is a timing-dependent failure mode that would be difficult to reproduce without understanding the underlying mechanism.
  2. The temporary blacklist workaround: A proven technique for preventing module autoloading during a controlled hardware reinitialization sequence. This is reusable for any scenario where a driver must be kept away from a device during a transient state.
  3. The complete initialization sequence: A step-by-step procedure for resetting Blackwell GPUs and configuring IOMMU identity domains, including timing parameters (500ms SBR hold, 10s GSP reset wait, 5s rescan wait).
  4. The verification pattern: A systematic approach to checking state after each phase, using multiple sources of truth (nvidia-smi, lsmod, /sys filesystem, dmesg).

The Thinking Process

The assistant's reasoning in this message is a textbook example of iterative debugging:

  1. Observe unexpected behavior: Two GPUs got grabbed by nvidia during rescan despite the module being unloaded.
  2. Form hypothesis: PCI rescan triggered autoloading of the nvidia module.
  3. Test hypothesis mentally: "rmmod nvidia was successful but PCI rescan triggered autoloading." This is consistent with the kernel's device-probing behavior.
  4. Design solution: Prevent autoloading during the critical window using a temporary blacklist.
  5. Implement solution: Write the blacklist file before rescan, remove it after identity domains are set.
  6. Add defensive measures: Also remove vfio-pci's new_id to prevent vfio from grabbing the GPUs. The "Wait — I see 2 GPUs work in nvidia-smi" self-correction is particularly telling. The assistant initially thought the nvidia module "was still in memory" but then realizes the more precise explanation: the module was unloaded but got reloaded by the rescan. This refinement of understanding is what separates a good diagnosis from a great one.

What Happened Next

The output shown at the end of the message confirms the script executed successfully through the GPU removal step. The following message ([msg 6292]) picks up where this left off: "All 4 GPUs detected with NO driver bound, and nvidia module is NOT loaded. Now I can set identity domains and then load nvidia." The assistant then sets identity on all four groups, removes the blacklist, loads nvidia modules, and checks the result.

The final outcome, however, was not success. Despite all this careful orchestration, the nvidia driver still failed with "No devices were found" after loading. The GSP initialization failed even with identity domains set before driver probe. As the chunk summary reveals, the fundamental issue is that Blackwell's FSP requires DMA translation mode during initialization—the FSP firmware boot sequence needs specific DMA mappings set up by the kernel's DMA API, and identity mode breaks this initialization with error code 0x177. This is a hardware/firmware limitation that no amount of software sequencing can overcome.

Broader Implications

This message illustrates a deeper truth about modern GPU computing: the hardware initialization path is increasingly complex and fragile. Blackwell's FSP represents a new layer of firmware that was absent in Hopper and earlier architectures. This firmware has its own requirements and assumptions about the system state, and violating those assumptions—even with the best intentions of improving performance—can render the GPUs unusable.

The IOMMU identity domain approach was an attempt to solve a real problem: P2P DMA corruption under SEV-SNP IOMMU. The assistant had previously discovered that NCCL_P2P_DISABLE=1 was necessary to prevent hangs and data corruption (<msg id=6290 context>), but this disables P2P entirely, hurting performance. Identity domains promised to restore P2P while keeping IOMMU protection for non-GPU devices. The discovery that Blackwell's FSP cannot initialize under identity mode closes this path permanently.

The lesson is that hardware security features (FSP, IOMMU) interact in ways that are not always documented or anticipated. The Blackwell FSP's dependency on DMA translation mode may be a deliberate security design—the FSP itself may use DMA to load its firmware from host memory, and identity mode would bypass the IOMMU protections that the FSP relies on. Or it may be an unintended side effect of the firmware's initialization sequence. Either way, the constraint is real and must be accepted.

Conclusion

Message [msg 6291] is a snapshot of a debugging session at its most productive: the moment when an unexpected failure mode is correctly identified and a targeted countermeasure is designed. The temporary blacklist technique is clever, well-reasoned, and correctly implemented. Even though the ultimate goal (P2P DMA via identity domains) proved unattainable due to a deeper hardware constraint, the work in this message was not wasted—it eliminated a plausible approach and narrowed the search space. In systems engineering, knowing that something cannot work is often as valuable as knowing what can.

The message also demonstrates the importance of understanding the full stack, from PCIe bridge registers at the hardware level to modprobe configuration files at the OS level. The assistant's ability to reason across these layers—to connect a race condition in kernel module autoloading to a fix involving a text file in /etc/modprobe.d/—is what makes this kind of debugging possible. It is a reminder that in complex systems, the solution to a hardware problem may lie in software, and the solution to a software problem may lie in understanding the hardware.