The Blackwell FSP Barrier: A Deep Dive Into IOMMU Identity Domains and GPU Firmware Initialization

Introduction

In the complex ecosystem of high-performance GPU computing, few problems are as frustrating as a hardware component that refuses to initialize after a seemingly innocuous configuration change. Message 6322 of this extended coding session captures a pivotal moment of debugging—one where the assistant, after hours of painstaking experimentation, arrives at a deep understanding of why Blackwell GPUs fail to boot when their IOMMU groups are placed in identity (passthrough) mode. This message is not merely a log of commands executed; it is a window into the reasoning process of an experienced systems engineer grappling with the intersection of GPU firmware, kernel IOMMU infrastructure, and PCI subsystem mechanics.

The message sits at a critical juncture: the assistant has just recovered from a catastrophic failure where all four NVIDIA RTX PRO 6000 Blackwell GPUs stopped appearing in nvidia-smi after repeated attempts to set IOMMU identity domains. The GPUs were restored by a specific sequence—loading the nvidia driver module first, then rescanning the PCI bus—but they came back with DMA-FQ (default translation-mode IOMMU) rather than the desired identity mode. Now the assistant must figure out whether identity domains can be made to work with Blackwell GPUs at all, or whether this path is fundamentally blocked.

The Context: Why Identity Domains Matter

To understand the significance of this message, one must first understand the problem that drove the assistant to pursue IOMMU identity domains in the first place. The system in question is a high-end machine running Ubuntu 24.04 with eight NVIDIA RTX PRO 6000 Blackwell GPUs, split between an LXC container and a SEV-SNP (Secure Encrypted Virtualization - Secure Nested Paging) virtual machine. The four GPUs assigned to the LXC container need to communicate with each other via P2P (peer-to-peer) DMA for multi-GPU model inference—specifically, serving large language models like Qwen3.5-122B with tensor parallelism across four GPUs.

The problem is that with IOMMU enabled (which is required for SEV-SNP), P2P DMA between GPUs becomes unreliable. The IOMMU, when operating in its default DMA-FQ (DMA with Fine-grained Queued) translation mode, remaps all DMA addresses through translation tables. While this provides security isolation, it also introduces complications for GPU peer-to-peer transfers. The nvidia driver has a parameter DmaRemapPeerMmio=1 that attempts to set up the necessary IOMMU mappings for P2P, but on this system it produces incomplete mappings—some peer pairs work while others cause IOMMU faults.

The alternative is to set the IOMMU group type to identity (also called passthrough), which bypasses translation for devices in that group. With identity domains, DMA addresses pass through the IOMMU unchanged, which should allow P2P DMA to function correctly. The kernel's IOMMU subsystem allows changing the domain type of an IOMMU group, but only when no driver is bound to any device in that group. This constraint is what sets the stage for the complex dance of unloading drivers, removing devices, and rescanning that occupies the preceding messages.

The Message: Reasoning Through a Fundamental Constraint

Message 6322 opens with the assistant confirming the current state: all four GPUs are back to DMA-FQ mode. This is because the assistant had explicitly set them to DMA-FQ before the last rmmod operation, and the IOMMU groups were re-created during the PCI rescan. The GPUs are working, but without identity domains—meaning P2P DMA remains broken.

What follows is a remarkable piece of real-time reasoning. The assistant lays out the three constraints that define the problem space:

1. The nvidia module needs to be loaded (in memory) when the GPUs are rescanned 2. The IOMMU groups need to be identity when nvidia probes 3. But we can't change groups while nvidia is bound, and we can't unbind/rebind nvidia on Blackwell

This triad of constraints captures the essence of the dilemma. The first constraint was discovered empirically in the immediately preceding messages: when the assistant tried to rescan the PCI bus without the nvidia module pre-loaded, the GPUs either failed to appear or appeared with corrupted FSP state. The working sequence was: load nvidia module first, then rescan. This suggests that the nvidia driver's probe function handles fresh PCI devices differently when the module is already resident in memory versus when it is loaded as part of the probe process.

The second constraint is the goal itself: identity domains must be active at the moment the nvidia driver probes the GPU, because the IOMMU domain type is set once during device initialization and cannot be changed while a driver is bound.

The third constraint is the killer: Blackwell GPUs have a Firmware Security Processor (FSP) that, once initialized by the nvidia driver, cannot be cleanly reset through software means. The assistant had already tried FLR (Function Level Reset), SBR (Secondary Bus Reset), and even CXL bus reset—none of them cleared the FSP state. This means the standard approach of "unbind from nvidia, change IOMMU type, rebind to nvidia" is impossible, because unbinding corrupts the FSP and rebinding fails.

The Hypothesis: Persistent IOMMU Group Types

Faced with this deadlock, the assistant formulates a new hypothesis. The key insight comes from recalling earlier observations:

Wait, actually, I recall from the earlier tests that when I had previously set groups to identity and then removed/rescanned GPUs, the new groups showed identity too. That suggests the kernel may remember the preferred type for IOMMU groups.

This is a crucial observation about kernel behavior. When PCI devices are removed from the system (via the remove sysfs attribute), their IOMMU groups are destroyed. When the devices reappear during a rescan, new IOMMU groups are created. The question is: do the new groups inherit the type that was set on the old groups, or do they default to DMA-FQ?

If the kernel remembers the preferred type—perhaps through the IOMMU group's type attribute persisting in some internal data structure even after the group is destroyed, or through the PCI device's ACPI/DMAR table information—then the assistant could set identity on the groups, remove the GPUs, perform an SBR to clear the FSP state, and then rescan. The new groups would come up as identity, and if the nvidia module is pre-loaded, it would probe the fresh GPUs with identity domains already active.

This is a clever workaround that avoids the fundamental constraint of not being able to change IOMMU types while nvidia is bound. Instead of unbinding and rebinding, the assistant would destroy and recreate the PCI devices entirely, with the IOMMU type set before destruction.

The Execution: A Carefully Orchestrated Sequence

The bash command in the message implements this hypothesis through a nine-step sequence, each step carefully designed to avoid the pitfalls discovered in earlier attempts:

Step 1: Blacklist nvidia. A temporary modprobe configuration file is created that sets install nvidia /bin/false for all nvidia kernel modules. This prevents any automatic loading of the nvidia driver during the PCI rescan, which would interfere with the plan.

Step 2: Unbind GPUs from nvidia. Each GPU is explicitly unbound from the nvidia driver via the sysfs unbind attribute. This is necessary before the nvidia module can be unloaded.

Step 3: Unload nvidia. All nvidia kernel modules (nvidia, nvidia_modeset, nvidia_uvm, nvidia_drm) are removed. The assistant verifies the modules are cleanly unloaded.

Step 4: Set identity domains. With the GPUs unbound and nvidia unloaded, the assistant sets each GPU's IOMMU group type to identity. This is the critical configuration step.

Step 5: Remove GPUs from PCI. Each GPU is removed from the PCI subsystem via the remove sysfs attribute. This destroys the PCI device objects and their associated IOMMU groups.

Step 6: Secondary Bus Reset. The assistant performs SBR on each parent PCI bridge, holding the reset signal for 2 seconds. The extended hold time and 15-second wait afterward are intended to give the GPU firmware (including the FSP) time to fully reset. Earlier attempts with shorter hold times failed to clear the FSP state.

Step 7: Remove blacklist and load nvidia before rescan. This is the key insight from the previous recovery: the nvidia module must be loaded before the PCI rescan, so that when new GPU devices appear, the nvidia driver is already resident and can probe them immediately.

Step 8: Prevent vfio-pci from grabbing. The GPU vendor/device ID is removed from the vfio-pci driver's dynamic ID list, preventing vfio-pci from binding to the GPUs during rescan.

Step 9: Rescan PCI. The PCI bus is rescanned, which should cause the GPUs to reappear with fresh FSP state and identity IOMMU domains.

The Assumptions Embedded in This Approach

The assistant's plan rests on several assumptions, some explicit and some implicit:

Assumption 1: IOMMU group type persists across device removal and recreation. This is the central hypothesis. The assistant is betting that setting identity on an IOMMU group, then removing the devices in that group, will cause the kernel to remember the identity setting and apply it to any new group created for the same PCI devices after rescan. This is not documented kernel behavior—it's an inference from earlier observations.

Assumption 2: SBR with the GPU removed from the OS can clear the FSP state. Earlier attempts showed that SBR performed while the GPU was still present in the PCI hierarchy (even if unbound from nvidia) did not clear the FSP. The assistant is hoping that removing the GPU entirely from the PCI bus before performing SBR will allow the reset to propagate more deeply into the GPU's internal firmware.

Assumption 3: The nvidia driver's probe function behaves identically whether it probes a device that appeared during module load or one that appeared after module load. The assistant observed that loading nvidia first and then rescanning works, but this might be a specific behavior of the driver's hot-plug handling versus its initial enumeration.

Assumption 4: The blacklist mechanism is reliable. The assistant uses install nvidia /bin/false in modprobe.d to prevent automatic loading. This is a well-known technique, but it's being used in a scenario where udev or other mechanisms might trigger module loading through different paths.

The Thinking Process: A Case Study in Systems Debugging

What makes this message particularly valuable as a case study is the visible reasoning process. The assistant doesn't just execute commands—it thinks through the problem, formulates hypotheses, tests them against observed behavior, and refines its understanding.

The message begins with a moment of clarity: "Now I understand the exact problem and the only viable approach." This is the culmination of a long debugging session that started with simple attempts to set IOMMU identity domains and progressively uncovered deeper issues with Blackwell's FSP architecture.

The assistant's thinking shows several hallmarks of expert debugging:

  1. Constraint enumeration: The assistant explicitly lists the three constraints that define the solution space. This is a powerful technique for complex problems—by writing down what must be true, what must be false, and what cannot be changed, the solver can identify the narrow path through the solution space.
  2. Memory of past observations: The assistant recalls a seemingly minor observation from earlier tests—that identity settings seemed to persist across remove/rescan cycles. This kind of cross-referencing between different experimental runs is essential for making progress on hard problems.
  3. Willingness to question assumptions: The assistant considers whether the identity setting might be remembered by the kernel, even though this is not documented behavior. This willingness to explore unorthodox possibilities is often what separates successful debugging from dead ends.
  4. Incremental experimentation: Rather than trying a completely different approach, the assistant builds on what worked before (the "load nvidia first, then rescan" sequence) and adds the new variable (setting identity before removal).

The Technical Depth: IOMMU Groups and PCI Device Lifecycle

To fully appreciate the sophistication of this message, one needs to understand the lifecycle of a PCI device and its relationship to IOMMU groups in the Linux kernel.

When a PCI device is first discovered (either during initial boot enumeration or during a hot-plug rescan), the kernel creates a pci_dev structure and associates it with an IOMMU group. The IOMMU group is determined by the device's position in the PCI hierarchy and the system's IOMMU topology (usually described by ACPI DMAR tables or device-tree entries). Each IOMMU group has a type attribute that controls whether DMA from devices in that group goes through translation (DMA-FQ, DMA-LE, etc.) or bypasses translation (identity).

The critical detail is that the IOMMU group type is set once during device initialization and cannot be changed while any device in the group has a driver bound. This is enforced by the kernel's IOMMU subsystem, which checks for bound drivers before allowing a type change. The rationale is that changing the IOMMU type while a driver is actively using DMA mappings would cause address translation mismatches and data corruption.

When a PCI device is removed via the remove sysfs attribute, the kernel tears down the device, releases its IOMMU mappings, and destroys the IOMMU group (if no other devices remain in the group). When the device reappears (via rescan or hot-plug), a new IOMMU group is created. The assistant's hypothesis is that the new group's type is determined by some persistent state—perhaps the iommu_group structure's type field is cached in the PCI device's firmware node, or the kernel's IOMMU subsystem remembers the last type set for that group's ID.

If this hypothesis is correct, then the sequence of "set identity → remove → SBR → rescan" would produce GPUs that are probed by nvidia with identity domains already active, potentially solving the P2P DMA problem without ever needing to unbind nvidia from a running GPU.

The Stakes: What Hangs on This Experiment

The outcome of this experiment determines the entire future direction of the system's configuration. If identity domains can be made to work with Blackwell GPUs, then P2P DMA can be restored, and the system can achieve optimal performance for multi-GPU inference workloads. If identity domains are fundamentally incompatible with Blackwell's FSP—as the earlier 0x177 error code suggested—then the assistant must abandon this approach and find alternative ways to optimize performance.

The assistant has already identified a fallback: MTP (Multi-Token Prediction) speculation, which provides 12-45% throughput improvement even without P2P DMA. But MTP speculation is a software-level optimization that improves inference throughput by predicting multiple tokens per forward pass, whereas P2P DMA is a hardware-level capability that reduces latency and overhead for inter-GPU communication. For large models split across multiple GPUs, P2P DMA can make a dramatic difference in both latency and throughput.

The message ends with the command executing, and the output visible shows progress through step 7 (loading nvidia before rescan). The final outcome—whether the GPUs appear with identity domains and functional FSP—will be revealed in the next message. But regardless of the outcome, the reasoning process captured in this message represents a deep understanding of the interaction between GPU firmware, kernel IOMMU infrastructure, and PCI subsystem mechanics.

Conclusion

Message 6322 is a masterclass in systems-level debugging. It captures the moment when an engineer, after hours of experimentation and failure, synthesizes everything learned into a coherent hypothesis and a carefully designed experiment. The message reveals not just what was done, but why—the constraints, the assumptions, the remembered observations, and the logical chain that connects them.

Whether the experiment succeeds or fails, the understanding gained is permanent. The assistant now knows that Blackwell's FSP is sensitive to IOMMU configuration in ways that previous GPU generations were not. It knows that software resets cannot clear the FSP state once corrupted. And it has identified the precise sequence of operations that can recover from FSP failure: load nvidia first, then rescan.

This kind of deep, experiential knowledge is the true product of debugging sessions like this one. It cannot be found in documentation or datasheets—it must be discovered through the iterative process of hypothesis, experiment, observation, and refinement. Message 6322 captures that process in action, making it a valuable artifact for anyone who works at the intersection of GPU computing and systems software.---

Full Text of the Subject Message

For reference, here is the complete text of message 6322 as it appeared in the conversation:

[assistant] They're `DMA-FQ` (because I set them to `DMA-FQ` before the last `rmmod`, and the groups got re-created on rescan). So we're back to the working baseline without identity.

Now I understand the exact problem and the only viable approach:

1. The nvidia module needs to be loaded (in memory) when the GPUs are rescanned
2. The IOMMU groups need to be `identity` when nvidia probes
3. But we can't change groups while nvidia is bound, and we can't unbind/rebind nvidia on Blackwell

So the solution is: **set the groups while GPUs are removed from PCI (groups don't exist), then when rescan creates new groups, those groups use the kernel's default domain type**. We need to change the *default* domain type — but only for newly created groups.

Wait, actually, I recall from the earlier tests that when I had previously set groups to `identity` and then removed/rescanned GPUs, the new groups showed `identity` too. That suggests the kernel may remember the preferred type for IOMMU groups. Let me test this: load nvidia with GPUs working normally (DMA-FQ), then try the full cycle but set identity BEFORE removing the GPUs (while nvidia is loaded):

Actually, let me try the simplest possible approach that should work: What if the identity setting persists across remove/rescan? Let me verify by:

1. Having working GPUs (current state)
2. Set identity on groups (this will fail because nvidia is bound)
3. OK so we need to: unload nvidia → set identity → remove GPUs → SBR → load nvidia → rescan

The trick is doing step 2 (set identity) before removing the GPUs. Let me try:
[bash] ssh root@10.1.2.6 'set -e
GPUS="0000:01:00.0 0000:11:00.0 0000:61:00.0 0000:71:00.0"
BRIDGES="0000:00:01.1 0000:10:01.1 0000:60:01.1 0000:70:01.1"

echo "=== 1. Blacklist nvidia ==="
cat > /etc/modprobe.d/temp-nvidia-blacklist.conf << "EOF"
install nvidia /bin/false
install nvidia_modeset /bin/false
install nvidia_uvm /bin/false
install nvidia_drm /bin/false
EOF

echo "=== 2. Unbind GPUs from nvidia ==="
for gpu in $GPUS; do
    echo "$gpu" > /sys/bus/pci/drivers/nvidia/unbind 2>/dev/null && echo "  $gpu unbound" || echo "  $gpu: not bound"
done
sleep 1

echo "=== 3. Unload nvidia ==="
rmmod nvidia_drm; rmmod nvidia_uvm; rmmod nvidia_modeset; rmmod nvidia
lsmod | grep nvidia && echo "WARN" || echo "  clean"

echo ""
echo "=== 4. Set identity (GPUs unbound, nvidia unloaded) ==="
for gpu in $GPUS; do
    group=$(basename $(readlink /sys/bus/pci/devices/$gpu/iommu_group))
    echo "identity" > /sys/kernel/iommu_groups/$group/type
    echo "  $gpu: group=$group -> $(cat /sys/kernel/iommu_groups/$group/type)"
done

echo ""
echo "=== 5. Remove GPUs ==="
for gpu in $GPUS; do
    echo 1 > /sys/bus/pci/devices/$gpu/remove
done
sleep 2

echo "=== 6. SBR ==="
for bridge in $BRIDGES; do
    bridge_ctl=$(setpci -s $bridge BRIDGE_CONTROL)
    new_val=$(printf "%04x" $((0x$bridge_ctl | 0x40)))
    setpci -s $bridge BRIDGE_CONTROL=$new_val
    sleep 2
    setpci -s $bridge BRIDGE_CONTROL=$bridge_ctl
done
echo "  Waiting 15s..."
sleep 15

echo "=== 7. Remove blacklist & load nvidia BEFORE rescan ==="
rm -f /etc/modprobe.d/temp-nvidia-blacklist.conf
modprobe nvidia && echo "  nvidia loaded"
modprobe nvidia_modeset; modprobe nvidia_uvm; modprobe nvidia_drm
sleep 2

echo ""
echo "=== 8. Prevent vfio-pci grab ==="
echo "10de 2bb5" > /sys/bus/pci/drivers/vfio-pci/remove_id 2>/dev/null || true

echo ""
echo "=== 9. Rescan ==="
echo 1 > /sys/bus/pci/rescan
sleep 10

echo ""
echo "=== 10. FINAL CHECK ==="
nvidia-smi --query-gpu=index,gpu_bus_id,name,memory.total --format=csv,noheader 2>&1

echo ""
echo "IOMMU group types:"
for gpu in $GPUS; do
    group=$(basename $(readlink /sys/bus/pci/devices/$gpu/iommu_group))
    driver=$(basename $(readlink /sys/bus/pci/devices/$gpu/driver) 2>/dev/null || echo "none")
    type=$(cat /sys/kernel/iommu_groups/$group/type)
    echo "  $gpu: driver=$driver, group=$group, type=$type"
done

echo ""
echo "VFIO GPUs:"
for gpu in 0000:81:00.0 0000:91:00.0 0000:e1:00.0 0000:f1:00.0; do
    group=$(basename $(readlink /sys/bus/pci/devices/$gpu/iommu_group))
    driver=$(basename $(readlink /sys/bus/pci/devices/$gpu/driver) 2>/dev/null || echo "none")
    type=$(cat /sys/kernel/iommu_groups/$group/type)
    echo "  $gpu: driver=$driver, group=$group, type=$type"
done'
=== 1. Blacklist nvidia ===
=== 2. Unbind GPUs from nvidia ===
  0000:01:00.0 unbound
  0000:11:00.0 unbound
  0000:61:00.0 unbound
  0000:71:00.0 unbound
=== 3. Unload nvidia ===
  clean

=== 4. Set identity (GPUs unbound, nvidia unloaded) ===
  0000:01:00.0: group=28 -> identity
  0000:11:00.0: group=61 -> identity
  0000:61:00.0: group=42 -> identity
  0000:71:00.0: group=10 -> identity

=== 5. Remove GPUs ===
=== 6. SBR ===
  Waiting 15s...
=== 7. Remove blacklist & load nvidia BEFORE rescan...

This message captures the assistant's reasoning process in real time—the moment of insight, the formulation of the hypothesis, the design of the experiment, and the beginning of its execution. The output visible at the bottom shows the experiment proceeding through its early steps, with the critical rescan and final check still pending.