The Boot Order Trap: Fixing Systemd Dependencies for Blackwell GPU IOMMU Identity Domains

In the high-stakes world of Blackwell GPU deployment, where every boot sequence is a delicate dance of firmware initialization and driver binding, a single misplaced systemd directive can unravel hours of careful configuration. Message [msg 6374] captures one such moment — a seemingly mundane systemd unit file update that encapsulates a hard-learned lesson about boot ordering, PCI driver binding, and the unforgiving nature of NVIDIA's Blackwell firmware security processor.

The Context: A Reboot Reveals the Flaw

The story begins with an ambitious goal: enabling peer-to-peer (P2P) DMA between NVIDIA RTX PRO 6000 Blackwell GPUs on a Proxmox host running an AMD IOMMU in full translation mode. The host has 8 GPUs split across two NUMA domains — 4 on NUMA0 designated for the SGLang inference server, and 4 on NUMA1 bound to vfio-pci for an SEV-SNP confidential VM. The challenge is that IOMMU translation mode (DMA-FQ) blocks P2P DMA, so the NUMA0 GPUs need their IOMMU groups switched to identity mode, which bypasses translation and allows direct GPU-to-GPU transfers.

In the previous session ([msg 6368]), the assistant had prepared for this by creating two boot-time mechanisms:

  1. gpu-iommu-identity.service — a oneshot service that unbinds the NUMA0 GPUs from the nvidia driver, sets their IOMMU groups to identity, and rebinds them
  2. gpu-vfio-split.service — a service that registers the NVIDIA PCI ID (10de 2bb5) with vfio-pci, causing the NUMA1 GPUs to be bound to vfio-pci for the VM The critical design decision was ordering: the identity service was configured with Before=nvidia-persistenced.service and, crucially, no explicit dependency on gpu-vfio-split.service. The implicit assumption was that the identity service should run first, before the nvidia driver fully initializes, to set up the IOMMU domains before the GPUs are claimed. When the user rebooted and asked to check P2P status ([msg 6369]), the assistant eagerly checked the IOMMU group types ([msg 6371]). The results were simultaneously encouraging and devastating:
0000:01:00.0: driver=vfio-pci, group=42, type=identity
0000:11:00.0: driver=vfio-pci, group=61, type=identity
0000:61:00.0: driver=vfio-pci, group=28, type=identity
0000:71:00.0: driver=vfio-pci, group=10, type=identity

The identity domains were set correctly — but every single NUMA0 GPU was bound to vfio-pci instead of the nvidia driver. The boot sequence had gone wrong in a subtle but catastrophic way.

The Diagnosis: A Race Condition in PCI Driver Binding

The assistant's diagnostic reasoning in [msg 6372] reveals the precise mechanism of failure:

The gpu-vfio-split.service ran after gpu-iommu-identity.service and bound ALL GPUs with matching PCI ID to vfio-pci (because new_id matches all 10de:2bb5 devices).

The boot sequence played out as follows:

  1. gpu-iommu-identity.service runs first. It unbinds the NUMA0 GPUs from the nvidia driver (which had auto-loaded during kernel init), sets their IOMMU groups to identity, but does not rebind them to nvidia.
  2. gpu-vfio-split.service runs next. It executes echo "10de 2bb5" > /sys/bus/pci/drivers/vfio-pci/new_id, which registers the PCI vendor:device ID with the vfio-pci driver. This causes the kernel's driver core to attempt binding all unbound devices matching that ID — including the NUMA0 GPUs that the identity service had just unbound. The NUMA0 GPUs, sitting in an unbound state with their IOMMU groups set to identity, were immediately claimed by vfio-pci. The identity service had inadvertently created the perfect condition for vfio-pci to steal the GPUs: it unbinds them but doesn't rebind them to nvidia before vfio-split runs. This is a classic systemd dependency ordering problem. The identity service needed to run after vfio-split, not before, so that vfio-pci would only claim the NUMA1 GPUs (which were never unbound from vfio-pci in the first place). The NUMA0 GPUs, still bound to nvidia at that point, would be unaffected by the new_id registration. Then the identity service could unbind them, set identity domains, and rebind them to nvidia without competition.## The Fix: Reversing the Dependency Order Message [msg 6374] is the assistant's response to this diagnosis. The reasoning is clear and direct: "Now update the service ordering to run AFTER vfio-split." The fix involves rewriting the systemd unit file with the correct dependency declarations:
[Unit]
Description=Set IOMMU identity domains for NUMA0 nvidia GPUs (enables P2P DMA)
# Must run AFTER vfio-split (which registers new_id and grabs NUMA1 GPUs)
After=gpu-vfio-split.service
After=systemd-modules-load.service
Before=nvidia-persistenced.service
DefaultDependencies=no

The key change is After=gpu-vfio-split.service. This single line reverses the entire boot ordering. The new sequence becomes:

  1. systemd-modules-load.service — loads kernel modules including nvidia and vfio-pci
  2. gpu-vfio-split.service — registers 10de 2bb5 with vfio-pci, which claims the NUMA1 GPUs (still bound to nothing) but leaves the NUMA0 GPUs (already bound to nvidia) untouched
  3. gpu-iommu-identity.service — unbinds NUMA0 GPUs from nvidia, sets identity domains, rebinds to nvidia
  4. nvidia-persistenced.service — initializes the nvidia persistence layer on the now-correctly-configured GPUs The DefaultDependencies=no directive is also notable. This tells systemd not to automatically add standard dependencies like sysinit.target and basic.target. It's used here because the service needs to run very early in the boot process, before the normal multi-user target dependencies kick in. The service explicitly declares WantedBy=multi-user.target in the [Install] section, which ensures it's activated during normal boot but with minimal implicit ordering constraints.

The Script Update: Handling the vfio-pci Steal

But the unit file change alone wasn't sufficient. The identity script itself needed to be hardened against the scenario where vfio-pci had already claimed the NUMA0 GPUs. In [msg 6373], the assistant updated the script to handle this:

The new script adds explicit unbinding from vfio-pci (not just nvidia), a drivers_probe mechanism to trigger nvidia binding after identity is set, and idempotency checks. The critical addition is the unbind-from-any-driver logic:

driver=$(basename $(readlink /sys/bus/pci/devices/$gpu/driver) 2>/dev/null || echo "none")
if [ "$driver" != "none" ]; then
    echo "$gpu" > /sys/bus/pci/devices/$gpu/driver/unbind 2>/dev/null
    echo "gpu-iommu-identity: $gpu unbound from $driver"
    sleep 0.5
fi

This handles both cases: if the GPU is on nvidia (the intended state before vfio-split) or on vfio-pci (if the ordering is wrong or the script runs manually). The sleep 0.5 between unbind and the next operation gives the kernel time to settle.

The Deeper Problem: Blackwell GSP/FSP Survivability

The boot ordering fix in [msg 6374] was necessary but ultimately insufficient for the larger P2P DMA goal. As the subsequent messages reveal ([msg 6377], [msg 6378], [msg 6382]), even with correct ordering, the Blackwell GPU's Firmware Security Processor (FSP) presents a fundamental obstacle.

When the identity service unbinds a GPU from the nvidia driver, the FSP session established during the driver's initial probe enters a locked state. The nvidia driver's kfspSendBootCommands_HAL function fails with error 0x62:0xffff:2142, and the GPU registers read as 0xbadf4100 — a hardware poison value indicating the GSP firmware is in an unrecoverable state. No software-level reset (FLR, SBR, CXL bus reset, module unload) can clear this state.

This means the entire approach of runtime IOMMU domain switching is fundamentally incompatible with Blackwell GPUs. The only viable path would be setting identity domains before the nvidia driver first touches the GPUs — which requires blacklisting nvidia, setting identity domains, then manually loading nvidia. But even this approach ultimately failed, as the chunk summary reveals: "Blackwell's FSP requires DMA translation mode during initialization, and no software-level reset can clear this state."

Assumptions and Knowledge Required

To understand [msg 6374], several pieces of input knowledge are essential:

  1. Systemd dependency ordering: Understanding After=, Before=, DefaultDependencies=no, and WantedBy= directives and how they control boot sequencing
  2. PCI driver binding model: How the kernel's driver core matches devices to drivers via vendor:device IDs, and how new_id registration triggers automatic binding of all unbound matching devices
  3. IOMMU domain types: The difference between DMA-FQ (translation mode, required for SEV-SNP VMs) and identity (passthrough mode, required for P2P DMA)
  4. The GPU topology: 4 NUMA0 GPUs for inference, 4 NUMA1 GPUs for VM, all sharing the same PCI vendor:device ID (10de 2bb5)
  5. Blackwell GSP/FSP behavior: The firmware security processor that initializes during the nvidia driver's first probe and cannot be reset without a full power cycle The output knowledge created by this message is the corrected systemd unit file with proper dependency ordering. This knowledge propagates forward: when the system reboots, the identity service will run after vfio-split, preventing the NUMA0 GPUs from being stolen. However, the deeper knowledge about Blackwell FSP incompatibility with identity IOMMU domains would only be discovered in the subsequent messages, ultimately leading to the abandonment of the P2P DMA approach.

Conclusion

Message [msg 6374] is a masterclass in systemd dependency debugging. It demonstrates how a single missing After= directive can cause a cascade of failures across multiple boot services, and how the fix requires understanding not just the intended behavior of each service but their interactions through the kernel's PCI driver binding mechanism. The assistant's reasoning — diagnose the race condition, identify the ordering inversion, update the unit file, and harden the script against the failure mode — follows the classic pattern of systems engineering: understand the failure, fix the ordering, and make the components robust against misordering.

Yet the message also serves as a poignant reminder that some problems cannot be solved by software configuration alone. The Blackwell FSP's locked-state behavior, discovered in the aftermath of this fix, would ultimately render the entire P2P DMA effort moot. The boot ordering fix was correct, but it was fighting a losing battle against hardware that refuses to be reconfigured after initialization. Sometimes the most valuable outcome of a debugging session is not the fix itself, but the discovery of an immovable constraint that redefines the problem space.