The Verification Bridge: Restoring Order After a Blackwell IOMMU Experiment

Introduction

In the course of a deep infrastructure debugging session spanning multiple hours, message 6404 stands as a quiet but critical moment of closure. After a series of increasingly creative attempts to enable P2P DMA between Blackwell-generation NVIDIA RTX PRO 6000 GPUs via IOMMU identity domains — attempts that ultimately failed due to a fundamental incompatibility with the GPU's Firmware Security Processor (FSP) — the assistant takes a deliberate step back to verify the system's integrity before returning to production operation. The message reads:

GPUs restored and working. Now let me verify the VFIO GPUs are still fine and start SGLang:

>

``bash ssh root@10.1.2.6 'for gpu in 0000:81:00.0 0000:91:00.0 0000:e1:00.0 0000:f1:00.0; do driver=$(basename $(readlink /sys/bus/pci/devices/$gpu/driver) 2>/dev/null || echo "none"); echo "$gpu: $driver"; done' ``

>

0000:81:00.0: vfio-pci 0000:91:00.0: vfio-pci 0000:e1:00.0: vfio-pci 0000:f1:00.0: vfio-pci

At first glance, this appears to be a simple verification step — a routine check that all eight GPUs in the system are accounted for. But beneath the surface, this message represents the culmination of a complex debugging journey, a deliberate pivot from a failed approach, and a methodical restoration of a multi-GPU production inference server to a known-good state.

The Context: A Deep Dive into Blackwell's IOMMU Quirks

To understand why this message was written, one must appreciate the journey that preceded it. The assistant had been working on a system with eight NVIDIA RTX PRO 6000 Blackwell GPUs, split into two groups: four GPUs on NUMA node 0 (PCIe addresses 01:00.0, 11:00.0, 61:00.0, 71:00.0) used directly by the SGLang inference server, and four GPUs on VFIO (81:00.0, 91:00.0, e1:00.0, f1:00.0) passed through to a virtual machine.

The core problem was P2P (peer-to-peer) DMA performance. Under the default IOMMU configuration with SEV-SNP (Secure Encrypted Virtualization - Secure Nested Paging) enabled, direct GPU-to-GPU memory transfers were corrupted. The symptom was clear: NCCL operations would hang or produce garbage data. The assistant had already deployed a workaround — NCCL_P2P_DISABLE=1 — but this came at a performance cost, forcing all cross-GPU communication through the CPU rather than direct PCIe peer-to-peer transfers.

The attempted solution was elegant in theory: set the IOMMU domain type to "identity" for the GPU groups, which would bypass IOMMU translation entirely and allow direct DMA mappings. This required setting each GPU's IOMMU group to identity mode before the nvidia driver claimed the device, because once the driver initialized the GPU's GSP firmware, the IOMMU configuration was locked in.

The Failed Experiment: Identity Domains and Blackwell's FSP

What followed was a meticulous but ultimately unsuccessful series of attempts. The assistant tried multiple strategies:

  1. Manual sequencing: Unbind the GPUs, remove them from the PCI bus, perform a Secondary Bus Reset (SBR) on the upstream bridges, set identity domains, then rescan. This failed because the nvidia driver's modalias-based autoloading during rescan left no window to set identity domains between device enumeration and driver binding.
  2. Modprobe install hook: A sophisticated approach using modprobe's install directive to run a script that sets identity domains before the real nvidia module loads. This worked in the sense that identity domains were correctly set, but the SBR operation failed to clear the Blackwell GSP firmware state, leaving the GPUs in a corrupted condition where rm_init_adapter failed with error code 0x177.
  3. Boot-time hook: The assistant realized the modprobe hook needed to be in place at the very first boot, before nvidia ever touched the GPUs. But this required a reboot to test, and the current session's GPUs were already corrupted. The critical discovery was that Blackwell's FSP requires specific DMA mappings set up by the kernel's DMA API in translation mode during initialization. Identity mode breaks this initialization, and no software-level reset — FLR (Function Level Reset), SBR (Secondary Bus Reset), or CXL bus reset — can clear the FSP state once it has been initialized. This is a hardware-level constraint, not a software bug.

Message 6404: The Verification Step

After this discovery, the assistant made a deliberate decision to revert. The modprobe hook was temporarily disabled, the SBR cycle was run one more time to recover the GPUs (this time without identity domains, allowing nvidia to load normally), and the hook was re-enabled for future boot testing. The result from the preceding message confirmed all four NUMA0 GPUs were back:

0, 00000000:01:00.0, NVIDIA RTX PRO 6000 Blackwell Server Edition, 97887 MiB
1, 00000000:11:00.0, NVIDIA RTX PRO 6000 Blackwell Server Edition, 97887 MiB
2, 00000000:61:00.0, NVIDIA RTX PRO 6000 Blackwell Server Edition, 97887 MiB
3, 00000000:71:00.0, NVIDIA RTX PRO 6000 Blackwell Server Edition, 97887 MiB

Message 6404 then performs the second half of the verification: checking the four VFIO GPUs. The assistant could have simply started SGLang after confirming the NUMA0 GPUs, but it chose to verify the VFIO GPUs as well. This is methodical operations at work — a full-system check before declaring the environment ready.

The command is straightforward: iterate over the four VFIO GPU PCI addresses, read each device's driver symlink, and print the result. The output confirms all four are bound to vfio-pci, exactly as expected. The SBR and rescan operations on the NUMA0 GPUs did not affect the VFIO group, which is on separate PCIe buses and was never touched during the recovery cycle.

Assumptions and Knowledge Required

This message makes several implicit assumptions that are critical to understanding its significance:

Assumption 1: The VFIO GPUs were unaffected by the NUMA0 recovery operations. The assistant assumes that removing, SBR-resetting, and rescanning PCI devices on NUMA0 (buses 01, 11, 61, 71) does not affect devices on other buses (81, 91, e1, f1). This is a reasonable assumption given PCIe topology — each GPU sits behind its own PCIe bridge — but it is not guaranteed. A misconfigured SBR could theoretically propagate across bridges if the hierarchy is shared. The verification confirms this assumption held.

Assumption 2: The modprobe hook file was correctly restored. In the recovery process, the assistant temporarily moved the modprobe hook to a .bak file to prevent it from interfering, then moved it back. The assistant implicitly trusts that this file operation succeeded and that the hook will be available for the next boot test. This is a reasonable assumption given that no errors were reported.

Assumption 3: The system is now in a stable state for SGLang. The assistant states "GPUs restored and working" and moves to start SGLang. This assumes that the SBR cycle fully reset the GPUs to a clean state, that the nvidia driver initialized them correctly, and that no residual corruption remains from the failed identity-mode attempts. The nvidia-smi output from the previous message supports this assumption.

Input knowledge required to fully understand this message includes:

The Thinking Process

The assistant's reasoning in this message reveals a disciplined operational mindset. Having just completed a complex and risky hardware manipulation — removing PCI devices from the bus, performing secondary bus resets, and rescaming — the assistant does not assume success based on partial verification. Instead, it performs a complete audit:

  1. First, verify the primary GPUs (NUMA0) are back on the nvidia driver with correct memory. This was done in the previous message via nvidia-smi.
  2. Second, verify the secondary GPUs (VFIO) are untouched. This is done in this message via direct sysfs inspection.
  3. Third, only then proceed to start the application (SGLang). This two-phase verification is characteristic of operations on expensive, high-value hardware. A mistake in the SBR procedure could have accidentally reset VFIO GPUs or caused them to be claimed by the nvidia driver instead of vfio-pci. By checking explicitly, the assistant catches any such issues before they cause problems in production. The choice of verification method is also telling. Rather than using nvidia-smi (which only sees nvidia-bound GPUs), the assistant reads the driver symlink directly from sysfs. This works for any driver — vfio-pci, nvidia, or none — and provides unambiguous results. It's a low-level, reliable check that doesn't depend on any particular driver being loaded.

Broader Significance

This message, while brief, marks an important inflection point in the conversation. It is the moment where the assistant transitions from exploration and debugging to production deployment. The IOMMU identity experiment is concluded — the approach is definitively blocked by Blackwell's hardware architecture. The assistant accepts this constraint and moves forward with the known-working configuration: NCCL_P2P_DISABLE=1 for P2P safety, MTP (Multi-Token Prediction) speculation for throughput, and the stable SGLang deployment.

The message also demonstrates a crucial operational principle: when a complex hardware manipulation fails, the first priority is not to try another approach, but to restore the system to a known-good state. Only from a clean baseline can the next experiment begin. The assistant could have continued trying variations of the identity-domain approach, but instead chose to revert, verify, and proceed with what works.

Conclusion

Message 6404 is a verification bridge — a deliberate, methodical check that the system is intact after a failed experiment. It confirms that all eight Blackwell GPUs are on their correct drivers, that the SBR recovery was successful, and that the environment is ready for production inference serving. While it lacks the drama of the preceding debugging battles, it is perhaps the most important message in the sequence: the one that says "we are safe, we can move forward." In the world of high-stakes infrastructure engineering, that is a message worth writing.