The Diagnostic That Killed a Hypothesis: Blackwell's GSP Refuses Identity IOMMU

In the high-stakes world of production ML infrastructure, few things are as frustrating as a hardware limitation that no amount of software cleverness can circumvent. Message 6400 of this opencode session captures the precise moment when a carefully engineered approach to restoring GPU peer-to-peer DMA on a Blackwell-based system met its definitive end — not with a bang, but with a register read of 0xbadf4100 and a terse rm_init_adapter failed from the nvidia kernel driver.

The Context: A Multi-Session Quest for P2P DMA

To understand the weight of this message, we must first understand the journey that led to it. The system in question is a Proxmox host with 8 NVIDIA RTX PRO 6000 Blackwell Server Edition GPUs, split between an LXC container (4 GPUs on NUMA0, bound to the nvidia driver) and a SEV-SNP VM (4 GPUs on vfio-pci). The container runs SGLang serving a Qwen3.5-122B-A10B BF16 model with tensor parallelism across 4 GPUs.

The problem: GPU peer-to-peer (P2P) DMA transfers are broken. Under the hood, the system has IOMMU enabled for SEV-SNP (Secure Encrypted Virtualization — Secure Nested Paging), which means all DMA transfers go through the IOMMU translation layer. The nvidia driver's DmaRemapPeerMmio=1 parameter, which is supposed to set up the necessary IOMMU mappings for P2P, produces incomplete mappings — some peer pairs work while others trigger IO_PAGE_FAULT errors. The result is that NCCL (NVIDIA Collective Communications Library) hangs or crashes when attempting P2P transfers, forcing the deployment to use NCCL_P2P_DISABLE=1 as a workaround, which routes all inter-GPU communication through the CPU rather than directly between GPUs.

The assistant had been pursuing a promising alternative: setting individual IOMMU groups to "identity" mode. In identity mode, the IOMMU bypasses address translation for devices in that group, effectively allowing direct DMA between devices. If the four NUMA0 GPUs could be placed in identity-mode IOMMU groups, P2P DMA would work without needing the nvidia driver's remapping at all.

The Modprobe Install Hook: A Clever Workaround

The challenge was timing. The assistant had discovered through painful experimentation that the nvidia driver can only be loaded when matching PCI devices exist on the bus, and it takes exclusive control of those devices immediately during the probe callback. There is no window between PCI device enumeration and nvidia driver binding to manually set IOMMU identity domains.

The solution was a modprobe install hook — a shell script registered via /etc/modprobe.d/nvidia-iommu-identity.conf that intercepts the modprobe nvidia call. The script, /usr/local/bin/gpu-set-identity-before-nvidia.sh, iterates over the four NUMA0 GPU PCI addresses, sets their IOMMU groups to identity mode, and then loads the real nvidia module via exec /sbin/modprobe --ignore-install nvidia. The idea was that this hook would run before nvidia's initialization code touched the GPU hardware, ensuring the IOMMU was already in identity mode when the GSP (GPU System Processor / Firmware Security Processor) booted.

Message 6399 put this to the test. The assistant executed a full PCI removal, Secondary Bus Reset (SBR), and rescan cycle on the remote host, expecting the modprobe install hook to set identity domains before nvidia loaded. The result was devastating: "No devices were found."

Message 6400: The Diagnostic

ssh root@10.1.2.6 'lsmod | grep nvidia; echo "---"; dmesg | grep "NVRM" | tail -5; echo "---"; journalctl -t gpu-iommu --since "2 minutes ago" --no-pager 2>/dev/null; echo "---"; for gpu in 0000:01:00.0 0000:11:00.0 0000:61:00.0 0000:71:00.0; do driver=$(basename $(readlink /sys/bus/pci/devices/$gpu/driver) 2>/dev/null || echo "none"); group=$(basename $(readlink /sys/bus/pci/devices/$gpu/iommu_group)); type=$(cat /sys/kernel/iommu_groups/$group/type); echo "$gpu: driver=$driver, group=$group, type=$type"; done'

This is a masterclass in systematic diagnostics. The assistant queries four distinct sources of information in a single command, each answering a different question:

  1. lsmod | grep nvidia — Are the nvidia kernel modules loaded at all? This tells us whether the modprobe hook successfully loaded the real nvidia module, or whether the entire loading process failed.
  2. dmesg | grep "NVRM" | tail -5 — What did the nvidia kernel driver log during its initialization? The NVRM (NVIDIA Resource Manager) messages are the primary source of truth for driver initialization failures.
  3. journalctl -t gpu-iommu — Did the install hook script actually run? The script logs its actions via logger -t gpu-iommu, so this tells us whether the hook was triggered at all and what IOMMU groups it modified.
  4. GPU driver/IOMMU status — For each of the four NUMA0 GPUs, what is the current driver binding, IOMMU group number, and IOMMU type? This gives the ground truth about what state the system is actually in.

What the Output Revealed

The output tells a nuanced story of partial success and fundamental failure:

nvidia_uvm           2031616  0
nvidia_drm            131072  0
nvidia_modeset       1679360  1 nvidia_drm
nvidia              14745600  2 nvidia_uvm,nvidia_modeset

The nvidia modules are loaded. The modprobe hook did its job — it intercepted the modprobe call, set identity domains, and loaded the real nvidia module. The module reference counts are low (2 for the main nvidia module, 0 for the submodules), indicating that no user-space processes are actively using the GPUs.

But then the dmesg output tells the real story:

[ 2977.551513] NVRM: GPU 0000:11:00.0: rm_init_adapter failed, device minor number 3
[ 2977.551770] NVRM: GPU0 gpuHandleSanityCheckRegReadError_GH100: Possible bad register read: addr: 0x110094,  regvalue: 0xbadf4100,  error code: Unknown SYS_PRI_ERROR_CODE

The nvidia driver loaded, but it failed to initialize the GPUs. The rm_init_adapter failed message indicates that the Resource Manager (RM) — the core firmware component that manages GPU hardware — could not initialize. The 0xbadf4100 register read is particularly telling: 0xbadf is a deliberately chosen "BAD F" (bad firmware?) pattern, a canary value that the GSP firmware writes to indicate it has crashed or failed to boot. The gpuHandleSanityCheckRegReadError_GH100 function name references GH100 (the Hopper architecture), suggesting this sanity check code path is shared across GPU architectures.

The journalctl output (not shown in the truncated result, but the command ran) would have shown the hook's log messages confirming that identity domains were set. The GPU driver status (also truncated) would have shown the GPUs still bound to nvidia but in a failed state.

The Critical Insight: Blackwell's GSP Requires DMA Translation

This failure is not a timing issue, a race condition, or a configuration mistake. It is a fundamental hardware compatibility problem. The Blackwell GSP firmware, during its boot sequence, requires specific DMA mappings that are set up by the kernel's DMA API when the IOMMU is in translation mode. When the IOMMU group is set to identity mode, these mappings are never created, and the GSP cannot complete its initialization sequence.

The error code 0x177 (referenced in the chunk summary as appearing in subsequent investigation) corresponds to a GSP firmware initialization failure that occurs specifically because the DMA address space is not configured in the way the firmware expects. The GSP apparently performs its own DMA operations during boot — loading firmware blobs, accessing video memory, or communicating with other system components — and these operations fail when the IOMMU is in pass-through mode.

This is a stark reminder that "identity mode" is not truly transparent. While it bypasses address translation for DMA, it also bypasses the DMA mapping infrastructure that the kernel provides. Some devices — particularly those with complex firmware that performs its own DMA setup — depend on this infrastructure being present, even if the actual translation is unnecessary.

Assumptions That Proved Wrong

The assistant made several assumptions that this message definitively disproved:

Assumption 1: SBR clears GSP state. The Secondary Bus Reset (SBR) was assumed to reset the GPU to a clean state, as it does for previous NVIDIA architectures. The assistant had even demonstrated this working in an earlier session when the GPUs had never been initialized by nvidia in the current boot cycle. But the Blackwell GSP appears to latch state that survives SBR — once the GSP has been initialized by the nvidia driver, no software-level reset (SBR, Function Level Reset, or even CXL bus reset) can clear it. The only way to get a clean GSP is a full power cycle (ACPI G3 → G0 transition).

Assumption 2: Identity mode is transparent to devices. The assumption that setting an IOMMU group to identity mode is invisible to the device — that the device simply gets direct memory access without translation — turned out to be false for Blackwell. The GSP firmware actively depends on the DMA API infrastructure that translation mode provides.

Assumption 3: The modprobe hook timing is sufficient. Even if the hook runs before nvidia's initialization code, the IOMMU type must be set before the PCI device is even discovered by the kernel, because the DMA API setup happens during device enumeration, not during driver probe. The modprobe hook runs too late in the boot process.

Input Knowledge Required

To fully understand this message, one needs:

Output Knowledge Created

This message produces several critical pieces of knowledge:

  1. Definitive proof that IOMMU identity mode is incompatible with Blackwell GPUs. The approach of setting per-group identity domains to enable P2P DMA is fundamentally impossible with Blackwell hardware. This closes an entire line of investigation.
  2. A diagnostic pattern for Blackwell GSP failures. The combination of rm_init_adapter failed, gpuHandleSanityCheckRegReadError_GH100, and register value 0xbadf4100 is a signature for GSP initialization failure under identity IOMMU.
  3. Confirmation that the modprobe install hook mechanism works correctly. The hook did execute and loaded the nvidia module — the failure is in the hardware initialization, not in the hook mechanism itself.
  4. A boundary condition for Blackwell's reset behavior. The fact that SBR doesn't clear the GSP state (even after the failed initialization) means the only recovery path is a full power cycle.

The Aftermath

The assistant immediately pivoted. Message 6401 recognized that the SBR didn't clear the GSP because the GPUs had been initialized by nvidia earlier in the boot cycle. The assistant then realized that even a fresh boot with the modprobe hook would fail — the Blackwell FSP simply cannot boot under identity IOMMU. By message 6403, the assistant had reverted the modprobe hook, restored the GPUs to working DMA-FQ configuration, and started SGLang back up.

The MTP (Multi-Token Prediction) speculation enabled in the previous session survived the reboot and continues to provide 12-45% per-request throughput improvement at low concurrency. P2P DMA remains disabled via NCCL_P2P_DISABLE=1, and the only remaining hope for P2P is if the nvidia driver's DmaRemapPeerMmio=1 parameter can be made to work correctly — a possibility the assistant has not yet exhausted.

The Broader Lesson

This message is a case study in the value of failing fast and failing definitively. The assistant invested significant effort in the modprobe hook approach — designing the script, testing the timing, verifying the mechanism. But when the diagnostic in message 6400 revealed the GSP initialization failure, the assistant recognized it as a fundamental hardware limitation rather than a solvable software problem, and pivoted within two messages.

The 0xbadf4100 register value is almost poetic in its explicitness. NVIDIA's engineers embedded a "BAD F" canary in the firmware to make precisely this kind of failure diagnosable. When you see 0xbadf, you know the firmware didn't just crash — it detected an unrecoverable condition and deliberately poisoned its state to prevent further damage. It's a message from the hardware saying "this configuration is not supported, and no amount of software trickery will make it work."

Sometimes the most valuable output of a diagnostic is not a solution, but the certainty that a particular path is a dead end.