The Modprobe Install Hook: A Last-Ditch Gambit for Blackwell P2P DMA

Message Overview

The message is deceptively simple — a single bash command that copies a shell script to a remote server and marks it executable:

scp /home/theuser/glm-kimi-sm120-rtx6000bw/gpu-set-identity-before-nvidia.sh root@10.1.2.6:/usr/local/bin/gpu-set-identity-before-nvidia.sh && ssh root@10.1.2.6 'chmod +x /usr/local/bin/gpu-set-identity-before-nvidia.sh && echo "Script deployed"'

The output confirms success: Script deployed. But this mundane file transfer represents the culmination of an extensive debugging session spanning dozens of messages, multiple failed approaches, and a deepening understanding of the intricate dance between the Linux kernel's IOMMU subsystem, the NVIDIA driver's PCI probe sequence, and the Blackwell GPU architecture's firmware initialization requirements.

The Problem: P2P DMA Under IOMMU

The broader context is a high-performance ML inference deployment on a Proxmox host with 8 NVIDIA RTX PRO 6000 Blackwell GPUs, split between an LXC container (4 GPUs on NUMA0) and a VM (4 GPUs on NUMA1, passed through via VFIO with SEV-SNP). The LXC container runs SGLang serving a Qwen3.5-122B-A10B BF16 model with tensor parallelism across 4 GPUs. For maximum throughput, peer-to-peer (P2P) DMA between GPUs is essential — it allows direct GPU-to-GPU transfers without staging through system memory.

However, the host has IOMMU enabled (required for the SEV-SNP VM), and the NUMA0 GPUs are assigned to IOMMU groups with type DMA-FQ (DMA with Fine-Grained translation). Under DMA-FQ, the IOMMU translates all DMA addresses, and the NVIDIA driver's P2P DMA mechanism fails — either hanging the system or producing IO_PAGE_FAULT errors in the kernel log. The solution, in theory, is to change the IOMMU group type from DMA-FQ to identity, which bypasses translation for that group's devices and allows direct peer addressing.

The Timing Problem: No Window Between Enumeration and Binding

The assistant had spent considerable effort trying to find a sequence of operations that would allow setting identity domains before the NVIDIA driver claimed the GPUs. The fundamental constraint emerged through trial and error:

  1. The NVIDIA module auto-loads via modalias. When PCI devices appear on the bus (e.g., after a rescan), the kernel's modalias mechanism triggers modprobe nvidia automatically via udev. There is no user-controlled window between device enumeration and driver binding.
  2. modprobe nvidia fails without matching devices. The assistant tried loading the NVIDIA module before the PCI rescan, hoping the driver would be in memory and ready to probe freshly enumerated devices. But modprobe nvidia returned "No such device" — the module refuses to load when no matching PCI devices exist on the bus.
  3. SBR doesn't clear Blackwell's GSP firmware. Secondary Bus Reset (SBR) on the PCIe bridge was attempted to force a clean GPU state, but the Blackwell GPU's GSP (GPU Security Processor) firmware survives SBR. This was confirmed by dmesg errors showing RmInitAdapter: Cannot initialize GSP firmware RM and the characteristic 0xbadf4100 bad register read pattern.
  4. Identity domains must be set before the driver binds. Once the NVIDIA driver claims a GPU and initializes its DMA mappings, changing the IOMMU group type is ineffective — the driver has already set up its translation tables. The identity domain change must happen before nvidia calls pci_probe on the device. The only approach that worked — loading NVIDIA after a PCI rescan — always resulted in DMA-FQ because the modalias trigger loaded NVIDIA before any user-space script could intervene.

The Modprobe Install Hook Solution

In message [msg 6393], the assistant recognized the only remaining avenue: the install directive in modprobe.d. This mechanism allows replacing the default module loading behavior with a custom command. Instead of modprobe nvidia directly loading the kernel module, it runs a script that:

  1. Iterates over the NUMA0 GPU PCI addresses
  2. For each GPU that exists on the bus and is unbound, reads its IOMMU group
  3. If the group type is not already identity, writes identity to the group's type file
  4. Then loads the real NVIDIA module via exec /sbin/modprobe --ignore-install nvidia "$@" The configuration file created in [msg 6393] was:
install nvidia /usr/local/bin/gpu-set-identity-before-nvidia.sh

This is a standard Linux mechanism, but using it to hijack GPU driver loading for IOMMU configuration is an unusual and creative application.

Crafting the Script

In [msg 6394], the assistant attempted to create the script directly on the remote host via a heredoc piped through SSH. The script contained:

#!/bin/bash
NUMA0_GPUS="0000:01:00.0 0000:11:00.0 0000:61:00.0 0000:71:00.0"
for gpu in $NUMA0_GPUS; do
    if [ ! -e /sys/bus/pci/devices/$gpu ]; then continue; fi
    group=$(basename $(readlink /sys/bus/pci/devices/$gpu/iommu_group) 2>/dev/null)
    if [ -z "$group" ]; then continue; fi
    current=$(cat /sys/kernel/iommu_groups/$group/type 2>/dev/null)
    if [ "$current" != "identity" ]; then
        echo "identity" > /sys/kernel/iommu_groups/$group/type 2>/dev/null
        logger -t gpu-iommu "Set group $group to identity for $gpu (was $current)"
    fi
done
exec /sbin/modprobe --ignore-install nvidia "$@"

However, the heredoc failed with a zsh parse error: zsh:13: parse error near ')'. The issue was the "$@" syntax — the double quotes and parentheses were being interpreted by the local shell (zsh) before being sent through SSH. This is a classic quoting nightmare when nesting shell scripts inside SSH commands.

The assistant pivoted in [msg 6395], using the write tool to create the file locally on the development machine, avoiding the heredoc complexity entirely. A typo was fixed in [msg 6396] via the edit tool. Then, in the subject message [msg 6397], the script was deployed to the remote server via scp and made executable.

Assumptions Embedded in the Approach

This solution rests on several critical assumptions, some of which would prove incorrect:

Assumption 1: The modprobe install hook runs before the NVIDIA driver's PCI probe callback. The hook replaces modprobe nvidia itself. When modalias triggers module loading, udev calls modprobe nvidia, which (due to the hook) runs the script. The script sets identity domains, then loads the real module. The real module's init function registers the PCI driver, and the kernel then calls the probe callback for each matching device. The assumption is that identity domains set before the module's init function runs will be in effect when probe is called. This is reasonable — the IOMMU group type is a property of the group, not the driver, and should be consumed by the driver during probe when it calls dma_set_mask and sets up its DMA mappings.

Assumption 2: The GPUs will be unbound when the script runs. This follows from the sequence: the script runs before the NVIDIA module is loaded, so no driver has claimed the GPUs yet. The VFIO driver is blocked by the remove_id command. The GPUs should be sitting on the PCI bus with driver = none.

Assumption 3: Identity domains can be set on an IOMMU group that has devices but no driver. The Linux IOMMU subsystem allows changing the group type as long as no device in the group has DMA enabled. Since no driver has bound to the devices, DMA should not be active yet.

Assumption 4: The Blackwell GSP firmware is compatible with identity IOMMU. This assumption would prove catastrophically wrong. As discovered later in the segment (see [chunk 41.0]), the Blackwell FSP boot sequence fails with error code 0x177 when IOMMU is in identity mode. The FSP apparently requires specific DMA mappings set up by the kernel's DMA API in translation mode. Identity mode breaks this initialization, causing rm_init_adapter to fail and the GPU to be unusable.

The Deeper Significance

Message [msg 6397] is a moment of apparent progress — a carefully crafted solution being deployed after hours of debugging. The assistant had traced the problem through multiple layers: PCI bus topology, IOMMU group assignment, driver probe ordering, modalias autoloading, and GSP firmware behavior. The modprobe install hook was the most elegant solution imaginable given the constraints: it inserts configuration at exactly the right point in the boot sequence, using a standard Linux mechanism that requires no kernel patches or custom init scripts.

The fact that this approach ultimately failed — not due to any flaw in the hook mechanism itself, but because of a deeper incompatibility between Blackwell's firmware and identity IOMMU — illustrates the treacherous nature of low-level GPU systems programming. The Blackwell FSP is a proprietary firmware component whose initialization requirements are undocumented. The assistant could not have known that identity IOMMU would break it without trying.

The failure also highlights a broader lesson: when dealing with modern GPUs that have their own firmware, security processors, and complex initialization sequences, the PCIe device model is an abstraction that hides enormous complexity. The GPU is not a simple DMA engine that can be transparently remapped by the IOMMU — it has its own internal DMA setup that the firmware performs during boot, and this setup may depend on specific IOMMU translation behavior.

What the Message Achieves

At the surface level, message [msg 6397] deploys a shell script to a remote server. But in the context of the conversation, it represents:

  1. The synthesis of a deep debugging effort. The script encodes everything learned about IOMMU groups, modprobe mechanics, and NVIDIA's probe sequence.
  2. A testable hypothesis. The assistant has a clear model of how the system should behave: the hook runs, sets identity, NVIDIA loads, P2P works. This hypothesis will be tested in the very next messages.
  3. A boundary of what is possible with standard Linux mechanisms. If this fails, the only remaining options are non-standard: kernel patches, custom initramfs, or accepting that P2P DMA is impossible under IOMMU with Blackwell GPUs. The message is a quiet pivot point. It deploys the best solution the assistant could devise, unaware that a deeper, unsolvable problem awaits.