Chunk 41.0

After the reboot, the assistant verified that the modprobe install hook successfully set IOMMU identity domains for the NUMA0 GPUs before the nvidia driver loaded. However, a critical discovery emerged: the nvidia Blackwell FSP (Firmware Security Processor) boot sequence fails with error code `0x177` when IOMMU is in identity mode. The FSP apparently requires specific DMA mappings set up by the kernel's DMA API in translation mode, and identity mode breaks this initialization. This means **per-group IOMMU identity domains are fundamentally incompatible with Blackwell GPUs** — the approach cannot work regardless of timing. The assistant immediately reverted by removing the modprobe hook and rebooting, restoring the working `DMA-FQ` configuration. The MTP/NEXTN speculation enabled in the previous session survived the reboot and continues to provide significant throughput improvements (12-45% per-request boost at low concurrency). The assistant confirmed SGLang is running stably with MTP enabled, P2P disabled (`NCCL_P2P_DISABLE=1`), and all 4 NUMA0 GPUs working correctly on the nvidia driver with `DMA-FQ` IOMMU type. The 4 VFIO GPUs remain on vfio-pci for the SEV-SNP VM. **Key outcomes**: P2P DMA restoration via IOMMU identity domains is definitively blocked — Blackwell's FSP requires DMA translation mode during initialization, and no software-level reset (FLR, SBR, CXL bus reset) can clear this state. The only remaining option for P2P would be if the nvidia driver's `DmaRemapPeerMmio=1` parameter can be made to work correctly (it's already enabled but produces incomplete IOMMU mappings, with some peer pairs working and others faulting). MTP speculation is the primary active optimization, delivering 12-45% per-request throughput improvement. The system is stable with SGLang serving Qwen3.5-122B-A10B BF16 on 4 GPUs with TP=4.

The Blackwell FSP Barrier: How IOMMU Identity Domains Met Their Match in NVIDIA's Firmware Security Processor 2712 words

Message Articles

Subagent Sessions