The IOMMU Barrier: Diagnosing P2P DMA Corruption Under SEV-SNP

In the high-stakes world of multi-GPU inference serving, few problems are as insidious as silent data corruption. When every tensor copy between GPUs produces subtly wrong results, the symptoms can masquerade as anything from random numerical errors to complete system hangs. Message <msg id=6217> captures the precise moment an engineer identified such a corruption, traced it to its root cause, and formulated the fix — all within a single, dense diagnostic message.

Context: The Hunt for an SGLang Hang

The message sits at the climax of a prolonged debugging session. The assistant had been deploying the Qwen3.5-122B-A10B model (a 122-billion-parameter Mixture-of-Experts architecture in BF16 precision) across four NVIDIA RTX PRO 6000 Blackwell GPUs using SGLang with tensor parallelism (TP=4). The server consistently hung during init_torch_distributed, specifically during NCCL's distributed initialization. Earlier rounds had ruled out port conflicts, socket connectivity issues, and software version mismatches. The assistant had even updated SGLang to the latest commit, rebuilt from source, and verified that all four TP workers could establish TCP connections to each other.

The breakthrough came when the user provided a crucial piece of evidence: IO_PAGE_FAULT logs from the Proxmox host's kernel, showing that the AMD IOMMU (AMD-Vi) was logging page faults for every GPU attempting DMA transfers. These faults pointed directly at a hardware-level problem — the GPUs were trying to talk to each other directly over PCIe, and the IOMMU was blocking those transfers.

The Subject Message: A Diagnostic Breakthrough

Message <msg id=6217> opens with a bold, capitalized summary that crystallizes the finding:

Every P2P transfer produces data mismatches. The IOMMU in full translation mode is breaking GPU-to-GPU P2P DMA. The driver reports canAccessPeer=True but the actual transfers are corrupted (the IO_PAGE_FAULTs we see in dmesg).

This single sentence accomplishes several things simultaneously. It states the empirical observation (data mismatches on every transfer), identifies the mechanism (IOMMU full translation mode), explains the discrepancy between CUDA's reported capability and actual behavior, and ties the observation back to the earlier evidence (the IO_PAGE_FAULTs). The assistant then connects this to the original problem: "This is the root cause of the SGLang hang — NCCL tries P2P all-reduce and gets corrupted data, causing the distributed init to deadlock."

The reasoning here is a textbook example of multi-layer diagnosis. The assistant had already established (in <msg id=6215> and <msg id=6216>) that torch.cuda.can_device_access_peer(i, j) returned True for every GPU pair, yet actual data transfers produced MISMATCH on every single test. The IO_PAGE_FAULTs from the host kernel (provided by the user in <msg id=6191>) provided the hardware-level confirmation. By connecting these three data points — CUDA's optimistic P2P report, the corrupted transfer results, and the IOMMU page faults — the assistant arrived at a diagnosis that no single piece of evidence alone could support.

The Fix: NCCL_P2P_DISABLE

The message then pivots from diagnosis to action. The assistant formulates a testable hypothesis: if P2P DMA is the broken path, then forcing NCCL to bypass it should restore correct operation. The command is:

NCCL_P2P_DISABLE=1 NCCL_SHM_DISABLE=0 ~/ml-env/bin/python3 -c "..."

The choice of NCCL_P2P_DISABLE=1 is significant. This environment variable tells NCCL to avoid GPU-to-GPU direct memory access entirely, falling back to shared memory (SHM) or socket-based communication. The NCCL_SHM_DISABLE=0 ensures shared memory remains available, which is critical for performance — without SHM, NCCL would fall back to TCP sockets over loopback, which would be dramatically slower for the large tensors involved in model inference.

The test itself is a minimal NCCL all-reduce verification: two processes, each holding a tensor of ones scaled by their rank, perform an all-reduce and check that the result equals the expected sum. This is the simplest possible distributed test that exercises the same communication path SGLang uses during initialization.

Why the Inline Test Failed

The message also documents a failure mode that is itself instructive. The inline NCCL test using python3 -c with torch.multiprocessing.Process and spawn start method fails with:

AttributeError: Can't get attribute 'worker' on <module '__main__' (<class '_frozen_importlib.BuiltinImporter'>)

This is a well-known limitation of Python's multiprocessing with the spawn start method: when code is executed from the command line with -c, there is no importable module, so child processes cannot find the worker function to execute. The assistant recognized this immediately and pivoted to writing a script file (as seen in the subsequent <msg id=6218>), demonstrating an understanding of Python's multiprocessing internals that is essential for distributed debugging.

Assumptions and Their Validity

The message rests on several assumptions, all of which proved correct:

The IO_PAGE_FAULTs are caused by P2P DMA, not by other GPU operations. This assumption was validated by the fact that single-GPU operations worked perfectly (as shown in <msg id=6214>), and only cross-GPU transfers produced mismatches.
NCCL uses P2P DMA during init_process_group. This is a reasonable assumption given NCCL's design: it probes available communication paths during initialization and prefers P2P for performance. The hang during init_torch_distributed was consistent with NCCL attempting a P2P transfer, receiving corrupted data, and entering an inconsistent state.
Disabling P2P would not break NCCL entirely. The assistant assumed that NCCL could fall back to SHM transport gracefully. This was validated in the subsequent message (<msg id=6219>) where NCCL_P2P_DISABLE=1 produced correct all-reduce results on all four ranks.
The corruption is the cause of the hang, not a symptom. It's theoretically possible that the hang and the corruption are independent issues, but the assistant correctly judged that corrupted P2P data would cause NCCL's internal consistency checks to fail, leading to deadlock.

Input Knowledge Required

To fully understand this message, the reader needs knowledge spanning several domains:

IOMMU concepts: Understanding that an IOMMU (Input-Output Memory Management Unit) translates device DMA addresses to system physical addresses, and that "full translation mode" (as opposed to "passthrough" mode) requires all DMA to go through this translation. The kernel command line amd_iommu=on (as seen in <msg id=6209>) enables full translation, while iommu=pt would enable passthrough.
SEV-SNP requirements: AMD's Secure Encrypted Virtualization with Secure Nested Paging requires IOMMU in full translation mode to enforce memory encryption and integrity for VM pages. This is why the Proxmox host had amd_iommu=on — it was necessary for the SEV-SNP VM running on the other four GPUs.
GPU P2P DMA mechanics: NVIDIA GPUs can directly read and write each other's memory over PCIe (or NVLink) without involving the CPU. This requires the IOMMU to map the remote GPU's BAR (Base Address Register) space into the local GPU's address space. When IOMMU is in full translation mode, these mappings must be explicitly set up, and the IO_PAGE_FAULTs indicate they were not.
NCCL internals: NVIDIA's Collective Communications Library probes available transport paths (P2P, NVLink, SHM, sockets) and selects the fastest available. The NCCL_P2P_DISABLE environment variable forces it to skip the P2P path.
Python multiprocessing: The spawn start method creates child processes from scratch, requiring functions to be importable from a module. The -c mode creates a __main__ module that is not importable, causing the AttributeError.

Output Knowledge Created

This message creates several valuable outputs:

Root cause identification: The definitive diagnosis that IOMMU full translation mode breaks GPU P2P DMA on this system, causing data corruption and NCCL hangs.
Actionable fix: The specific environment variable (NCCL_P2P_DISABLE=1) that bypasses the broken path, along with a test command to validate the fix.
Diagnostic methodology: A reusable pattern for diagnosing IOMMU-related GPU communication issues: check kernel cmdline for IOMMU mode, look for IO_PAGE_FAULTs in dmesg, test P2P with simple CUDA copies, verify with NCCL all-reduce, and apply the environment variable fix.
Documentation of a failure mode: The Python multiprocessing spawn limitation with -c is documented as a dead end, saving future debugging time.

The Broader Significance

This message represents more than just a single bug fix. It illustrates a class of problems that become increasingly common as GPU servers scale to 8+ GPUs and virtualization technologies like SEV-SNP become mainstream. The tension between security (IOMMU translation, memory encryption) and performance (P2P DMA, direct GPU communication) is a fundamental architectural challenge. The assistant's diagnosis — identifying that canAccessPeer=True does not guarantee correct P2P operation — is a subtle but critical insight. CUDA's capability query checks whether the hardware topology supports P2P, but it cannot predict whether the IOMMU configuration will allow it to function correctly.

The fix itself, NCCL_P2P_DISABLE=1, is a pragmatic workaround rather than a true solution. It sacrifices the performance benefits of direct GPU-to-GPU communication (which on PCIe Gen5 can reach 32 GB/s per direction) in favor of correctness via shared memory (which is limited to CPU memory bandwidth, typically 50-100 GB/s for the whole system). For the Qwen3.5-122B model with TP=4, this tradeoff proved acceptable — subsequent benchmarks (shown in <msg id=6228> and beyond) achieved up to 2,800 tok/s at high concurrency, demonstrating that SHM-based communication was sufficient for excellent throughput.

The message also showcases the importance of reading the kernel's diagnostic output. The IO_PAGE_FAULTs in dmesg were the critical clue that the user provided, and the assistant's ability to interpret them correctly — understanding that address=0x24000000000 and flags=0x0030 indicated a GPU BAR address and a DMA translation fault — was essential to the diagnosis. This kind of low-level system knowledge, combining GPU internals, IOMMU architecture, and distributed computing concepts, is what separates a superficial debugging session from a definitive root cause analysis.

Conclusion

Message <msg id=6217> is a masterclass in diagnostic reasoning under pressure. In a single message, the assistant synthesizes evidence from multiple sources (CUDA P2P tests, kernel IOMMU faults, NCCL behavior), identifies the root cause of a persistent hang, formulates a testable fix, and documents both a successful approach and a dead end. The message demonstrates that effective debugging is not about knowing the answer in advance, but about systematically connecting observations across layers of abstraction — from application-level distributed initialization down to hardware-level DMA translation — until the fault reveals itself.