Diagnosing and Fixing GPU P2P DMA Corruption Under SEV-SNP IOMMU

In a single, deceptively simple bash command, a critical breakthrough was achieved in an extended debugging session spanning GPU topology reconfiguration, IOMMU analysis, and distributed computing diagnostics. The message at index 6219 reads:

[assistant] [bash] scp /home/theuser/glm-kimi-sm120-rtx6000bw/nccl_test.py root@10.1.230.174:/root/nccl_test.py && ssh root@10.1.230.174 'NCCL_P2P_DISABLE=1 ~/ml-env/bin/python3 /root/nccl_test.py 4 2>&1'
Testing NCCL all-reduce with 4 GPUs, P2P disabled...
Rank 3: all_reduce OK (got 10.0, expected 10.0)
Rank 1: all_reduce OK (got 10.0, expected 10.0)
Rank 2: all_reduce OK (got 10.0, expected 10.0)
Rank 0: all_reduce OK (got 10.0, expected 10.0)
Done

This message represents the moment when a week-long investigation into GPU-to-GPU communication failures finally converged on a confirmed diagnosis and a working mitigation. To understand why this simple test output was so significant, one must trace the chain of reasoning that led to it.

The Context: A GPU Topology Split Gone Wrong

The session leading up to this message was part of a larger effort to deploy large language models across a server with 8 NVIDIA RTX PRO 6000 Blackwell GPUs. The Proxmox host had been reconfigured to split these GPUs: 4 were bound to the nvidia driver for an LXC container running SGLang (a high-performance LLM serving framework), while the other 4 were passed through to a SEV-SNP confidential VM via vfio-pci. This split required enabling AMD's IOMMU in full translation mode (amd_iommu=on), which is mandatory for SEV-SNP but fundamentally changes how PCIe devices access each other's memory.

When the assistant attempted to launch SGLang with tensor parallelism across the 4 container GPUs, the server hung during init_torch_distributed. The user reported IO_PAGE_FAULT events in the host kernel log — AMD-Vi (the IOMMU) was blocking DMA transactions from the GPUs. This was the first clue that something was deeply wrong with GPU-to-GPU communication.

The Diagnostic Chain: From Symptom to Root Cause

The assistant's reasoning unfolded through a methodical series of tests, each eliminating a hypothesis and narrowing the search space. The first step was to verify that the GPUs themselves were functional. A simple PyTorch test confirmed that single-GPU operations worked correctly — each GPU could allocate tensors and perform computations. This ruled out driver installation issues, CUDA toolkit problems, or hardware faults at the individual GPU level.

Next, the assistant checked peer-to-peer (P2P) accessibility using torch.cuda.can_device_access_peer(). This CUDA runtime API queries the driver's P2P capability table, which is built during driver initialization based on PCIe topology and BAR mappings. The driver reported that all 4 GPUs could directly access each other's memory — canAccessPeer=True for every pair. This was misleading, as the next test would dramatically demonstrate.

The critical experiment was an actual P2P data transfer test. The assistant wrote a loop that copied a random tensor from each GPU to every other GPU and compared the result with the original. The output was devastating: every single GPU-to-GPU transfer produced data mismatches. Not a single pair survived the round-trip intact. The IOMMU, operating in full translation mode for SEV-SNP, was intercepting and corrupting the P2P DMA transactions. The GPUs' BAR addresses (the physical memory addresses they expose for peer access) were being translated by the IOMMU, and the translation was failing — hence the IO_PAGE_FAULTs in the host log.

This was the root cause of the SGLang hang. NCCL, NVIDIA's collective communications library, relies heavily on P2P DMA for efficient all-reduce operations during tensor-parallel inference. When NCCL attempted to set up its P2P communication channels, it received corrupted data, which caused the distributed initialization to deadlock or produce incorrect results.

The Failed Inline Attempt and the Pivot to a Script

Armed with this diagnosis, the assistant attempted to validate a fix: disable NCCL's P2P transport and force it to use shared memory (SHM) or socket-based communication instead. The first attempt used an inline Python script passed via -c to the remote SSH command. This script attempted to spawn two multiprocessing workers, each initializing a NCCL distributed process group and performing an all-reduce operation.

This attempt failed with a AttributeError: Can't get attribute 'worker' — a classic Python multiprocessing limitation. The spawn start method, which is required for NCCL to work correctly with CUDA, cannot pickle functions defined in the __main__ module of a -c script. The function worker was defined inline and was not importable by the child processes. This is a well-known gotcha in distributed PyTorch programming, and the assistant recognized it immediately.

The response was pragmatic: write the test as a proper Python file and copy it to the remote machine. The assistant created nccl_test.py in the local workspace directory and prepared to transfer it via scp.

The Confirming Message: NCCL_P2P_DISABLE=1 Works

Message 6219 executed this plan. The command performed two operations in sequence: first, it copied the test script to the remote container; second, it executed the script with NCCL_P2P_DISABLE=1 set in the environment. This environment variable tells NCCL to avoid all P2P communication channels — no NVLink, no PCIe peer-to-peer, no GPUDirect. Instead, NCCL falls back to using shared memory (for GPUs within the same NUMA node) or TCP sockets (for GPUs across NUMA boundaries).

The output was unambiguous and complete:

Testing NCCL all-reduce with 4 GPUs, P2P disabled...
Rank 3: all_reduce OK (got 10.0, expected 10.0)
Rank 1: all_reduce OK (got 10.0, expected 10.0)
Rank 2: all_reduce OK (got 10.0, expected 10.0)
Rank 0: all_reduce OK (got 10.0, expected 10.0)
Done

All four ranks successfully performed a distributed all-reduce operation. Each rank started with a tensor initialized to (rank + 1) (so values 1, 2, 3, 4), and after the all-reduce, every rank held the sum: 10.0. The fact that all four ranks reported the same correct value proved that NCCL was functioning correctly over the alternative transport.

Why This Matters: Assumptions, Knowledge, and Implications

Several assumptions underlay this diagnostic work. The assistant assumed that the IO_PAGE_FAULTs were the cause of the SGLang hang, not a symptom of a different problem. It assumed that NCCL would gracefully fall back to non-P2P transports when NCCL_P2P_DISABLE=1 was set — an assumption that held true for this hardware configuration but might not hold for all GPU topologies. It also assumed that the shared memory transport would provide sufficient bandwidth for tensor-parallel inference, which was later validated by throughput benchmarks.

The input knowledge required to interpret this message includes: understanding of PCIe P2P DMA and how IOMMU translation affects it; familiarity with NCCL's transport layers (P2P, SHM, socket); knowledge of the NCCL_P2P_DISABLE environment variable and its semantics; understanding of distributed all-reduce as a correctness test; and awareness of the Python multiprocessing spawn limitation that necessitated the script-based approach.

The output knowledge created by this message is substantial. It confirmed that the IOMMU-induced P2P corruption was the sole cause of the SGLang deployment failure. It validated a deployable fix — setting NCCL_P2P_DISABLE=1 in the SGLang service environment. It established that NCCL's non-P2P transports work correctly across the 4 Blackwell GPUs in this configuration. And it ruled out more complex explanations such as driver bugs, PCIe topology issues, or PyTorch/NCCL version incompatibilities.

The Thinking Process Visible in the Message

The message itself reveals a disciplined diagnostic methodology. The assistant did not jump to conclusions or apply fixes blindly. It followed a logical progression: verify basic GPU functionality, check P2P capability reporting, test actual P2P data integrity, attempt NCCL with P2P disabled, handle the inevitable multiprocessing edge case, and finally execute the clean script-based test. Each step either confirmed a hypothesis or eliminated a variable.

The choice to use NCCL_P2P_DISABLE=1 rather than more drastic measures (like NCCL_NET=Socket or NCCL_IB_DISABLE=1) was deliberate. P2P disable is the minimal intervention — it only affects GPU-to-GPU direct memory access, leaving other NCCL optimizations (like NVLink bandwidth if available, or InfiniBand networking) intact. On these Blackwell GPUs connected via PCIe Gen5, the P2P disable forces NCCL to route inter-GPU communication through the CPU's shared memory, which adds latency but preserves correctness.

The test itself was well-designed. Using torch.ones(1024, device=f"cuda:{rank}") * (rank + 1) creates distinct per-rank values that make the all-reduce result trivially verifiable: the sum of 1+2+3+4 = 10. Any deviation from 10.0 would indicate data corruption. Testing all 4 ranks simultaneously also validates that the distributed group initialization works correctly, not just the data transfer.

The Broader Significance

This message represents a turning point in the deployment. Before it, the SGLang server was hanging indefinitely with no clear path forward. After it, the fix was known and could be applied to the production configuration. The assistant went on to update the SGLang systemd service with NCCL_P2P_DISABLE=1, launch the server successfully, and achieve throughput exceeding 2,800 tokens per second at high concurrency.

The deeper lesson is about the tension between security virtualization (SEV-SNP) and high-performance GPU computing. SEV-SNP requires full IOMMU translation to protect VM memory from the hypervisor, but this translation breaks the GPU driver's assumption that P2P DMA addresses are identity-mapped. The NVIDIA driver reports canAccessPeer=True based on PCIe topology information gathered at driver load time, but it does not verify that IOMMU translation will actually allow the transactions to complete. This creates a dangerous gap between capability reporting and actual functionality — a gap that the assistant's diagnostic tests exposed and then worked around.

For practitioners deploying GPUs in virtualized environments with IOMMU enabled, this message demonstrates both the diagnostic technique (systematic P2P data integrity testing) and the mitigation (disabling NCCL P2P) that can save days of debugging. It also highlights the importance of reading kernel logs: the IO_PAGE_FAULT events in dmesg were the canary in the coal mine, visible long before the SGLang hang manifested.