Fixing GPU-to-GPU DMA Corruption Under IOMMU: The NCCL_P2P_DISABLE=1 Intervention

The Message

ssh root@10.1.230.174 'cat > /usr/lib/python3.12/sitecustomize.py << '\''PYEOF'\''
# install the apport exception handler if available
try:
    import apport_python_hook
except ImportError:
    pass
else:
    apport_python_hook.install()

# CUDA 13 environment
import os as _os
_os.environ.setdefault("CUDA_HOME", "/usr/local/cuda-13.0")
_os.environ.setdefault("TRITON_PTXAS_PATH", "/usr/local/cuda-13.0/bin/ptxas")
_os.environ.setdefault("SGLANG_DISABLE_CUDNN_CHECK", "1")

# NCCL tuning for PCIe Gen5 4xRTX PRO 6000 (IOMMU full translation mode)
# P2P is DISABLED because IOMMU in non-passthrough mode blocks GPU-to-GPU DMA
for _k, _v in [("NCCL_P2P_DISABLE", "1"), ("NCCL_PROTO", "LL"),
               ("NCCL_ALGO", "Ring"), ("NCCL_MAX_NCHANNELS", "16"),
               ("NCCL_BUFFSIZE", "16777216"), ("NCCL_NTHREADS", "512")]:
    if _k not in _os.environ:
        _os.environ[_k] = _v
del _k, _v
PYEOF
echo "sitecustomize updated"'

This single bash command, executed at message index 6221 in a long-running opencode coding session, represents a pivotal moment in an infrastructure debugging saga. It is the culmination of a diagnostic chain that began with a silent failure—NCCL hanging during init_torch_distributed—and ended with a precise, surgical fix that restored the entire distributed inference pipeline to working order. To understand why this message was written, we must trace the fault back to its root cause: the interaction between AMD's IOMMU (I/O Memory Management Unit) in full translation mode and NVIDIA's GPU-to-GPU Peer-to-Peer (P2P) DMA.

Why This Message Was Written: The Diagnostic Chain

The context leading up to this message is a story of progressive isolation. The assistant had been deploying the Qwen3.5-122B-A10B BF16 model across 4 RTX PRO 6000 Blackwell GPUs in an LXC container running on a Proxmox host. The server would hang during distributed initialization, specifically at the NCCL init_process_group step. Initial investigations ([msg 6177] through [msg 6189]) showed that TCPStore connections were established successfully—all four ranks could communicate via TCP—but the process would stall immediately after, during model parallelism setup.

The user then provided a crucial clue: IO_PAGE_FAULT events in the Proxmox host's kernel log ([msg 6191]). These faults, logged by the AMD-Vi (AMD Virtualization) IOMMU, showed the NVIDIA GPUs attempting DMA accesses to addresses like 0x24000000000 and 0x34000010000—addresses that correspond to other GPUs' BAR (Base Address Register) memory regions. The flags 0x0030 indicated that the IOMMU was blocking these transactions because the addresses had not been properly mapped through the translation tables.

This was the smoking gun. The Proxmox host had been configured for SEV-SNP (Secure Encrypted Virtualization - Secure Nested Paging) confidential computing, which requires amd_iommu=on—full IOMMU translation mode. In this mode, every DMA transaction must go through the IOMMU's address translation tables. GPU-to-GPU P2P DMA, which bypasses the CPU and directly transfers data between GPU memory spaces, relies on the GPUs knowing each other's physical addresses. Under full IOMMU translation, these physical addresses are not directly accessible; the GPUs must use I/O virtual addresses (IOVAs) that the IOMMU translates. The NVIDIA driver and hardware were attempting raw P2P transfers using physical addresses, and the IOMMU was dutifully blocking them, producing the IO_PAGE_FAULTs.

The assistant confirmed this hypothesis with a precise diagnostic sequence. First, a simple CUDA test ([msg 6215]) showed that torch.cuda.can_device_access_peer() returned True for all GPU pairs—the driver believed P2P was available. But an actual data transfer test ([msg 6216]) revealed the truth: every single GPU-to-GPU copy produced data mismatches. The driver's capability check was lying; the hardware path was corrupted.

Then, the assistant tested NCCL with NCCL_P2P_DISABLE=1 ([msg 6219]), which forces NCCL to use system memory (SHM/socket transport) instead of GPU P2P DMA. This worked perfectly: all four ranks completed all-reduce correctly. The root cause was confirmed, and the fix was clear.

How Decisions Were Made

The decision to apply the fix via sitecustomize.py rather than through environment variables in a systemd service file or a wrapper script was deliberate and reveals the assistant's understanding of the deployment architecture. The sitecustomize.py file is a Python mechanism: it is automatically imported by the Python interpreter at startup, before any user code runs. By placing environment variable assignments here, the assistant ensured that every Python process launched in the container—whether by systemd, by hand, or by a subprocess—would inherit these settings. This is more robust than modifying a single systemd service file, because it covers all execution paths.

The choice of os.environ.setdefault() for the CUDA variables and the conditional if _k not in _os.environ pattern for the NCCL variables is also deliberate. setdefault() only sets a key if it doesn't already exist, which means environment variables set before Python starts (e.g., via systemd environment directives or shell exports) will take precedence. The NCCL loop uses the same logic explicitly. This allows for override: if someone needs to test with P2P enabled or a different NCCL protocol, they can set the environment variable externally and the sitecustomize.py will respect that choice.

The NCCL tuning parameters themselves—NCCL_PROTO=LL (Low Latency), NCCL_ALGO=Ring, NCCL_MAX_NCHANNELS=16, NCCL_BUFFSIZE=16777216 (16 MB), NCCL_NTHREADS=512—were carried forward from a previous working configuration. In [msg 6220], the assistant read the existing sitecustomize.py and found these parameters already in use. The only new addition was NCCL_P2P_DISABLE=1. The assistant made a conscious decision to preserve the existing tuning while adding the critical fix, rather than starting from scratch.

Assumptions Made

The message makes several assumptions, most of which are well-supported by the diagnostic evidence. The primary assumption is that NCCL_P2P_DISABLE=1 is a complete and sufficient fix—that disabling P2P DMA will not cause other issues such as performance degradation, memory bandwidth bottlenecks, or correctness problems in other parts of the SGLang pipeline. The NCCL test with 4 GPUs confirmed that all-reduce works correctly, but it did not test the full SGLang server with attention backends, KV cache management, or the model's MoE (Mixture of Experts) layers. The assumption is that NCCL's SHM/socket transport is a faithful substitute for P2P DMA at the correctness level, differing only in performance.

A second assumption is that the IOMMU configuration is stable and will not change. The assistant explicitly notes in the comment that "IOMMU in non-passthrough mode blocks GPU-to-GPU DMA," but does not account for the possibility that the Proxmox host's IOMMU mode might be changed in the future (e.g., if the SEV-SNP VM is no longer needed). The fix is hardcoded to disable P2P unconditionally; it does not attempt to detect whether P2P would actually work at runtime.

A third assumption is that all 4 GPUs in the container are equally affected. The diagnostic tests showed that every GPU pair produced mismatches, so this is well-supported. However, the assistant did not test whether P2P between GPUs on the same PCIe root complex (within the same NUMA node) might work while cross-NUMA P2P fails. The IO_PAGE_FAULTs in dmesg showed faults on multiple PCI domains (0x002c, 0x003f, 0x000c), suggesting the problem is universal.

Mistakes and Incorrect Assumptions

One potential oversight is that the assistant did not verify that the NCCL_P2P_DISABLE=1 fix actually works with the full SGLang server before committing it to sitecustomize.py. The NCCL test was a minimal script that performed a single all-reduce operation. The actual SGLang server uses NCCL for much more complex distributed operations, including tensor parallelism across attention layers, MoE expert routing, and potentially custom all-reduce kernels (which SGLang implements via FlashInfer and other backends). The disable-custom-all-reduce flag was already present in the SGLang launch command from [msg 6188], but the interaction between NCCL_P2P_DISABLE=1 and SGLang's custom communication primitives was not tested.

Another subtle issue is the use of os.environ.setdefault() for the NCCL variables via the loop. The loop checks if _k not in _os.environ before setting, which is functionally equivalent to setdefault(). However, the comment says "P2P is DISABLED because IOMMU in non-passthrough mode blocks GPU-to-GPU DMA" but the code does not enforce this—it only sets the default. If another part of the system or a future update to the container sets NCCL_P2P_DISABLE=0 in the environment before Python starts, the fix would be silently overridden. A more defensive approach would be to force-set the value (unconditionally assign) and document why overriding is dangerous.

The assistant also did not explore alternative fixes. The IO_PAGE_FAULTs could potentially be resolved by configuring the IOMMU in passthrough mode for specific PCIe devices (via driver_override or ACS settings), or by using the iommu=pt kernel parameter for the container's GPUs while keeping full translation for the VM's GPUs. The user explicitly warned against changing BIOS settings or rebooting the host ([msg 6196]), but there might have been runtime IOMMU configuration options available. The assistant chose the software workaround (disable P2P) rather than the hardware configuration fix (fix IOMMU mapping), which is pragmatic given the constraints but leaves performance on the table.

Input Knowledge Required

To understand this message, the reader needs knowledge in several domains:

GPU Architecture and PCIe Topology: Understanding that GPUs can communicate directly via P2P DMA over PCIe, bypassing system memory and the CPU. This requires that GPUs know each other's physical address ranges (BAR spaces) and that the PCIe switch fabric supports peer-to-peer transactions.

IOMMU and Virtualization: Knowledge that an IOMMU translates device-visible addresses (I/O virtual addresses) to physical addresses, and that in full translation mode (amd_iommu=on), every DMA transaction must go through this translation. In passthrough mode (iommu=pt), the IOMMU identity-maps device addresses to physical addresses, allowing P2P to work transparently. SEV-SNP requires full translation for memory encryption and isolation.

NCCL (NVIDIA Collective Communications Library): Understanding that NCCL provides collective operations (all-reduce, all-gather, etc.) for multi-GPU communication, and that it can use multiple transport methods: P2P (direct GPU-to-GPU via NVLink or PCIe), SHM (shared memory via CPU), and socket (TCP/IP). NCCL_P2P_DISABLE=1 forces NCCL to use the slower SHM/socket paths.

Python Site Customization: The sitecustomize.py mechanism, which is automatically imported by the Python interpreter at startup, allowing environment-wide configuration without modifying application code.

The Deployment Context: The container runs SGLang (an inference serving framework) with tensor parallelism across 4 GPUs, using PyTorch's distributed package with NCCL backend. The model is Qwen3.5-122B-A10B, a large language model with Mixture of Experts architecture.

Output Knowledge Created

This message creates a permanent configuration change on the target system. The new sitecustomize.py file:

  1. Preserves existing functionality: The apport exception handler and CUDA 13 environment variables are retained.
  2. Disables GPU P2P DMA for NCCL: NCCL_P2P_DISABLE=1 is the critical fix that prevents NCCL from attempting corrupted P2P transfers through the IOMMU.
  3. Configures NCCL transport parameters: Sets LL (Low Latency) protocol, Ring algorithm, 16 channels, 16 MB buffer size, and 512 threads. These parameters were previously tuned for PCIe Gen5 GPUs and are preserved.
  4. Documents the rationale: The comment "P2P is DISABLED because IOMMU in non-passthrough mode blocks GPU-to-GPU DMA" serves as inline documentation for any future administrator who reads this file.
  5. Creates a precedent for future fixes: The pattern of using sitecustomize.py to inject NCCL environment variables can be extended if other NCCL issues arise. The knowledge created extends beyond the file itself. The diagnostic methodology—testing can_device_access_peer vs. actual data transfer, correlating IO_PAGE_FAULTs with P2P corruption, and isolating NCCL from P2P—is a reusable pattern for debugging similar issues on IOMMU-protected systems.

The Thinking Process

The reasoning visible in this message and its surrounding context shows a systematic, hypothesis-driven debugging approach. The assistant did not jump to conclusions. It:

  1. Observed the symptom: NCCL hangs during distributed init.
  2. Eliminated network issues: Verified TCPStore connections were working.
  3. Followed the user's clue: The IO_PAGE_FAULTs in dmesg pointed to IOMMU.
  4. Formulated a hypothesis: IOMMU in full translation mode is breaking P2P DMA.
  5. Tested the hypothesis: Ran a P2P data transfer test that confirmed corruption.
  6. Isolated the variable: Tested NCCL with P2P disabled and confirmed it works.
  7. Applied the fix: Updated sitecustomize.py to disable P2P by default.
  8. Documented the reasoning: Added explanatory comments to the configuration file. The elegance of this fix is that it addresses the symptom (corrupted P2P transfers) rather than the root cause (IOMMU configuration), which was off-limits due to the shared host environment. The assistant recognized the constraint—"another tenant is active on the machine so changes will need coordination" ([msg 6196])—and chose the pragmatic path: a software workaround that restores functionality without requiring a host reboot or BIOS change. The message also reveals the assistant's understanding of the system's layered architecture. The fix is applied at the Python interpreter level (sitecustomize.py), which is the lowest common denominator for all Python-based workloads on this system. It could have been applied at the systemd service level, the container level, or the shell profile level, but sitecustomize.py ensures that even ad-hoc Python scripts and subprocesses inherit the fix. This is a sign of mature systems thinking: apply the fix at the point where it will have the broadest correct coverage.

Conclusion

Message 6221 is a deceptively simple intervention—a single bash command that writes a configuration file—but it represents the resolution of a complex, multi-layered debugging effort. The fix itself is a one-line environment variable (NCCL_P2P_DISABLE=1), but the reasoning behind it required understanding GPU architecture, IOMMU virtualization, NCCL transport internals, and the specific constraints of a shared Proxmox host. The assistant's methodical approach—hypothesize, test, isolate, fix, document—turned a silent data corruption bug into a clean, documented workaround that restored the distributed inference pipeline to full operation.