The IOMMU Barrier: Diagnosing and Fixing GPU P2P DMA Corruption Under SEV-SNP

Message Overview

In message [msg 6222] of this opencode session, the assistant takes a decisive step toward resolving a critical production-blocking issue: SGLang's distributed initialization hangs when loading the Qwen3.5-122B-A10B BF16 model across 4 NVIDIA RTX PRO 6000 Blackwell GPUs. The message is deceptively brief — a single read tool call to inspect a systemd service file, paired with a declarative statement of intent to add two environment variables (NCCL_P2P_DISABLE=1 and CUDA_DEVICE_MAX_CONNECTIONS=1) before relaunching the server. But this message represents the culmination of an extensive diagnostic chain and the pivot from investigation to remediation.

The full content of the message is:

[assistant] Now let me also add NCCL_P2P_DISABLE=1 and CUDA_DEVICE_MAX_CONNECTIONS=1 to the service file, and try launching SGLang again: [read] /home/theuser/glm-kimi-sm120-rtx6000bw/sglang-qwen.service [path]/home/theuser/glm-kimi-sm120-rtx6000bw/sglang-qwen.service[/path] [type]file[/type] [content]1: [Unit] 2: Description=SGLang Qwen3.5-122B-A10B BF16 3: After=network.target 4: After=nvidia-persistenced.service 5: 6: [Service] 7: Type=simple 8: User=root 9: WorkingDirectory=/root/sglang-main 10: 11: # CUDA and NCCL environment 12: Environment=CUDA_HOME=/usr/local/cuda-13.0 13: Environment=PATH=/root/ml-env/bin:/usr/local/cuda-13.0/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin...

To understand why this seemingly minor edit matters, one must trace the full arc of the diagnosis that led here.

The Diagnostic Chain: From Hang to IO_PAGE_FAULT

The story begins with SGLang failing to initialize. The server process would start, all four GPU ranks would connect to the TCPStore (indicating init_process_group completed), but then the process would hang indefinitely during initialize_model_parallel or immediately after. No error messages, no stack traces — just a silent deadlock.

The user provided a crucial clue: IO_PAGE_FAULT entries in the Proxmox host's kernel log. These AMD-Vi (IOMMU) events showed the NVIDIA GPUs attempting DMA transactions to addresses like 0x24000000000 and 0x34000010000 — addresses that correspond to other GPUs' BAR (Base Address Register) regions. Every such transaction was being blocked by the IOMMU.

The assistant connected this to a configuration change documented in the setup notes (v1-uefi-settings.md and snp-pxm8-v1.md). The Proxmox host had been configured for SEV-SNP (Secure Encrypted Virtualization — Secure Nested Paging) confidential computing, which requires the IOMMU to operate in full translation mode (amd_iommu=on on the kernel command line). In this mode, the IOMMU translates every DMA address through its page tables, rather than allowing devices to bypass translation (the iommu=pt passthrough mode). GPU-to-GPU P2P DMA, which relies on direct memory access between devices using physical BAR addresses, breaks under full IOMMU translation because those addresses are not mapped in the IOMMU page tables.

The assistant confirmed this hypothesis with a simple but devastating test: a CUDA program that copies a tensor from one GPU to another and checks for data integrity. Every single GPU-to-GPU transfer produced data mismatches. The driver reported canAccessPeer=True — meaning the CUDA driver believed P2P was available — but the actual data arriving at the destination GPU was corrupted. This is the classic symptom of DMA transactions being silently mangled by IOMMU translation: the source GPU writes to a physical address, the IOMMU attempts to translate it, fails to find a valid mapping, and either blocks the transaction or lets corrupted data through.

The Fix: NCCL_P2P_DISABLE=1

With the root cause identified, the solution was clear: disable GPU-to-GPU P2P DMA in NCCL, forcing it to use alternative transport mechanisms. The assistant tested this approach by writing a small NCCL test script (nccl_test.py) and running it with NCCL_P2P_DISABLE=1. The all-reduce operations completed successfully across all four GPUs, confirming that NCCL could fall back to shared memory (SHM) transport — routing data through CPU system memory via PCIe — when P2P was unavailable.

The assistant had already updated /usr/lib/python3.12/sitecustomize.py to set NCCL_P2P_DISABLE=1 as a default for all Python processes. But this message represents the recognition that a second, more robust layer of configuration is needed: the systemd service file that launches SGLang in production. The sitecustomize approach works for interactive Python sessions and scripts, but systemd services have their own environment management that can override or bypass Python-level defaults. By adding the environment variable directly to the [Service] section's Environment= directives, the assistant ensures that NCCL will never attempt P2P DMA regardless of how the process is started.

The Second Variable: CUDA_DEVICE_MAX_CONNECTIONS=1

The assistant also adds CUDA_DEVICE_MAX_CONNECTIONS=1, which is a more subtle but important addition. This environment variable limits the number of concurrent CUDA driver connections that a single process can establish. In multi-GPU distributed setups, especially those using NCCL, the CUDA driver can create multiple connections to each GPU for different operations (kernel launches, memory transfers, synchronization). Under certain conditions — particularly when combined with IOMMU translation overhead and PCIe topology constraints — excessive concurrent connections can lead to driver-level deadlocks or resource exhaustion.

Setting this to 1 forces a serialized connection model, which reduces the potential for race conditions in the CUDA driver's connection management. This is a conservative tuning choice that prioritizes stability over peak throughput. The assistant likely included this based on experience with similar multi-GPU PCIe configurations where connection management became a bottleneck or failure point.

Assumptions and Trade-offs

The fix rests on several assumptions, each carrying implications for performance and correctness:

First assumption: Disabling P2P DMA is a complete workaround. NCCL's P2P transport is the fastest path for GPU-to-GPU communication because it uses direct memory access through the PCIe fabric without involving the CPU or system memory. The SHM fallback routes data through CPU RAM, which adds latency (data must travel GPU→CPU→GPU instead of GPU→GPU) and consumes system memory bandwidth. For a 122B parameter model using tensor parallelism across 4 GPUs, every all-reduce operation during inference will be slower. The assistant implicitly accepts this performance penalty as the cost of correctness.

Second assumption: The IOMMU configuration is immutable. The assistant correctly notes that another tenant is using the machine and that SEV-SNP requires full IOMMU translation. Changing the kernel command line to iommu=pt or amd_iommu=off would break the confidential VM's security guarantees and require coordination with the other tenant. The assistant explicitly avoids this path, instead working within the constraint.

Third assumption: No other component attempts P2P DMA outside of NCCL. The fix addresses NCCL specifically, but other CUDA libraries (like cuBLAS, cuDNN, or custom kernels) might also attempt P2P transfers. The assistant's earlier P2P test showed that even basic torch.Tensor.to() between devices produces corruption — this means any framework operation that moves data between GPUs is potentially affected. The fix assumes that SGLang's distributed operations go through NCCL, which is true for collective communication but may not cover all data movement paths.

Fourth assumption: The service file edit is sufficient for production deployment. Systemd service files have specific syntax requirements — multiple Environment= directives are allowed and accumulate, but the assistant needs to ensure the new variables don't conflict with existing ones. The message shows the current file content, which already has CUDA_HOME and PATH set, confirming the pattern is established.

The Broader Context: PCIe GPUs Under IOMMU

This issue highlights a fundamental tension in modern GPU computing: the conflict between confidential computing security guarantees and high-performance GPU-to-GPU communication. SEV-SNP encrypts VM memory and enforces strict IOMMU translation to prevent malicious hypervisors from accessing guest data. But GPU P2P DMA bypasses the CPU entirely — GPUs communicate directly over the PCIe bus using physical addresses that the IOMMU cannot translate without significant performance overhead.

NVIDIA's solution to this is the NCCL_P2P_DISABLE flag, which essentially says "use the CPU as a intermediary for all inter-GPU communication." This is reliable but slow. For Blackwell GPUs connected via PCIe Gen5 x16, the theoretical P2P bandwidth is approximately 128 GB/s per direction. With SHM fallback, the effective bandwidth drops to the PCIe-to-system-memory path, which is typically 50-60 GB/s and competes with CPU memory traffic.

The assistant's diagnostic approach — systematically testing P2P availability at the CUDA level, then NCCL with and without P2P — is a textbook example of layered debugging. By isolating the failure to the DMA transport layer, the assistant avoided chasing false leads in SGLang's distributed initialization code, model loading, or PyTorch version compatibility.

What This Message Creates

This message produces two forms of output knowledge. First, it surfaces the current state of the production service configuration, showing the existing CUDA environment setup and confirming the file path and structure. Second, it establishes the remediation plan: the two environment variables that will be added, and the intent to relaunch SGLang. The subsequent messages in the conversation will execute this plan and verify that the server starts successfully.

The message also implicitly documents a critical operational constraint: this machine cannot support GPU P2P DMA as long as SEV-SNP is enabled. Any future deployment that requires high-performance inter-GPU communication (e.g., larger tensor parallelism degrees, pipeline parallelism, or expert parallelism for MoE models) will need to either accept the SHM performance penalty or find a way to enable P2P under IOMMU — perhaps through NVIDIA's GPUDirect RDMA with appropriate IOMMU mappings, or by reconfiguring the BIOS to use IOMMU passthrough for the GPU devices while maintaining SEV-SNP for the VM.

Conclusion

Message [msg 6222] is a quiet turning point in a complex debugging session. It represents the moment when understanding crystallizes into action — when the root cause is confirmed, the fix is known, and the only remaining task is to apply it to the production configuration. The assistant's choice to modify both the Python sitecustomize and the systemd service file reflects a defense-in-depth approach to environment configuration, ensuring that the NCCL P2P disable takes effect regardless of how the process is launched. The addition of CUDA_DEVICE_MAX_CONNECTIONS=1 shows an awareness of secondary failure modes that could manifest under the constrained IOMMU configuration. While the performance implications of disabling P2P are significant, the fix is correct, minimal, and reversible — exactly what a production debugging intervention should be.