The Diagnostic Pivot: When IOMMU Broke GPU P2P DMA

Message: [bash] ssh root@10.1.230.174 'kill -9 $(pgrep -f sglang) 2>/dev/null; fuser -k /dev/nvidia* 2>/dev/null; sleep 2; ~/ml-env/bin/python3 /root/gpu_diag.py 2>&1'

At first glance, this message looks like a routine command: kill any lingering SGLang processes, release GPU device handles, wait a moment, then run a diagnostic script. But in the arc of this coding session, this message represents a critical inflection point — the moment when the assistant pivoted from trying to fix a broken deployment through configuration changes to running bare-metal diagnostics to understand why the deployment was broken at the hardware level. It is the bridge between a failed hypothesis and a confirmed root cause.

The Crisis That Preceded This Message

To understand why this message was written, one must understand the cascade of failures that led to it. The assistant had been deploying the Qwen3.5-122B-A10B model (a 234 GB BF16 model) across 4 NVIDIA RTX PRO 6000 Blackwell GPUs in an LXC container on a Proxmox host. The server had been reconfigured in a previous segment ([msg 6196]): the 8 GPUs were split, with 4 bound to the nvidia driver for the container and 4 moved to vfio-pci for a SEV-SNP confidential VM. This reconfiguration enabled amd_iommu=on in full translation mode — a requirement for SEV-SNP — which fundamentally changed how PCIe devices could communicate.

The symptom was unmistakable: SGLang would hang during init_torch_distributed, with all 4 TP workers blocked on futex and socket reads ([msg 6172]). The assistant spent multiple rounds chasing this as a software issue — checking NCCL environment variables, testing different transport protocols, pulling the latest SGLang commits, reinstalling from source. Each attempt failed identically. The server would start loading, the GPUs would allocate ~1 GB of memory each, and then everything would freeze.

Then the user provided the missing piece: IO_PAGE_FAULT logs from the Proxmox host's kernel ([msg 6191]). Lines like nvidia 0000:01:00.0: AMD-Vi: Event logged [IO_PAGE_FAULT domain=0x002c address=0x24000000000 flags=0x0030] revealed that the NVIDIA GPUs were attempting Direct Memory Access to addresses that the IOMMU (AMD-Vi) refused to translate. The flags=0x0030 pattern indicated a write to a non-present page — the IOMMU had no mapping for the target address.

Why This Message Was Written

The assistant's reasoning, visible in the preceding messages ([msg 6209], [msg 6211]), shows a clear diagnostic chain:

1. Observation: IO_PAGE_FAULTs involve addresses like 0x24000000000 and 0x34000010000, which match the BAR (Base Address Register) ranges of other GPUs.
2. Hypothesis: The GPUs are attempting P2P (peer-to-peer) DMA directly to each other's memory, which is the standard path for NCCL all-reduce operations. Under full IOMMU translation, these direct physical addresses are invalid because the IOMMU requires IOVA (I/O Virtual Addresses), not raw physical addresses.
3. Implication: If P2P DMA is broken, NCCL cannot initialize because it relies on cudaDeviceEnablePeerAccess to establish direct GPU-to-GPU communication channels.
4. Action: Before attempting any more complex fixes (like patching NCCL or SGLang to avoid P2P), the assistant needed to confirm that the GPUs themselves were functional and that the only broken path was P2P DMA. This message executes that action. It runs gpu_diag.py — a diagnostic script the assistant had just written and transferred via SCP ([msg 6212]) — which was designed to test basic single-GPU CUDA operations and explicit P2P transfers between GPU pairs.

The Thinking Process: What the Assistant Expected

The assistant's reasoning reveals several assumptions. First, it assumed that single-GPU operations would work fine — the GPUs had been verified earlier in the session (segment 39) and were allocating memory normally. The real question was whether cudaDeviceCanAccessPeer and cudaDeviceEnablePeerAccess would succeed or fail. The script was structured to test both paths: basic tensor operations on each GPU individually, followed by cross-GPU P2P transfers.

Second, the assistant assumed that the diagnostic script would produce visible output — either confirming P2P failure with specific error messages, or unexpectedly succeeding (which would point the investigation back toward software). The 2>&1 redirection was deliberate: it captured both stdout and stderr to ensure no diagnostic information was lost.

Third, the assistant assumed that killing SGLang processes and releasing GPU handles with fuser -k /dev/nvidia* was necessary to avoid resource conflicts. The sleep 2 provided a grace period for the NVIDIA driver to release internal state.

Input Knowledge Required

Understanding this message requires knowledge spanning several domains:

• GPU architecture: That Blackwell GPUs (RTX PRO 6000, SM120) communicate via P2P DMA over PCIe Gen5, and that NCCL depends on this for tensor-parallel communication.
• IOMMU and virtualization: That amd_iommu=on (full translation mode) is distinct from iommu=pt (passthrough mode), and that SEV-SNP confidential computing requires full translation. In passthrough mode, the IOMMU essentially lets devices use physical addresses directly; in translation mode, every DMA must go through an IOVA mapping.
• NVIDIA driver internals: That fuser -k /dev/nvidia* releases GPU file handles held by crashed processes, which is necessary before reinitializing CUDA.
• Linux process management: That pgrep -f sglang matches processes by full command line, and that kill -9 sends SIGKILL.
• The specific hardware topology: 8 RTX PRO 6000 GPUs on an ASUS ESC8000A-E13 with dual AMD EPYC 9335 CPUs, split between LXC and VM.

Output Knowledge Created

The output of this message was, unexpectedly, nothing at all. The next message ([msg 6214]) reveals: "No output at all." The diagnostic script produced zero output — not even an error message. This silence was itself a data point. It meant the script either:

• Crashed before producing any output (perhaps an import error or segfault),
• Hung indefinitely on a CUDA API call (perhaps cudaDeviceEnablePeerAccess blocks when IOMMU is misconfigured),
• Or produced output that was swallowed by some shell buffering issue. The assistant's immediate response was to fall back to a simpler inline test: ~/ml-env/bin/python3 -c "import torch; print(torch.cuda.device_count()); x=torch.randn(10,device='cuda:0'); print(x)". This succeeded, confirming 4 GPUs were visible and basic CUDA operations worked. The contrast between the silent diagnostic script and the working inline test was itself diagnostic: the failure was specific to whatever gpu_diag.py did beyond basic tensor operations — almost certainly the P2P test.

The Broader Significance

This message exemplifies a pattern that recurs throughout the session: when high-level configuration fixes fail, the assistant systematically strips away layers of abstraction to test the hardware directly. The diagnostic script was the assistant's attempt to isolate the IOMMU hypothesis from confounding variables like NCCL versions, SGLang patches, or PyTorch compatibility. By running a minimal CUDA test outside the SGLang framework, the assistant could determine whether the problem was in the application layer or the hardware/driver layer.

The empty output forced a further simplification — the inline Python one-liner — which finally confirmed the baseline. From there, the assistant would go on to discover that NCCL_P2P_DISABLE=1 (forcing NCCL to use shared memory transport instead of P2P DMA) resolved the hang entirely, and the server would eventually achieve 2,800 tok/s at high concurrency. But that solution depended entirely on first understanding that P2P DMA was the broken link — understanding that began with this seemingly unremarkable command.