The Weight of a Single Command: Deploying a Fix for IOMMU-Broken P2P DMA

scp /home/theuser/glm-kimi-sm120-rtx6000bw/sglang-qwen.service root@10.1.230.174:/etc/systemd/system/sglang-qwen.service && ssh root@10.1.230.174 'systemctl daemon-reload && systemctl start sglang-qwen && echo "starting..."'
starting...

On its surface, message [msg 6225] is unremarkable: a single bash command that copies a systemd service file to a remote machine, reloads the daemon, and starts a service. The output is a single word: starting.... Yet this message represents the culmination of an intense diagnostic journey spanning dozens of interactions, the resolution of a subtle hardware-virtualization interaction, and the moment when a production inference service finally came back to life after being broken by an architectural change at the host level.

The Context: A Machine Torn in Two

To understand why this message matters, one must understand what preceded it. The machine in question — an ASUS ESC8000A-E13 server with dual AMD EPYC 9335 processors and eight NVIDIA RTX PRO 6000 Blackwell GPUs — had been running SGLang serving a Qwen3.5-122B-A10B model across all eight GPUs with tensor parallelism (TP=8). The system worked. Then a reconfiguration happened: the eight GPUs were split, with four bound to an LXC container running SGLang and four assigned to a SEV-SNP confidential VM via vfio-pci passthrough. The SGLang service was updated to TP=4 and the model was placed on a new storage path (/shared).

The first sign of trouble came from the host kernel logs: IO_PAGE_FAULT errors from the AMD-Vi IOMMU subsystem, with the NVIDIA GPUs attempting DMA to addresses like 0x24000000000 and 0x34000010000 — the BAR addresses of peer GPUs. These were GPU-to-GPU P2P DMA transfers being blocked by the IOMMU.

The Diagnostic Chain

The assistant's response to these errors was methodical. First, it confirmed basic GPU functionality: a simple PyTorch tensor creation on a single device worked fine. Then it checked P2P capability reporting: torch.cuda.can_device_access_peer() returned True for every pair, meaning the CUDA driver believed P2P was available. But when the assistant actually tested P2P data transfer — copying a tensor from one GPU to another and comparing values — every single transfer produced a mismatch. Every GPU-to-GPU copy, across all 12 directed pairs, returned corrupted data.

This was the critical finding: the CUDA driver was advertising P2P capability (because the hardware supports it and PCIe topology hasn't changed), but the IOMMU in full translation mode was silently corrupting every P2P DMA transaction. The SEV-SNP configuration required amd_iommu=on (not iommu=pt passthrough mode), which meant every DMA transfer had to go through IOMMU translation tables. GPU-to-GPU P2P DMA, which relies on direct BAR-to-BAR transfers, was fundamentally incompatible with this setup.

The assistant then confirmed the fix path: running NCCL all-reduce with NCCL_P2P_DISABLE=1 worked perfectly across all four GPUs. The solution was to force NCCL to use shared memory (SHM) transport instead of P2P DMA.

Two Layers of Configuration

The fix was applied at two levels. First, the assistant updated /usr/lib/python3.12/sitecustomize.py on the remote machine ([msg 6221]), adding NCCL_P2P_DISABLE=1 along with a set of NCCL tuning parameters optimized for PCIe Gen5 Blackwell GPUs under IOMMU translation: the LL (Low Latency) protocol, Ring algorithm, 16 channels, 16 MB buffer size, and 512 threads. The comment in the code explicitly documents the reason: "P2P is DISABLED because IOMMU in non-passthrough mode blocks GPU-to-GPU DMA."

Second, the assistant edited the systemd service file (sglang-qwen.service) to add NCCL_P2P_DISABLE=1 and CUDA_DEVICE_MAX_CONNECTIONS=1 to the environment block ([msg 6223]). This ensured the fix would persist across service restarts regardless of how the process was launched.

What This Message Actually Does

Message [msg 6225] performs three operations chained together with &&:

  1. scp: Copies the locally-edited service file to the remote container's systemd directory. This is a deliberate choice — the assistant could have edited the file in-place on the remote machine, but copying from a local source ensures the version control and local state remain consistent.
  2. systemctl daemon-reload: Tells systemd to re-read the service unit files. This is necessary because the file was replaced.
  3. systemctl start sglang-qwen: Actually launches the service. The trailing && echo "starting..." provides immediate feedback — the service may take minutes to load the 234 GB model, so the echo confirms the launch sequence initiated successfully. The output starting... is the first positive signal after a long chain of failures. Earlier attempts to start SGLang had resulted in hangs during init_torch_distributed because NCCL would attempt P2P all-reduce, get corrupted data, and deadlock. This time, with P2P disabled, the distributed initialization would use SHM transport instead.

Assumptions and Knowledge

This message makes several implicit assumptions. It assumes the service file on disk is correct — that the edit applied in [msg 6223] actually modified the right lines and didn't introduce syntax errors. It assumes the remote machine is reachable and has the sglang-qwen.service path writable. It assumes the model weights are present at the path specified in the service file. It assumes the CUDA environment, Python virtual environment, and all dependencies are still correctly configured from previous setup work.

The input knowledge required to understand this message is substantial. One must know that NCCL_P2P_DISABLE=1 is the critical environment variable that fixes the IOMMU issue. One must understand that SGLang uses NCCL for distributed tensor parallelism across GPUs. One must know the service file was previously edited and contains the correct model path, TP=4 setting, and other parameters. One must understand the systemd service lifecycle — that daemon-reload is needed after modifying a unit file, and that start will trigger ExecStart which launches the Python server.

Output Knowledge Created

This message creates a running SGLang inference server serving Qwen3.5-122B-A10B in BF16 across four Blackwell GPUs, with NCCL operating in P2P-disabled mode. It also creates a persistent configuration: the service file on the remote machine now contains the P2P fix, so any future restart (whether manual or after a reboot) will apply the same workaround.

The message also implicitly validates the entire diagnostic chain. If the service starts successfully (which subsequent messages confirm it does), then the hypothesis was correct: P2P DMA corruption was the root cause, and disabling it resolves the issue. The message is, in effect, the experimental confirmation of a hypothesis formed through careful observation of IO_PAGE_FAULTs, systematic elimination of alternative causes, and targeted testing.

The Broader Significance

This episode illustrates a class of problem increasingly common in modern AI infrastructure: the tension between security virtualization features (SEV-SNP, IOMMU) and high-performance GPU computing. SEV-SNP provides confidential computing guarantees by encrypting VM memory and enforcing strict IOMMU translation, but this breaks the direct P2P DMA that multi-GPU frameworks rely on for efficient all-reduce. The workaround — disabling P2P and falling back to SHM transport — carries a performance cost, but it's the only option when the hardware topology cannot be changed.

The message also demonstrates the importance of testing assumptions. The CUDA driver reported canAccessPeer=True for all GPU pairs, which would lead most operators to believe P2P worked. Only by actually transferring data and comparing results was the silent corruption revealed. In high-performance computing, "can" does not mean "works correctly."

For the reader, this single bash command — 156 characters — represents hours of diagnostic work, a deep understanding of GPU architecture, IOMMU virtualization, NCCL internals, and systemd service management. It is a reminder that in complex systems, the most valuable code is often not the code that implements new features, but the code that correctly applies a hard-won fix.