Chunk 40.0

In this chunk, the assistant reconfigured the GPU topology on the Proxmox host, splitting the 8× RTX PRO 6000 Blackwell GPUs so that 4 (NUMA 0) remained bound to the nvidia driver for the LXC container running SGLang, while the other 4 (NUMA 1) were moved to vfio-pci for SEV-SNP VM passthrough. This involved updating the LXC config, creating a new PCI mapping (`pro6000-vm`), and setting up a systemd service (`gpu-vfio-split.service`) to persist the binding across reboots. The SGLang service was updated from TP=8 to TP=4, and the Qwen3.5-397B NVFP4 model (previously on `/data`) was replaced with Qwen3.5-122B-A10B BF16 (native precision, 234 GB) stored on `/shared`. A critical issue emerged: the SEV-SNP configuration enabled full IOMMU translation (`amd_iommu=on`), which broke GPU-to-GPU P2P DMA. Every P2P transfer produced corrupted data, causing NCCL to hang during `init_torch_distributed`. The assistant diagnosed this via IO_PAGE_FAULTs in dmesg and a CUDA P2P test, then fixed it by adding `NCCL_P2P_DISABLE=1` to the environment, forcing NCCL to use SHM transport. The server then loaded successfully, achieving 108 tok/s single-stream and up to 2,800 tok/s at C=128. The assistant also fixed a driver version mismatch (container had 565 userspace vs host 590 kernel) by installing the matching 590 packages. The chunk concludes with the assistant investigating BIOS-level options to re-enable P2P DMA under IOMMU translation, exploring PCIe ACS, ARI, and IOMMU coverage settings.

The Blackwell GPU Split: From Topology Reconfiguration to P2P DMA Debugging on Proxmox 2391 words

Message Articles