The Blackwell GPU Split: From Topology Reconfiguration to P2P DMA Debugging on Proxmox

Introduction

In the life of a production machine learning infrastructure, few moments are as consequential as the decision to repartition scarce hardware resources. This article chronicles a single, intense session in an opencode coding conversation where an AI assistant reconfigured eight NVIDIA RTX PRO 6000 Blackwell GPUs on a Proxmox hypervisor host, splitting them between an LXC container running SGLang and a confidential computing VM. What began as a straightforward topology change evolved into a multi-layered debugging saga involving driver version mismatches, persistent boot-time binding mechanisms, and a subtle P2P DMA corruption issue under SEV-SNP IOMMU translation that nearly derailed the entire deployment.

The session spans over 160 messages (indexes 6026 through 6188) and covers four major phases: GPU topology reconfiguration and persistent binding, model swap from Qwen3.5-397B to Qwen3.5-122B, driver version mismatch diagnosis and resolution, and the eventual discovery of P2P DMA corruption under IOMMU translation. Each phase built on the previous one, and the final resolution — forcing NCCL to disable P2P transport — required understanding the interaction between AMD's Secure Encrypted Virtualization, PCIe passthrough, and NVIDIA's collective communication library.

Phase 1: The GPU Topology Split

The session's pivot point came when the user issued a decisive redirection in message 6034: "For now assume it's good enough. I now want to reconfigure the proxmox host such that the lxc container only has 4 GPUs and the other 4 are available for another VM / VMs through passthrough (nvidia driver to 4 on one numa, other 4 to vfio)." This single sentence marked the end of an extensive optimization effort for the Qwen3.5-397B-A17B-NVFP4 model and the beginning of a fundamental infrastructure reconfiguration.

The assistant's first task was to understand the existing GPU topology. Through a series of diagnostic commands documented across messages 6035-6043, the assistant mapped the eight Blackwell GPUs to their PCI addresses and NUMA nodes. The discovery was clean: four GPUs on NUMA 0 (PCI addresses 01:00.0, 11:00.0, 61:00.0, 71:00.0) and four on NUMA 1 (81:00.0, 91:00.0, e1:00.0, f1:00.0). The NUMA 0 GPUs would stay bound to the NVIDIA driver for the LXC container; the NUMA 1 GPUs would be moved to vfio-pci for VM passthrough.

The actual reconfiguration required multiple steps: stopping the SGLang server and LXC container, unbinding the NUMA 1 GPUs from the NVIDIA driver via sysfs, binding them to vfio-pci, updating the LXC configuration to only expose four GPUs, and creating a new Proxmox PCI mapping (pro6000-vm) for the VM-side GPUs. The assistant performed the initial split at runtime using direct sysfs manipulation — writing PCI addresses to /sys/bus/pci/drivers/vfio-pci/bind after unbinding them from the NVIDIA driver.

But runtime binding is ephemeral. A single reboot would undo all the work, and since all eight GPUs share the same PCI device ID (10de:2bb5), the NVIDIA driver would reclaim all of them on the next boot. The assistant recognized this fragility and committed to making the split persistent ([msg 6077]).

Persistence by Design: The Systemd Service

The assistant's reasoning about persistence, documented in messages 6077-6081, is a textbook example of constraint-driven design. The standard mechanism for binding devices to vfio-pci at boot is the vfio-pci.ids= kernel parameter, which tells the driver to claim any device matching the given vendor:device IDs. But as the assistant correctly identified, "Since all 8 GPUs have the same device ID (10de:2bb5), I can't use vfio-pci.ids= (that would grab all of them)." This constraint forced a more surgical approach: binding by PCI address rather than by device ID.

The solution was a systemd service (gpu-vfio-split.service) that runs a shell script (/usr/local/bin/gpu-vfio-split.sh) early in boot, before the NVIDIA driver claims the GPUs. The service file uses Before=nvidia-persistenced.service to ensure it runs before the NVIDIA stack initializes, and After=systemd-modules-load.service to ensure the vfio-pci kernel module is loaded first. The script itself iterates over the four NUMA 1 PCI addresses, unbinds them from their current driver (if any), registers the device ID with vfio-pci via the new_id sysfs interface, and binds them.

This design reflects a nuanced understanding of Linux PCI driver binding, boot ordering, and the constraints imposed by homogeneous hardware IDs. The service was enabled and verified, transforming a fragile manual procedure into a reliable boot-time automation.

Phase 2: The Model Swap

With the GPU topology reconfigured, the assistant turned to the model deployment. The Qwen3.5-397B-A17B-NVFP4 model — a 397-billion-parameter Mixture-of-Experts model using 4-bit NVFP quantization — had been running on all eight GPUs. But the user had already rendered a quality verdict in message 6091: "rm qwen, in use it's actually very low quality, we'll be deploying a similarly sized model." This was a decisive rejection delivered with the efficiency of a shell command annotation.

The replacement was Qwen3.5-122B-A10B, a 122-billion-parameter model (10 billion active parameters per token due to MoE) running in BF16 native precision. At approximately 234 GB, it fit comfortably on four 96 GB GPUs with tensor parallelism set to TP=4. The assistant downloaded the model to /shared (the /data volume was being retired), updated the SGLang systemd service file, and prepared to start the server.

But the first launch attempt crashed immediately. The error traceback revealed that the server was failing during argument parsing — specifically, the ServerArgs.from_cli_args method was rejecting the combination of flags for Multi-Token Prediction (MTP) with the hybrid GDN architecture. The assistant diagnosed the issue in message 6123: "Clear error — MTP with the hybrid GDN model needs --mamba-scheduler-strategy extra_buffer and SGLANG_ENABLE_SPEC_V2=1."

This diagnosis required deep knowledge of SGLang's internals. The Qwen3.5-122B model uses a hybrid architecture with 36 linear attention (GDN/recurrent) layers and 12 full attention layers, arranged with a full attention layer every fourth position. When speculative decoding (MTP) is enabled with this non-uniform layer structure, the scheduler needs additional KV cache buffers to handle the mismatch between draft tokens and verified tokens. The --mamba-scheduler-strategy extra_buffer flag allocates those buffers, and SGLANG_ENABLE_SPEC_V2=1 enables an alternative speculative decoding path designed for complex architectures.

The fix was applied as a six-character edit to the service file ([msg 6125]). But it wasn't enough.

Phase 3: The Driver Version Mismatch

After the MTP fix, the server still wouldn't start. This time, it hung indefinitely at the log line "Init torch distributed begin" — the point where NCCL initializes communication channels between the four GPU workers. The assistant spent over thirty messages (6152-6185) systematically debugging this hang.

The diagnostic process was methodical. The assistant checked NCCL debug output ([msg 6157]) and found that NCCL initialization actually completed successfully — the bootstrap phase finished in 4.4 milliseconds, and all four ranks connected. It checked for port conflicts, stale rendezvous files, and process leaks. It tried overriding NCCL environment variables, tested TP=1 (which confirmed the model loaded but OOM'd on a single GPU), and even pulled the latest SGLang main branch for potential fixes. Every attempt produced the same result: the four TP processes would connect via NCCL, establish their TCPStore, and then freeze — blocked on futex and socket reads.

The breakthrough came from an unexpected source. The user observed in message 6138: "So on the proxmox host we have made some changes to the driver, maybe we need to update in the VM to match?" This single sentence bridged a coordination gap. Another agent had been working on the Proxmox host simultaneously, upgrading the NVIDIA driver to version 590.48.01. The assistant, working inside the LXC container, was unaware of this change.

The investigation revealed a stark mismatch: the kernel module (inherited from the host) was version 590.48.01, but the container's userspace libraries — libnvidia-compute, libnvidia-decode, and nvidia-utils — were all version 565.57.01. This is a dangerous situation. The NVIDIA driver stack has two layers: the kernel module (running in the host OS) and the userspace libraries (running inside the container). These must be version-compatible. A mismatch can cause undefined behavior, including the exact NCCL hang symptom the assistant was observing.

The fix was surgical: install the matching 590 userspace packages from the Ubuntu apt repositories, remove the old 565 packages, and verify with nvidia-smi that version numbers were now consistent (<msg id=6146-6148>). The assistant then restarted the server ([msg 6149]).

Phase 4: The P2P DMA Corruption Under IOMMU

Even after the driver fix, the hang persisted. The assistant's NCCL debug output had shown that NCCL initialization itself completed fine, but something after that was blocking. The strace output ([msg 6172]) revealed all four processes blocked on futex and restart_syscall — classic symptoms of a deadlock or a blocking collective operation.

The assistant's investigation eventually led to the source code of SGLang's distributed initialization (<msg id=6175-6176>), where it traced the exact execution path from "Init torch distributed begin" through NCCL communicator creation to the barrier synchronization. The code revealed that after NCCL init, the processes enter a barrier that waits for all ranks to reach the same point before proceeding with model loading. If one rank never reaches that barrier — or if the barrier itself deadlocks — the entire initialization hangs.

The root cause, discovered through dmesg inspection and CUDA P2P tests, was P2P DMA corruption under SEV-SNP IOMMU translation. The Proxmox host's SEV-SNP (Secure Encrypted Virtualization with Secure Nested Paging) configuration enabled full IOMMU translation (amd_iommu=on), which broke GPU-to-GPU P2P DMA. Every P2P transfer produced corrupted data, causing NCCL to hang during init_torch_distributed — not during NCCL's own initialization (which uses shared memory for bootstrap), but during the subsequent barrier synchronization where NCCL attempts P2P transfers for the first time.

The fix was a single environment variable: NCCL_P2P_DISABLE=1. This forces NCCL to use SHM (shared memory via CPU) transport instead of P2P DMA, bypassing the broken IOMMU translation entirely. It's a performance hit — P2P DMA is faster than SHM — but it got the server running.

The Outcome: A Working Deployment

With NCCL_P2P_DISABLE=1 in place, the server loaded successfully. The assistant benchmarked the Qwen3.5-122B-A10B BF16 model on four Blackwell GPUs, achieving 108 tok/s single-stream and up to 2,800 tok/s at concurrency level 128. These are impressive numbers that vindicate the assistant's persistence through the debugging ordeal.

The chunk concludes with the assistant investigating BIOS-level options to potentially re-enable P2P DMA under IOMMU translation, exploring PCIe ACS (Access Control Services), ARI (Alternative Routing-ID Interpretation), and IOMMU coverage settings. This is forward-looking work — the P2P disable is a workaround, not a true fix, and re-enabling P2P DMA would improve performance for future workloads.

Themes and Lessons

Several themes emerge from this session:

The importance of cross-layer visibility. The driver version mismatch was invisible from within the container — the assistant had to be told about the host-level change by the user. This highlights a fundamental challenge in containerized GPU environments: the kernel module and userspace libraries live in different administrative domains, and a mismatch can cause failures that look like application bugs.

Constraint-driven design. The homogeneous device IDs of the Blackwell GPUs ruled out the standard vfio-pci.ids= approach for persistent binding, forcing the assistant to design a more surgical address-based solution. Good engineering is often about recognizing when the standard tool doesn't fit and crafting an alternative.

Systematic hypothesis elimination. The NCCL hang debugging is a masterclass in methodical troubleshooting. The assistant checked NCCL debug output, port conflicts, strace, source code, and driver versions before reaching for the "update the code" lever. Each negative result narrowed the search space and guided the next investigation.

The subtlety of virtualization bugs. The P2P DMA corruption under SEV-SNP IOMMU is a class of bug that is notoriously hard to diagnose. It doesn't produce clean error messages — it produces corrupted data that manifests as hangs, crashes, or silent correctness issues. The assistant's eventual discovery through dmesg and CUDA P2P tests required understanding the entire virtualization stack from the hypervisor's IOMMU configuration to NCCL's transport selection.

Conclusion

This session captures the full lifecycle of a complex ML infrastructure deployment: from topology reconfiguration and persistent automation, through model selection and deployment, through multi-layered debugging spanning driver versions and hardware virtualization, to a working production service. The assistant's systematic approach, its ability to pivot between hypotheses, and its deep understanding of the entire stack — from Linux PCI driver binding to NCCL transport protocols to SGLang's hybrid GDN architecture — made the difference between a stuck deployment and a running server serving 2,800 tokens per second.

The final investigation into BIOS-level P2P DMA options reminds us that in production systems, the work is never truly done. Every fix creates new questions, and every workaround points toward a deeper understanding of the hardware-software boundary.## References

[1] "The Architecture of Knowledge: How a Single Message Captured an Entire ML Infrastructure Deployment" — Analysis of message 6026, the comprehensive project status document that preceded the GPU topology reconfiguration.

[9] "The Pivot: Reconfiguring GPU Topology on a Proxmox Host" — The user's decisive redirection in message 6034 that initiated the GPU split.

[52] "Persistence by Design: Making a GPU Split Survive Reboot on Proxmox" — The assistant's reasoning about persistent GPU binding in message 6077.

[56] "Binding the Unbindable: Making GPU Passthrough Persistent on Proxmox with Systemd" — The systemd service implementation in message 6081.

[66] "The Pivot: A Model Swap Decision in One Line" — The user's quality verdict on Qwen3.5-397B in message 6091.

[76] "From Cleanup to Deployment: The Planning Message That Bridges Two Eras" — The structured todo list in message 6101 that planned the Qwen3.5-122B deployment.

[100] "The Six-Character Fix: How a Single Edit Unblocked Multi-Token Prediction on Blackwell" — The MTP configuration fix in message 6125.

[113] "The Driver Mismatch That Almost Broke Distributed Inference" — The user's diagnostic hypothesis in message 6138 about the driver version mismatch.

[124] "The Moment of Reckoning: Restarting After the Driver Fix" — The server restart attempt in message 6149 after the driver fix.

[132] "The NCCL Debug That Changed Everything" — The NCCL_DEBUG=INFO diagnostic in message 6157 that revealed NCCL initialization was not the problem.

[144] "The Empty Log: A Pivotal Diagnostic Moment in GPU Topology Troubleshooting" — The empty log discovery in message 6169 that shifted the diagnostic focus.

[161] "The Blackwell All-Reduce Hypothesis: A Pivot Point in Distributed Debugging" — The source code inspection in message 6186 targeting all-reduce architecture handling.

[163] "The Debugger's Dilemma: Retrying After Every Fix" — The reinstallation attempt in message 6188 after pulling latest SGLang.