The Pivot: Reconfiguring GPU Topology on a Proxmox Host

In the middle of a high-concurrency benchmarking run, the user issues a decisive redirection. Message [msg 6034] reads in full:

For now assume it's good enough. I now want to reconfigure the proxmox host such that the lxc container only has 4 GPUs and the other 4 are available for another VM / VMs through passthrough (nvidia driver to 4 on one numa, other 4 to vfio)

This single sentence marks a major inflection point in the coding session. The assistant had just initiated a benchmark of the Qwen3.5-397B-A17B-NVFP4 model at concurrency levels 1, 4, 16, 32, 64, 128, 256, and 512 (see [msg 6033]), following an extensive multi-session effort to build, patch, and deploy the model on eight NVIDIA RTX PRO 6000 Blackwell GPUs. The user cuts this work short with a pragmatic "assume it's good enough" and pivots to an entirely different problem: reconfiguring the GPU topology of the Proxmox hypervisor host itself.

The Reasoning and Motivation

The user's motivation is rooted in resource utilization. The machine (hostname kpro6) has eight Blackwell GPUs split across two NUMA nodes, with four GPUs on NUMA 0 and four on NUMA 1. Until this point, all eight were bound to the NVIDIA driver and passed through to a single LXC container (CT 129, llm-two) running the SGLang inference server. This meant the entire machine's GPU compute capacity was dedicated to a single workload — the Qwen3.5 model serving.

The user now wants to split the GPUs: keep four on the NVIDIA driver for the LXC container (the inference workload), and move the other four to the vfio-pci driver for passthrough to other virtual machines. This is a classic hypervisor resource partitioning decision. Rather than dedicating all hardware to one task, the user wants to enable multiple concurrent workloads — perhaps a separate VM for experimentation, training pipelines, or other ML tasks that don't need to share the same inference server.

The NUMA-awareness in the request is significant. The user explicitly says "nvidia driver to 4 on one numa, other 4 to vfio." This reflects an understanding that keeping GPUs on the same NUMA node together minimizes cross-NUMA memory traffic and PCIe contention. The four GPUs on NUMA 0 (PCI addresses 01:00.0, 11:00.0, 61:00.0, 71:00.0) will stay with the NVIDIA driver for the LXC, while the four on NUMA 1 (81:00.0, 91:00.0, e1:00.0, f1:00.0) will be unbound from NVIDIA and handed to vfio-pci for VM passthrough.

Why "Assume It's Good Enough"

The phrase "For now assume it's good enough" is telling. The assistant had been pursuing a detailed optimization agenda with multiple next steps: benchmarking at higher concurrency, testing MTP/NEXTN speculative decoding with real prompts, trying custom all-reduce optimizations, and exploring MSCCL++ for GPU-initiated communication (see [msg 6028]). The user is explicitly deprioritizing all of these.

This could reflect several things. Perhaps the benchmark results already met the user's performance requirements. Perhaps the user has a more urgent need to get the VM passthrough working — maybe another team member needs GPU access, or there's a separate experiment waiting to run. Or perhaps the user recognizes that further optimization of a single workload has diminishing returns compared to enabling new capabilities (VM passthrough). Whatever the reason, the user is making a deliberate tradeoff: accepting the current performance as sufficient in exchange for infrastructure flexibility.

Assumptions Embedded in the Request

The user makes several assumptions, some of which prove incorrect. First, they assume the reconfiguration is straightforward — that you can simply unbind four GPUs from the NVIDIA driver and rebind them to vfio-pci without complications. The subsequent conversation (see [msg 6035] onward) reveals this is far from simple: the assistant must map PCI addresses to NUMA nodes, update the LXC configuration, create new PCI mappings in Proxmox, build a systemd service to persist the binding across reboots, and — most critically — diagnose and fix a severe P2P DMA corruption issue caused by the SEV-SNP IOMMU configuration.

Second, the user assumes the SGLang server can simply be reconfigured from TP=8 (tensor parallelism across 8 GPUs) to TP=4 (across 4 GPUs) without issues. This turns out to be true eventually, but only after fixing the NCCL hang caused by broken P2P DMA under IOMMU translation.

Third, the user assumes the model currently deployed (Qwen3.5-397B-A17B-NVFP4, 223 GB) will still fit on 4 GPUs. This assumption is incorrect — the model is too large for 4 GPUs, which forces the assistant to replace it with a smaller model (Qwen3.5-122B-A10B BF16, 234 GB stored on /shared) later in the session.

Input Knowledge Required

To understand this message, one needs significant context about the infrastructure. The Proxmox host has eight NVIDIA RTX PRO 6000 Blackwell GPUs (PCIe Gen5, no NVLink), split across two NUMA nodes. The LXC container (CT 129) runs SGLang serving the Qwen3.5 model. There is an existing VM (131, ml-pipelines) that previously used all eight GPUs via a PCI mapping called pro6000. The host uses vfio-pci for GPU passthrough to VMs, and the GPUs have been bound to the NVIDIA driver for the LXC workload. The assistant has been working on this machine for many sessions, building CUDA 13 toolchains, patching SGLang for Blackwell SM120 support, and deploying production inference services.

One also needs to understand Proxmox's PCI mapping system (/etc/pve/mapping/pci.cfg), LXC device mount entries, and the vfio-pci driver binding mechanism. The assistant's subsequent investigation of NUMA topology using /sys/bus/pci/devices/*/numa_node and nvidia-smi output shows the kind of low-level system interrogation required.

Output Knowledge Created

This message creates the decision framework for the entire remainder of segment 40. It establishes:

  1. The split target: 4 GPUs on NVIDIA driver (NUMA 0) for LXC, 4 GPUs on vfio-pci (NUMA 1) for VM passthrough.
  2. The priority shift: Optimization work is deprioritized; infrastructure reconfiguration is now the focus.
  3. The acceptance criterion: Current performance is "good enough" — no further tuning needed before moving on. From this decision flows a cascade of concrete work: stopping the SGLang server, stopping the LXC container, unbinding GPUs from the NVIDIA driver, creating a new Proxmox PCI mapping (pro6000-vm), updating the LXC config to only mount 4 GPUs, setting up a systemd service (gpu-vfio-split.service) for persistent binding, and — critically — diagnosing the P2P DMA corruption under SEV-SNP IOMMU that nearly derails the entire effort.

The Thinking Process Visible in the Message

The user's thinking is concise but reveals several layers. The opening "For now assume it's good enough" is a judgment call — the user is evaluating the optimization work and deciding it has reached an acceptable plateau. The phrase "I now want to reconfigure" signals a deliberate task switch, not a reaction to a problem. The specificity of "nvidia driver to 4 on one numa, other 4 to vfio" shows the user has already thought through the NUMA topology and knows exactly which GPUs should go where.

The user does not specify which NUMA node gets which driver — that is left to the assistant to determine. Nor does the user specify what the VM should run or how to handle the model size change. This is a high-level directive, trusting the assistant to work out the details. The user is thinking in terms of resource allocation policy, not implementation mechanics.

Mistakes and Consequences

The most significant incorrect assumption is that the model fits on 4 GPUs. The Qwen3.5-397B NVFP4 model requires approximately 223 GB of GPU memory. With four RTX PRO 6000 GPUs providing 96 GB each (384 GB total), it might fit — but the assistant later replaces it with the smaller Qwen3.5-122B model, suggesting the 397B model either doesn't fit or doesn't perform well at TP=4. This forces a model swap that changes the entire character of the deployment.

The second major issue is the P2P DMA corruption. The SEV-SNP (Secure Encrypted Virtualization) configuration on the Proxmox host enables full IOMMU translation (amd_iommu=on), which breaks GPU-to-GPU direct memory access. Every P2P transfer produces corrupted data, causing NCCL to hang during init_torch_distributed. The assistant eventually fixes this with NCCL_P2P_DISABLE=1, forcing NCCL to use SHM (shared memory) transport instead. This is a performance hit — P2P DMA is faster than SHM — but it gets the system working.

The user also assumes the reconfiguration can happen quickly, but the assistant ends up spending many messages investigating IO_PAGE_FAULTs in dmesg, running CUDA P2P tests, fixing driver version mismatches (container had 565 userspace vs host 590 kernel), and exploring BIOS-level options for PCIe ACS, ARI, and IOMMU coverage settings to potentially re-enable P2P DMA.

Conclusion

Message [msg 6034] is a classic example of a high-level infrastructure decision that triggers deep technical work. In one sentence, the user redirects the entire session from performance optimization to hardware reconfiguration, from benchmarking to system administration. The message reveals the user's priorities (flexibility over peak performance), their understanding of the hardware topology (NUMA-aware GPU splitting), and their trust in the assistant to handle the implementation details. The subsequent session demonstrates just how much hidden complexity lies beneath that simple request — PCI bindings, IOMMU translation faults, NCCL transport selection, and model capacity planning all become problems that must be solved before the four-GPU LXC container can serve inference again.