The Verification That Matters: Testing GPU Topology Splits Under Proxmox

In the middle of a complex infrastructure reconfiguration spanning multiple machines, GPU generations, and virtualization boundaries, message [msg 6082] stands out as a quiet but critical moment: a verification step. The assistant has just finished creating a systemd service and shell script to permanently split eight NVIDIA RTX PRO 6000 Blackwell GPUs between an LXC container and a future SEV-SNP virtual machine on a Proxmox host. Before moving on to the next task, the assistant pauses to test that the mechanism works. The message is brief—a single bash command and its output—but it encapsulates a discipline that separates fragile infrastructure from robust production deployments: the habit of verifying every automated change before considering it done.

The Broader Context: Why Split GPUs?

To understand why this message matters, one must appreciate the infrastructure challenge that preceded it. The Proxmox host (kpro6) has eight RTX PRO 6000 Blackwell GPUs physically installed, spanning two NUMA domains. Four GPUs (PCI addresses 01:00.0, 11:00.0, 61:00.0, 71:00.0) live on NUMA 0, and four (81:00.0, 91:00.0, e1:00.0, f1:00.0) on NUMA 1. Earlier in the session, the assistant had been running a single SGLang server with tensor parallelism across all eight GPUs (TP=8), serving the massive Qwen3.5-397B-A17B-NVFP4 model. But a new requirement emerged: the NUMA 1 GPUs needed to be passed through to a confidential VM using AMD SEV-SNP, while the NUMA 0 GPUs would continue serving the LXC-based SGLang deployment.

This split is nontrivial because all eight GPUs share the same PCI device ID (10de:2bb5). You cannot simply use the kernel's vfio-pci.ids= parameter to bind them, because that would grab all eight. Instead, the assistant had to create a custom early-boot mechanism that unbinds specific PCI addresses from the nvidia driver and rebinds them to vfio-pci. This required a shell script (/usr/local/bin/gpu-vfio-split.sh) and a systemd oneshot service (gpu-vfio-split.service) that runs before nvidia-persistenced.service, ensuring the nvidia driver never claims the NUMA 1 GPUs in the first place.

The Verification Mindset

Message [msg 6082] is the moment the assistant puts this mechanism to the test. The phrasing is deliberate: "Let me verify the script is correct and the service works by doing a dry run." The assistant explicitly notes that the GPUs are already on vfio-pci from an earlier manual binding step, so the service should be a no-op. This is not a test of the binding logic per se—that was already validated when the assistant manually wrote to the PCI driver sysfs files. Rather, this is a test of the automation: will the script parse correctly? Will systemd execute it without errors? Will the quoting survive shell and systemd parsing? Will the service start and exit cleanly?

This distinction matters. In infrastructure automation, there are two failure modes: the mechanism failing to do its job (e.g., the unbind/bind logic being wrong) and the automation wrapper failing to invoke the mechanism correctly (e.g., a syntax error in the systemd unit file, a path that doesn't exist at boot time, a missing dependency). The assistant had already validated the mechanism manually. Now it validates the automation.

Technical Deep Dive: The vfio-pci Split Service

The script that the service invokes performs a careful dance with the Linux PCI driver model. For each of the four NUMA 1 PCI addresses, it:

  1. Checks the current driver by reading the symlink at /sys/bus/pci/devices/$dev/driver.
  2. If the device is already bound to vfio-pci, it skips it (the "already on vfio-pci" path, which is what happened in the dry run).
  3. Otherwise, it unbinds from the current driver by writing the PCI address to the driver's unbind file.
  4. Registers the vendor/device ID pair (10de 2bb5) with vfio-pci via new_id.
  5. Binds the device to vfio-pci via the bind file.
  6. Verifies the new driver and reports success or failure. The systemd service is configured as Type=oneshot with RemainAfterExit=yes, meaning systemd considers the service "active" after it runs once and won't restart it. It runs before nvidia-persistenced.service (via Before=), ensuring the nvidia driver never claims the NUMA 1 GPUs. The DefaultDependencies=no flag prevents systemd from inserting its standard ordering constraints, which might otherwise cause the service to run too late. This approach is more robust than a udev rule for this particular case. Udev rules fire when devices appear, but by the time the nvidia driver has claimed a GPU, unbinding and rebinding requires the same sysfs dance. A systemd service that runs early in boot, before the nvidia driver initializes, is a cleaner solution—provided it actually works.

The Dry Run and Its Significance

The actual test in [msg 6082] is straightforward:

ssh root@10.1.2.6 'systemctl start gpu-vfio-split.service && systemctl status gpu-vfio-split.service'

The output shows the service started successfully, ran the script (process 625980, exit code 0, 11ms CPU time), and exited cleanly. The status line Active: active (exited) is exactly what a Type=oneshot service should show after successful completion. The timestamp (Sat 2026-03-07 17:08:34 CET) confirms it ran just moments ago.

But what does this output not tell us? It doesn't show the script's stdout or stderr. The assistant doesn't see the "already on vfio-pci" messages that the script would have printed for each device. The verification is somewhat implicit: if the service exited with status 0 and the GPUs remain bound correctly (which the assistant could check separately), the automation is working. The assistant trusts that the script's logic is correct because it was validated earlier, and the service's clean exit confirms that systemd can invoke it without error.

This is a reasonable approach. The alternative—parsing the script's output to confirm each device was skipped—would add complexity without much benefit. The critical failure modes (service fails to start, script crashes, wrong GPUs get bound) would all produce visible errors in the status output.

Assumptions and Potential Pitfalls

The verification in [msg 6082] rests on several assumptions:

  1. The GPUs are already on vfio-pci. This is true now, but what about after a reboot? The service is designed to handle both cases—it checks the current driver and acts accordingly. The dry run only validates the "already on vfio-pci" path, not the unbind/rebind path. A full test would require rebooting or manually unbinding a GPU and running the service again.
  2. The PCI addresses are stable. PCI addresses can change with hardware reconfiguration, BIOS updates, or kernel changes. The addresses (81:00.0, 91:00.0, e1:00.0, f1:00.0) are hardcoded in the script. If the topology changes, the script will silently fail to bind the correct devices.
  3. The vfio-pci module is available at boot time. The script calls modprobe vfio-pci, which should work if the module is installed. But if the kernel doesn't have vfio support compiled in, the service will fail.
  4. Systemd ordering is correct. The service runs before nvidia-persistenced.service, but what if the nvidia driver is loaded even earlier (e.g., via initramfs or built-in kernel module)? The Before= directive only affects systemd-ordered services; if the nvidia driver is loaded by the kernel before systemd starts, the unbind/rebind approach still works (the script handles unbinding from the current driver), but it's less clean. None of these assumptions are unreasonable, and the assistant's verification strategy is appropriate for the context. The dry run confirms that the automation layer (systemd + script) is syntactically correct and can be invoked. The deeper validation—that the split survives a reboot—would require a full system restart, which is impractical in the middle of a development session.

Conclusion

Message [msg 6082] is a small but instructive moment in a larger infrastructure story. It demonstrates a pattern that experienced infrastructure engineers know well: create the automation, then verify the automation, then move on. The verification doesn't need to be exhaustive—it needs to catch the most likely failure modes and confirm that the system is in the expected state. In this case, the expected state is a systemd service that exits cleanly and leaves the GPU binding unchanged.

The broader lesson is about trust in automation. Every script, service, and configuration file is a hypothesis about how the system should behave. Verification is the experiment that tests that hypothesis. The assistant's dry run in [msg 6082] is a lightweight experiment—quick to run, easy to interpret, and sufficient to confirm that the automation is wired correctly. The deeper experiments (reboot tests, failure recovery) can wait until the system is deployed and the stakes are higher.

In the fast-paced world of ML infrastructure deployment, where models are measured in hundreds of billions of parameters and GPUs in thousands of dollars each, this kind of disciplined verification is what separates a reliable production service from a fragile prototype. Message [msg 6082] is a reminder that the most important infrastructure code is often the code that checks whether the other code works.