Binding the Unbindable: Making GPU Passthrough Persistent on Proxmox with Systemd

Introduction

In the complex dance of virtualizing high-end GPUs for machine learning workloads, one of the most delicate operations is ensuring that the right GPUs end up bound to the right drivers at boot time. The subject message — <msg id=6081> from an opencode coding session — captures the final act of a multi-step reconfiguration: creating and enabling a systemd service that persistently binds four NVIDIA RTX PRO 6000 Blackwell GPUs to the vfio-pci driver for VM passthrough on a Proxmox hypervisor host. This seemingly small step — a service file written and enabled — represents the culmination of careful reasoning about boot ordering, driver binding mechanics, and the constraints of homogeneous GPU hardware. To understand why this message exists, we must trace the decisions that led to it and the infrastructure it secures.

Context: The GPU Topology Problem

The session's broader narrative is one of resource partitioning. The operator manages a Proxmox host (kpro6) equipped with eight NVIDIA RTX PRO 6000 Blackwell GPUs, physically distributed across two NUMA domains. Four GPUs reside on NUMA 0 (PCI addresses 01:00.0, 11:00.0, 61:00.0, 71:00.0), and four on NUMA 1 (81:00.0, 91:00.0, e1:00.0, f1:00.0). The goal is to split these eight GPUs between two consumers: an LXC container running SGLang (which gets the four NUMA 0 GPUs, bound to the NVIDIA driver), and a future SEV-SNP VM (which gets the four NUMA 1 GPUs, bound to vfio-pci for direct passthrough).

Earlier in the segment, the assistant had already performed the manual split: stopping the LXC container, unbinding the NUMA 1 GPUs from the NVIDIA driver, and binding them to vfio-pci via sysfs. But manual binding is ephemeral — a reboot would cause the NVIDIA driver to reclaim all eight GPUs (since they share the same PCI device ID 10de:2bb5), breaking the topology. The subject message addresses this persistence problem.

The Message: What Actually Happens

The message contains two operations, both executed via SSH on the Proxmox host at 10.1.2.6. First, the assistant writes a systemd unit file to /etc/systemd/system/gpu-vfio-split.service:

[Unit]
Description=Bind NUMA1 GPUs to vfio-pci for VM passthrough
Before=nvidia-persistenced.service
After=systemd-modules-load.service
DefaultDependencies=no

[Service]
Type=oneshot
RemainAfterExit=yes
ExecStart=/usr/local/bin/gpu-vfio-split.sh

[Install]
WantedBy=multi-user.target

Second, it runs systemctl daemon-reload && systemctl enable gpu-vfio-split.service, which creates a symlink at /etc/systemd/system/multi-user.target.wants/gpu-vfio-split.service and confirms with "enabled."

The service itself is minimal — it defers all the real work to a shell script (/usr/local/bin/gpu-vfio-split.sh) that was written in the preceding message ([msg 6080]). That script iterates over the four NUMA 1 PCI addresses, checks whether each is already bound to vfio-pci, unbinds it from its current driver if necessary, registers the device ID with vfio-pci, and binds it. The script uses modprobe vfio-pci to ensure the driver is loaded, and includes careful error handling: it checks whether the unbind target exists and whether the new driver took effect.

Why This Approach? The Reasoning Behind the Design

The assistant's choice to use a systemd service rather than alternatives like udev rules or kernel module parameters reveals several layers of reasoning visible in the preceding messages.

The homogeneous device ID problem. All eight GPUs share the same PCI vendor/device ID (10de:2bb5). This means a simple vfio-pci.ids=10de:2bb5 kernel parameter would bind all eight GPUs to vfio-pci, not just the NUMA 1 set. The assistant explicitly considered and rejected this approach in [msg 6077], noting "I can't use vfio-pci.ids= (that would grab all of them)." This constraint forces a more surgical approach: binding by PCI address rather than by device ID.

Why not udev? The assistant considered udev rules but judged them less reliable for this use case, noting in [msg 6077] that "a script that runs early and binds specific PCI addresses" is cleaner. The reasoning is sound: udev rules fire when devices appear, but the unbind/rebind dance (unbinding from the NVIDIA driver, then binding to vfio-pci) requires careful sequencing that a script handles more deterministically. A udev rule might fire too early (before vfio-pci is loaded) or race with the NVIDIA driver's own probe.

Boot ordering is critical. The Before=nvidia-persistenced.service directive is the key insight. NVIDIA's nvidia-persistenced service initializes the GPU state early in boot. If the service runs after the NVIDIA driver has claimed the NUMA 1 GPUs, the unbind operation might fail (the driver might not release the device gracefully) or cause a race condition. By placing the service before nvidia-persistenced, the assistant ensures the NUMA 1 GPUs are already claimed by vfio-pci when the NVIDIA driver probes them. The After=systemd-modules-load.service ensures the vfio-pci kernel module is loaded before the script attempts to use it.

Why DefaultDependencies=no? This directive prevents systemd from adding automatic ordering dependencies (like After=sysinit.target), giving the service maximum flexibility to run early. Combined with Before=nvidia-persistenced.service, it slots the service into the exact right point in the boot sequence: after kernel modules are loaded but before the NVIDIA stack initializes.

Oneshot with RemainAfterExit. The Type=oneshot means systemd considers the service "active" after the script runs once. RemainAfterExit=yes means it stays marked as active even after the process exits. This is important because other services that depend on the GPU split being complete can use After=gpu-vfio-split.service to wait for it. It also prevents systemd from trying to restart the service on failure (since oneshot services don't restart by default).

Assumptions Embedded in the Design

Several assumptions underpin this message, some explicit and some implicit.

Assumption: The script exists and is correct. The service file references /usr/local/bin/gpu-vfio-split.sh, which was written in the previous message. The assistant assumes this script is already in place, executable, and functionally correct. If the script had a bug (e.g., wrong PCI addresses, incorrect unbind logic), the service would silently fail to bind the GPUs, and the VM would not see them.

Assumption: The boot order is deterministic. The assistant assumes that Before=nvidia-persistenced.service will reliably cause the service to run before the NVIDIA driver claims devices. This depends on systemd's dependency resolution and the assumption that nvidia-persistenced does not start in parallel with this service. If the NVIDIA driver is built into the kernel or loaded via initramfs (before systemd even starts), this ordering guarantee might not hold. On modern Ubuntu with modular NVIDIA drivers loaded via nvidia-persistenced, this is a reasonable assumption, but it is not universally true.

Assumption: The PCI addresses are stable. The four PCI addresses (0000:81:00.0, 0000:91:00.0, 0000:e1:00.0, 0000:f1:00.0) are hardcoded in both the service and the script. This assumes the PCI topology does not change — no hardware reconfiguration, no BIOS changes that reorder slots, no hotplug events. On a server with fixed GPU installation, this is stable, but it is a brittleness worth noting.

Assumption: vfio-pci supports the device. The script registers 10de 2bb5 with vfio-pci via new_id. This assumes the vfio-pci driver can handle the Blackwell RTX PRO 6000. For most modern GPUs, vfio-pci works as a generic PCI passthrough driver, but it does not support GPU-specific features like framebuffer access or CUDA — it simply passes the device through to the VM. The assumption is that the VM will have its own NVIDIA driver to manage the GPU.

Assumption: No other driver claims the device first. The script's unbind step assumes the device is currently bound to a driver (likely nvidia). If some other driver (like nouveau) claims it first, the unbind might still work, but the script does not handle this case explicitly. On a system with nvidia-persistenced and nvidia loaded, this is the expected state.

Potential Mistakes and Risks

While the design is sound, there are subtle risks worth examining.

The race with nvidia-persistenced. The Before=nvidia-persistenced.service directive is not a hard guarantee. If nvidia-persistenced has a After=local-fs.target or similar dependency that systemd resolves before this service, the ordering might not be strict. Systemd's transaction ordering is generally reliable, but the assistant did not verify the exact dependency chain of nvidia-persistenced.service. A more robust approach might use Before=sys-devices-virtual-misc-nvidia.device or bind directly into the kernel's driver probe sequence via a custom initramfs hook.

The new_id race. The script writes to /sys/bus/pci/drivers/vfio-pci/new_id to register the device ID. This is a one-shot operation — once the ID is registered, vfio-pci will automatically bind any future device with that ID that appears on the bus. If another GPU with the same ID appears later (e.g., after a hotplug), it would also be bound to vfio-pci, which might not be intended. The script does not clean up the new_id registration after binding the specific devices.

No failure notification. The service is Type=oneshot with no OnFailure= directive. If the script fails (e.g., a PCI address is wrong, vfio-pci module is missing), systemd will mark the service as failed, but there is no alerting. The operator would only discover the issue when the VM fails to see its GPUs.

The script path is hardcoded. If the script is moved or deleted, the service will fail silently. A more defensive approach might embed the binding logic directly in the service's ExecStart (avoiding the script dependency) or add a ConditionFileNotEmpty= or ExecStartPre= check.

Knowledge Required to Understand This Message

To fully grasp the significance of this message, one needs knowledge spanning several domains:

Proxmox VE internals: Understanding that Proxmox uses a cluster filesystem (/etc/pve/) for VM and container configs, that PCI mappings are stored in /etc/pve/mapping/pci.cfg, and that GPU passthrough requires binding to vfio-pci rather than the native NVIDIA driver.

Linux PCI driver model: Knowing that PCI devices can be unbound from one driver and rebound to another via sysfs (/sys/bus/pci/drivers/.../unbind and bind), and that vfio-pci uses the new_id interface to accept devices by vendor/device ID.

Systemd boot ordering: Understanding the significance of Before=, After=, DefaultDependencies=no, and WantedBy=multi-user.target in the context of early boot services that must run before driver initialization.

NVIDIA driver architecture: Knowing that nvidia-persistenced is the service that initializes GPU state, and that it runs relatively early in boot, making it the critical deadline for this binding service.

NUMA topology awareness: Understanding that the GPUs are split by NUMA domain, and that the PCI addresses correspond to physical slots on different NUMA nodes. This matters for VM performance — a VM pinned to NUMA 1 will have optimal memory access to GPUs on the same NUMA node.

Knowledge Created by This Message

The message produces several concrete outputs:

  1. A systemd unit file at /etc/systemd/system/gpu-vfio-split.service that defines the boot-time GPU binding service.
  2. A systemd symlink at /etc/systemd/system/multi-user.target.wants/gpu-vfio-split.service, which enables the service to start at boot.
  3. Documentation of intent in the service description: "Bind NUMA1 GPUs to vfio-pci for VM passthrough" — this is self-documenting infrastructure that tells future administrators what the service does and why. More abstractly, the message creates reproducibility. Before this service, the GPU split was a manual procedure requiring SSH access and sysfs manipulation after every reboot. After this service, the split happens automatically at boot, making the configuration resilient to power cycles, kernel updates, and maintenance reboots. This is the difference between a one-time hack and a production deployment.

The Thinking Process: What the Reasoning Reveals

The assistant's reasoning is most visible in the messages leading up to [msg 6081]. In [msg 6077], the assistant explicitly evaluates alternatives:

"Now let's make the GPU split persistent across reboots. I need to ensure the NUMA 1 GPUs get bound to vfio-pci early in boot, before the nvidia driver claims them. The cleanest way is a udev rule or a modprobe config that binds specific PCI IDs to vfio-pci by address. Since all 8 GPUs have the same device ID (10de:2bb5), I can't use vfio-pci.ids= (that would grab all of them). I need a script that runs early and binds specific PCI addresses."

This is a textbook example of constraint-driven design. The assistant identifies the requirement (persistent binding), evaluates the standard solution (kernel parameter vfio-pci.ids=), discovers a constraint (homogeneous device IDs make this impossible), and pivots to a more surgical approach (address-specific binding via a script).

In [msg 6078], the assistant initially tries to inline the binding logic in the service's ExecStart using a complex bash one-liner, but then realizes the quoting is problematic:

"Hmm, the nested quoting might be mangled."

This leads to the decision in [msg 6080] to use a separate script file instead, which is cleaner and avoids the quoting nightmare of embedding bash in systemd's ExecStart. This is a pragmatic engineering judgment — the script file is more maintainable, testable, and debuggable than an inline command.

The final message ([msg 6081]) then ties it together: write the clean service file that calls the script, and enable it. The service file itself is notably sparse — it contains no logic, only orchestration. This separation of concerns (the service orchestrates, the script executes) is good Unix practice.

Conclusion

Message [msg 6081] appears, on its surface, to be a routine system administration task: write a service file and enable it. But beneath this simplicity lies a carefully reasoned solution to a non-trivial infrastructure problem. The homogeneous device IDs of the Blackwell GPUs rule out the easy path. The early-boot timing constraint requires precise systemd dependency ordering. The need for persistence across reboots demands automation rather than manual intervention. The assistant navigates these constraints by decomposing the problem into a systemd orchestration layer and a script execution layer, using Before=nvidia-persistenced.service to win the race against the NVIDIA driver, and hardcoding PCI addresses to select only the NUMA 1 GPUs. The result is a robust, self-documenting configuration that transforms a fragile manual procedure into a reliable boot-time automation — a small but essential step in building a production-grade GPU virtualization infrastructure.