The Systemd Service That Splits GPUs: Engineering Persistent PCI Passthrough for Blackwell

In the sprawling infrastructure of a high-performance machine learning deployment, few moments are as deceptively simple as the creation of a systemd service file. Message [msg 6078] appears, at first glance, to be a routine administrative task: the assistant writes a oneshot systemd unit that binds four NVIDIA RTX PRO 6000 Blackwell GPUs to the vfio-pci driver at boot. But beneath this seemingly mundane operation lies a carefully reasoned architectural decision, born from the specific constraints of PCIe device driver management, the boot order of GPU drivers, and the need to split a pool of eight identical GPUs between two virtualized environments running on the same Proxmox host.

The Problem: Identical Hardware, Different Destinies

The context leading up to this message reveals a critical infrastructure challenge. The assistant had been working on a Proxmox VE host (kpro6) equipped with eight NVIDIA RTX PRO 6000 Blackwell GPUs. These GPUs are physically partitioned across two NUMA domains: NUMA 0 contains four GPUs (PCI addresses 01:00.0, 11:00.0, 61:00.0, 71:00.0), and NUMA 1 contains the other four (81:00.0, 91:00.0, e1:00.0, f1:00.0). The goal was to allocate NUMA 0's GPUs to an LXC container running an SGLang inference server, while dedicating NUMA 1's GPUs to a SEV-SNP VM for passthrough.

The assistant had already performed the manual unbind/rebind dance in [msg 6050], moving the NUMA 1 GPUs from the nvidia driver to vfio-pci. The LXC config had been updated to only mount nvidia0-3 ([msg 6055]), and a new PCI mapping pro6000-vm had been created in Proxmox's PCI configuration ([msg 6058]). But all of this was ephemeral — a reboot would undo everything. The vfio-pci binding would be lost, the nvidia driver would reclaim all eight GPUs, and the LXC container would suddenly see devices it shouldn't, while the VM would find no GPUs available for passthrough.

This is the problem that message [msg 6078] exists to solve: making the GPU split persistent across system reboots.

The Decision: Why Systemd Over Udev

The assistant's first action in the message is to probe for existing infrastructure: checking for gpu-split or vfio-related files in both /etc/systemd/system/ and /etc/udev/rules.d/, and checking /etc/modprobe.d/vfio.conf. Finding nothing, the assistant must choose an approach.

The reasoning, visible in the message's opening statement, is explicit: "This is more reliable than udev for this case since we need to handle the unbind/rebind dance." This is a critical design judgment. A udev rule would typically match on device ID or PCI vendor/device pairs and automatically bind a driver when the device is discovered. However, the challenge here is that all eight GPUs share the same PCI device ID (10de:2bb5). A udev rule or modprobe configuration using vfio-pci.ids=10de:2bb5 would indiscriminately bind all eight GPUs to vfio-pci, defeating the purpose of the split. The assistant needs selective binding based on PCI bus address, not device ID.

A udev rule could theoretically match on PCI path (KERNELS=="0000:81:00.0"), but the assistant judges the systemd approach more reliable. This is a reasonable assessment: udev rules for driver unbinding and rebinding can be fragile, especially when dealing with the timing of driver probing during boot. The nvidia driver may claim devices before udev rules have a chance to intervene. A systemd service with explicit Before=nvidia-persistenced.service ordering gives stronger control over the sequence of operations.

The Architecture of the Service

The service file the assistant creates is a Type=oneshot unit with RemainAfterExit=yes, meaning it runs once at boot and is considered active afterwards. The key design elements are:

Dependency ordering: The service declares Before=nvidia-persistenced.service and After=systemd-modules-load.service. This positions it in the boot sequence after kernel modules are loaded (so vfio-pci is available) but before the NVIDIA persistence daemon initializes, which is the point at which the nvidia driver would claim all compatible devices. This ordering is crucial: if the service ran too late, the nvidia driver would already own the NUMA 1 GPUs, and unbinding might race with GPU initialization.

The unbind/rebind logic: The ExecStart command is a bash one-liner that iterates over the four PCI addresses. For each device, it:

  1. Checks if the device is already on vfio-pci (idempotency guard)
  2. If bound to another driver, unbinds it by writing the PCI address to the driver's unbind sysfs file
  3. Writes the vendor:device ID pair (10de 2bb5) to vfio-pci/new_id to register the device as supported
  4. Writes the PCI address to vfio-pci/bind to attach the device The new_id step is important: it tells the vfio-pci driver that it should handle this vendor/device combination, even though the driver wasn't loaded with that ID in its static table. Without this, the bind operation would fail because vfio-pci wouldn't recognize the device.

Assumptions Embedded in the Design

The assistant makes several assumptions in this message, most of which are well-founded but worth examining:

Assumption 1: The PCI addresses are stable across reboots. On modern systems with PCIe topology that doesn't change, PCI bus addresses are deterministic. The assistant is relying on the fact that the four NUMA 1 GPUs will always appear at 81:00.0, 91:00.0, e1:00.0, and f1:00.0. This is generally safe for server hardware with fixed PCIe slot assignments, but could break if GPUs are physically moved to different slots or if the BIOS reorders PCI enumeration.

Assumption 2: The nvidia driver will claim the devices after the service runs. The Before=nvidia-persistenced.service ordering assumes that the nvidia driver's initialization happens after this service completes. If the nvidia driver module is loaded earlier (e.g., via initramfs or built-in kernel module), it might claim the NUMA 1 GPUs before the service has a chance to intervene. In that case, the unbind step would still work (writing to unbind removes the driver), but there's a risk of race conditions.

Assumption 3: vfio-pci is already loaded. The service runs After=systemd-modules-load.service, which loads modules specified in /etc/modules-load.d/. The assistant assumes that vfio-pci (and its dependencies like vfio_pci_core, vfio_iommu_type1) are either built into the kernel or loaded by this point. If they aren't, the new_id and bind operations would fail.

Assumption 4: The unbind/rebind dance is safe at boot time. Unbinding a GPU from its driver and rebinding it to vfio-pci is a hot-plug-like operation. The assistant assumes that doing this early in boot, before any GPU workloads are running, is safe and won't leave the device in an inconsistent state. This is generally correct, but some GPU firmware may not handle driver unbinding gracefully even at boot.

Input Knowledge Required

To fully understand this message, one needs:

  1. Understanding of PCI device driver binding in Linux: How the driver symlink in /sys/bus/pci/devices/ works, how unbind and bind sysfs files control driver attachment, and how new_id registers new device IDs with a driver.
  2. Knowledge of vfio-pci and GPU passthrough: That vfio-pci is a generic driver that presents PCI devices to userspace for VFIO-based passthrough (used by QEMU/KVM), and that it must be bound to a device before the VM can claim it.
  3. Systemd service unit semantics: Understanding Type=oneshot, RemainAfterExit=yes, Before/After ordering directives, and how DefaultDependencies=no affects the service's position in the boot sequence.
  4. The Proxmox context: That the host uses Proxmox VE, that LXC containers and QEMU VMs coexist, and that PCI passthrough requires the device to be on vfio-pci (or similar) rather than the native nvidia driver.
  5. The hardware topology: That the eight GPUs are split across two NUMA nodes, and that the assistant has already identified which PCI addresses belong to which NUMA domain.

Output Knowledge Created

This message produces a tangible artifact: the file /etc/systemd/system/gpu-vfio-split.service on the Proxmox host. This service encapsulates the GPU split logic in a declarative, persistent form. The output knowledge includes:

The Thinking Process Visible in the Message

The assistant's reasoning unfolds in several stages within this single message:

First, there's the reconnaissance phase: checking for existing infrastructure. The ls commands probe three possible locations where GPU-splitting logic might already exist — systemd services, udev rules, and modprobe configuration. Finding nothing, the assistant knows it must create from scratch.

Second, the architectural decision: systemd over udev. The assistant articulates the rationale explicitly — the "unbind/rebind dance" is more reliably handled in a systemd service. This shows awareness of the limitations of udev for multi-step operations that involve driver unbinding.

Third, the service design: the assistant chooses Type=oneshot with RemainAfterExit=yes, which is the standard pattern for boot-time initialization tasks that don't run as daemons. The Before=nvidia-persistenced.service ordering is the critical insight — it positions the GPU split before NVIDIA's initialization, ensuring the nvidia driver never claims the NUMA 1 GPUs.

Fourth, the implementation: the bash one-liner is carefully constructed with error handling (checking current driver, reporting success/failure per device). The use of new_id before bind shows understanding of the vfio-pci driver registration mechanism.

Finally, the verification: the assistant runs the creation command and gets back "service created", confirming the file was written successfully.

Was This the Right Approach?

The systemd service approach is sound, but it's worth considering alternatives. A udev rule with ACTION=="add", KERNELS=="0000:81:00.0", RUN+="/bin/sh -c '...'" could achieve the same result, but would be harder to debug and more dependent on udev event ordering. A kernel command-line parameter or initramfs hook could bind devices even earlier, but would require rebuilding the initramfs. The systemd approach hits the right balance of reliability, maintainability, and simplicity.

One potential improvement the assistant doesn't implement is adding a ConditionPathExists or checking that the PCI devices actually exist before attempting the bind. If a GPU is physically removed or fails, the service would log errors but still complete. This is a minor edge case, and the current implementation handles it gracefully enough (the loop simply continues to the next device).

Conclusion

Message [msg 6078] is a masterclass in translating an ephemeral manual procedure into a persistent, boot-safe automation. The assistant doesn't just write a systemd service — it reasons about boot ordering, driver binding mechanics, and the specific constraints of identical GPUs that must be treated differently based on their physical location. The resulting gpu-vfio-split.service is a clean, self-documenting piece of infrastructure that encodes deep knowledge about Linux PCI device management, GPU passthrough, and systemd service design. It's the kind of artifact that, once created, quietly ensures that reboots don't break complex hardware configurations — and that's the mark of truly robust systems engineering.