Persistence by Design: Making a GPU Split Survive Reboot on Proxmox

Introduction

In the life of any production system, the gap between a one-shot manual fix and a durable, reboot-proof configuration is where engineering discipline separates itself from mere tinkering. Message [msg 6077] captures this exact transition. After successfully splitting eight NVIDIA RTX PRO 6000 Blackwell GPUs between an LXC container and a SEV-SNP VM on a Proxmox host—binding four to the nvidia driver and four to vfio-pci—the assistant turns its attention to the critical question of persistence. The message is a short but dense reasoning segment followed by a reconnaissance bash command, and it reveals the assistant's mental model of Linux device binding, the constraints imposed by identical hardware IDs, and the architectural trade-offs between different early-boot mechanisms.

The Message in Full

The subject message reads:

Now let's make the GPU split persistent across reboots. I need to ensure the NUMA 1 GPUs get bound to vfio-pci early in boot, before the nvidia driver claims them. The cleanest way is a udev rule or a modprobe config that binds specific PCI IDs to vfio-pci by address. Since all 8 GPUs have the same device ID (10de:2bb5), I can't use vfio-pci.ids= (that would grab all of them). I need a script that runs early and binds specific PCI addresses.

>

``bash ssh root@10.1.2.6 'ls /etc/systemd/system/gpu-split* 2>/dev/null; ls /etc/udev/rules.d/*gpu* 2>/dev/null; ls /etc/udev/rules.d/*vfio* 2>/dev/null; echo "---"; cat /etc/modprobe.d/vfio.conf 2>/dev/null || echo "no vfio.conf"' ``

>

Result: no vfio.conf

This is the entirety of the message—a reasoning preamble and a single shell command that probes the existing infrastructure.

Why This Message Matters

The message sits at a pivotal moment in the broader session. In the preceding messages ([msg 6046] through [msg 6076]), the assistant has:

  1. Stopped the SGLang server and LXC container
  2. Unbound four NUMA 1 GPUs from the nvidia driver and bound them to vfio-pci manually
  3. Updated the LXC config to only expose nvidia0-3 (the NUMA 0 GPUs)
  4. Created a new Proxmox PCI mapping (pro6000-vm) for the VM-side GPUs
  5. Started the LXC, updated the SGLang service to --tp 4, and verified the server works All of this was done live—the GPUs were moved at runtime while the host was fully operational. But the manual echo commands that wrote PCI addresses into /sys/bus/pci/drivers/vfio-pci/bind are ephemeral. A single reboot would undo all of it, and the nvidia driver would reclaim all eight GPUs on the next boot because it matches the device ID 10de:2bb5 globally. Message [msg 6077] is the moment the assistant recognizes this fragility and commits to solving it. The phrase "Now let's make the GPU split persistent across reboots" signals a shift from operational work (getting the system into the desired state) to architectural work (ensuring the system stays in that state).

The Reasoning Process

The assistant's reasoning is worth unpacking in detail because it demonstrates a nuanced understanding of Linux PCI driver binding.

Step 1: Define the goal. The NUMA 1 GPUs must be bound to vfio-pci "early in boot, before the nvidia driver claims them." This is the critical timing constraint. The nvidia driver will match any device with ID 10de:2bb5 during its initialization. To prevent this, the vfio-pci driver must claim those specific devices first, or the nvidia driver must be prevented from claiming them.

Step 2: Evaluate the simplest approach. The standard mechanism for binding devices to vfio-pci at boot is the vfio-pci.ids= kernel parameter (or options vfio-pci ids=10de:2bb5 in a modprobe config). This tells the vfio-pci driver to claim any device matching those vendor:device IDs. The assistant correctly identifies why this won't work: "Since all 8 GPUs have the same device ID (10de:2bb5), I can't use vfio-pci.ids= (that would grab all of them)." This is a crucial insight. The ids parameter is a global filter—it doesn't allow per-address selectivity. Using it would bind all eight GPUs to vfio-pci, which would break the LXC container that needs four of them under the nvidia driver.

Step 3: Identify the correct approach. The assistant concludes: "I need a script that runs early and binds specific PCI addresses." This means using a mechanism that can distinguish devices by their PCI bus address (e.g., 0000:81:00.0) rather than by device ID alone. The candidate mechanisms mentioned are "a udev rule or a modprobe config that binds specific PCI IDs to vfio-pci by address." A udev rule using KERNEL=="0000:81:00.0" or ATTR{devspec}=="..." can match on address, and a systemd service that runs before the nvidia driver loads can write specific PCI addresses to the vfio-pci bind interface.

Step 4: Reconnaissance. Before implementing any solution, the assistant checks what already exists on the system. The bash command probes three locations:

Assumptions and Knowledge Required

To understand this message, a reader needs significant background knowledge:

Output Knowledge Created

This message creates several pieces of knowledge:

  1. A confirmed clean slate: The system has no existing GPU-splitting infrastructure. No stale configs, no conflicting rules, no legacy services. This simplifies the design and reduces the risk of unexpected interactions.
  2. A design constraint documented: The identical device ID problem is explicitly identified and ruled out as a solution path. This prevents future wasted effort exploring the vfio-pci.ids= approach.
  3. A decision point reached: The assistant has committed to building a custom early-boot binding mechanism. The next message will presumably implement a systemd service or udev rule.
  4. A record of the reasoning: The message itself serves as documentation of why the simple approach won't work, which is valuable for anyone reviewing the system later.

The Thinking Process in Context

Looking at the broader conversation, this message is part of a pattern where the assistant alternates between doing and planning. The previous messages were action-heavy: unbinding GPUs, editing configs, starting containers, testing endpoints. Message [msg 6077] is a planning pause—a moment to step back and think about durability before moving forward.

The assistant's thinking is also notable for what it doesn't say. It doesn't consider the possibility of using PCI ACS (Access Control Services) or IOMMU groups to filter, because those are hardware-level mechanisms that don't control driver binding. It doesn't consider modifying the nvidia driver's probe routine, because that would require recompiling the driver. It stays within the realm of practical, configurable Linux kernel interfaces.

The bash command is also revealing. The assistant checks three specific paths, each targeting a different mechanism: systemd services (for a script-based approach), udev rules (for an event-driven approach), and modprobe config (for a module-parameter approach). This shows systematic thinking—the assistant is surveying the entire landscape of early-boot device binding mechanisms before choosing one.

Conclusion

Message [msg 6077] is a small but dense piece of engineering reasoning. In just a few sentences and a single shell command, the assistant identifies a persistence problem, evaluates the standard solution and finds it inadequate due to the identical-device-ID constraint, narrows the design space to address-specific binding, and surveys the existing infrastructure. It's a textbook example of how to approach a system configuration problem: understand the goal, know the tools, check what exists, and reason about constraints before writing code. The message sets the stage for the implementation that follows, and it captures a critical design decision that anyone maintaining this system will need to understand.