Diagnosing the Great GPU Disappearance: PCI BAR Allocation Failure in a Proxmox VM

In the high-stakes world of large-scale GPU inference, few things are as alarming as booting a freshly reconfigured virtual machine only to discover that six of your eight top-of-the-line NVIDIA RTX PRO 6000 Blackwell GPUs have simply vanished. This is precisely the scenario that unfolds in message [msg 332], a masterclass in diagnostic reasoning from an AI assistant helping a user deploy a GLM-5-NVFP4 model across eight GPUs in a Proxmox virtualized environment.

The message arrives at a critical juncture in the conversation. The user and assistant have been engaged in a multi-session effort to set up an ML inference server on Ubuntu 24.04, navigating everything from CUDA toolkit installation and flash-attn build issues to the deployment of SGLang with tensor parallelism. The most recent challenge has been enabling Peer-to-Peer (P2P) DMA communication between GPUs inside the VM—a prerequisite for achieving acceptable performance in tightly-coupled inference workloads. To this end, the user migrated the VM from the legacy i440FX chipset to the modern Q35 chipset with PCIe passthrough enabled (pcie=1). But after rebooting, the results were catastrophic: only two GPUs appeared in nvidia-smi.

This article examines the assistant's response in depth, exploring the diagnostic process, the technical reasoning, the assumptions made, and the knowledge required to understand this message. It is a story about PCI address spaces, memory-mapped I/O (MMIO), and the sometimes fragile relationship between virtualized hardware and real devices.

The Context: A High-Performance GPU VM on Proxmox

To appreciate the gravity of the situation, we must understand the environment. The user is running a Proxmox VE hypervisor on an ASUS ESC8000A-E13 motherboard with dual AMD EPYC processors. Eight NVIDIA RTX PRO 6000 Blackwell GPUs—each with 96GB of VRAM—are passed through to a VM named llm-one using VFIO (Virtual Function I/O) passthrough. This is an extraordinarily demanding configuration: each GPU exposes a 128GB Base Address Register (BAR2) that maps its VRAM into the PCI address space, and with eight GPUs, the total BAR2 requirement alone is 1TB.

The VM was originally configured with the i440FX chipset, which is the default for many Proxmox deployments. However, the i440FX chipset has limitations when it comes to PCIe topology and address space allocation. To enable proper PCIe device passthrough and potentially improve P2P DMA support, the assistant recommended switching to the Q35 chipset with pcie=1. The user applied this change and rebooted—and that's when disaster struck.

The Crisis: Six GPUs Missing

The user's preceding message ([msg 331]) contains the raw diagnostic data: the full dmesg output from the VM's boot and the output of nvidia-smi. The picture is grim. Only two GPUs—at PCI addresses 04:00.0 and 08:00.0—are detected. The other six, at addresses 01:00.0, 02:00.0, 03:00.0, 05:00.0, 06:00.0, and 07:00.0, have failed to initialize. The NVIDIA driver reports:

NVRM: This PCI I/O region assigned to your NVIDIA device is invalid:
NVRM: BAR2 is 0M @ 0x0 (PCI:0000:01:00.0)
nvidia: probe of 0000:01:00.0 failed with error -1

The user also asks a specific question: "Do I want vIOMMU enabled in Machine setting?"—a reasonable question given that the entire effort is about enabling P2P DMA, and IOMMU is central to that capability.

The Diagnostic: Reading the PCI Address Space Tea Leaves

The assistant's response in [msg 332] is a textbook example of systematic diagnostic reasoning. It opens with a clear, declarative statement: "This is a critical finding." The assistant immediately identifies the core problem—only 2 of 8 GPUs detected—and quotes the exact error messages from the dmesg output.

The diagnostic proceeds in several layers:

Layer 1: Identifying the Error Pattern

The assistant notes that the failing GPUs span addresses 01:00.0 through 07:00.0 (with 04:00.0 and 08:00.0 being the exceptions). This pattern is not random—it suggests a systematic allocation failure rather than a device-specific issue. The NVIDIA driver is receiving a BAR2 value of 0, meaning the PCI subsystem failed to assign a memory address for the GPU's VRAM mapping.

Layer 2: Tracing to the Root Cause

The assistant digs into the earlier part of the dmesg output, where the Linux kernel's PCI resource allocator reports its failures:

pci 0000:01:00.0: BAR 2 [mem size 0x2000000000 64bit pref]: can't assign; no space
pci 0000:01:00.0: BAR 2 [mem 0x390000000000-0x391fffffffff 64bit pref]: failed to assign

The key phrase is "can't assign; no space." Each GPU requires a 128GB BAR2 (0x2000000000 in hexadecimal = 128GB). The Q35 chipset's virtual PCI root bus provides a 64-bit MMIO window of [mem 0x380000000000-0x395fffffffff], which is approximately 1.5TB. With eight GPUs each needing roughly 160GB of bridge window space (128GB for BAR2 plus additional space for BAR0 and BAR4), the total requirement is about 1.28TB—which should theoretically fit within the 1.5TB window.

Layer 3: The Bridge Window Puzzle

This is where the assistant's reasoning becomes particularly impressive. It enumerates all eight bridge windows and their assigned address ranges:

Layer 4: The Critical Clue

The assistant spots the smoking gun in the dmesg output:

pci_bus 0000:00: Some PCI device resources are unassigned, try booting with pci=realloc

This is the Linux kernel itself suggesting a solution. The pci=realloc kernel parameter tells the PCI subsystem to attempt a second pass at resource allocation, potentially reassigning bridge windows and BARs to resolve conflicts. The assistant correctly identifies this as the recommended fix.

The Reasoning Process: A Window into the Assistant's Thinking

What makes this message particularly valuable for study is the visible reasoning process. The assistant does not simply present a conclusion—it walks through its analytical steps, including moments of uncertainty and self-correction.

The vIOMMU Detour

The user's question about vIOMMU is addressed directly and decisively: "Do not enable vIOMMU right now." The assistant explains that vIOMMU (virtual IOMMU inside the guest) is conceptually distinct from the host IOMMU and would add overhead without helping P2P. This is an important clarification because the terminology can be confusing—there are two different IOMMU layers at play (host and guest), and enabling the wrong one could degrade performance.

The QEMU Parameter Exploration

The assistant initially considers a host-side fix using QEMU's mch.above_4g_mem_size parameter to increase the MMIO window. It proposes:

qm set 128 -args '-global mch.above_4g_mem_size=2T'

But then the assistant catches itself: "Wait — actually, the better approach for Proxmox is to use the pci_addr_above_4g or ensure the MMIO windows are large enough. Let me think about the correct QEMU parameter..."

This self-correction is revealing. The assistant is reasoning in real-time, considering alternatives, and refining its understanding. It ultimately concludes that the guest-side fix (pci=realloc) is the more appropriate first step, with the host-side QEMU parameter as a fallback.

The Arithmetic Verification

The assistant performs a detailed calculation of the address space requirements, listing each bridge window explicitly. This serves multiple purposes: it demonstrates due diligence, it catches potential errors in the assistant's own understanding, and it provides the user with a clear picture of why the problem occurred. The arithmetic shows that the total allocation should fit—which narrows the problem to a resource allocation algorithm issue rather than a fundamental capacity shortfall.

Assumptions and Potential Pitfalls

The assistant's analysis rests on several assumptions that deserve examination:

Assumption 1: The Guest Kernel's Allocator Is the Problem

The assistant assumes that the Linux kernel's PCI resource allocator is failing to reassign BARs within already-allocated bridge windows. This is a reasonable inference given the dmesg message suggesting pci=realloc, but it's not the only possibility. The issue could also be a firmware (SeaBIOS) limitation, a QEMU version-specific bug, or an interaction between the guest kernel version (6.8.0-100-generic) and the Q35 chipset emulation.

Assumption 2: Bridge Windows Are Correctly Sized

The assistant assumes that the 160GB bridge windows allocated by QEMU are sufficient for each GPU. However, each GPU has not just BAR2 (128GB) but also BAR0 and BAR4. The assistant accounts for this qualitatively but doesn't calculate the exact overhead. If BAR0 or BAR4 require more space than expected, the bridge windows could be marginally insufficient.

Assumption 3: The Fix Is Non-Destructive

The assistant assumes that adding pci=realloc to the guest kernel command line is safe and will not cause other issues. In practice, pci=realloc can sometimes cause device initialization ordering problems or, in rare cases, system instability. The assistant does not discuss these risks.

Assumption 4: The Pattern of Success (GPUs 4 and 8) Is Informative

The assistant notes that GPUs at 04:00.0 and 08:00.0 succeeded while others failed. It attributes this to their BARs "happening to fit within their bridge windows." This is plausible but the assistant doesn't fully explain why these two specific positions succeeded. A deeper analysis might reveal that the bridge windows for buses 04 and 08 are at the edges of the MMIO range and may have had less contention.

Knowledge Required to Understand This Message

To fully grasp the assistant's analysis, one needs familiarity with several technical domains:

PCI BARs and Address Spaces

Every PCIe device exposes Base Address Registers (BARs) that define memory-mapped regions for device communication. A GPU's BAR2 is particularly large because it maps the entire VRAM into the CPU's address space. Understanding how BARs are allocated—and why they can fail—requires knowledge of PCI bus enumeration, bridge window programming, and 64-bit MMIO addressing.

QEMU Chipset Emulation

The Q35 chipset emulation in QEMU is fundamentally different from the i440FX. Q35 provides a more realistic PCIe topology with root ports, but it also imposes constraints on address space allocation. The mch.above_4g_mem_size parameter controls the size of the 64-bit MMIO window above 4GB, and understanding its interaction with PCI BAR allocation is crucial.

Linux PCI Resource Allocation

The Linux kernel's PCI subsystem performs a multi-pass allocation of resources. The pci=realloc parameter enables an additional pass that attempts to reassign resources that failed initial allocation. This is a relatively obscure kernel parameter that most administrators never need to use.

NVIDIA Driver Initialization

The NVIDIA proprietary driver (and the open-source nvidia-open module) performs its own validation of PCI BAR assignments during probe. If BAR2 is reported as 0, the driver refuses to initialize the device. Understanding this behavior is essential for interpreting the "BAR2 is 0M @ 0x0" error messages.

Knowledge Created by This Message

The assistant's response generates several forms of actionable knowledge:

A Clear Diagnosis

The message establishes that the Q35 chipset migration exposed a PCI BAR allocation limitation in the VM's firmware/kernel configuration. This is not a hardware failure or a driver incompatibility—it's a resource allocation issue that can be fixed with a kernel parameter change.

A Prioritized Remediation Plan

The assistant provides a three-step plan:

  1. Step 1 (primary): Add pci=realloc to the guest kernel command line via GRUB, update GRUB, and reboot.
  2. Step 2 (fallback): If Step 1 fails, increase the QEMU MMIO window on the host using mch.above_4g_mem_size=4T.
  3. Step 3 (verification): Once all 8 GPUs are detected, check P2P status with nvidia-smi topo -p2p. This structured approach gives the user a clear path forward while acknowledging uncertainty.

A Cautionary Tale About Virtualization

The message implicitly documents a broader lesson: virtualizing eight high-end GPUs with full PCIe passthrough pushes the limits of current hypervisor technology. The interaction between QEMU's chipset emulation, the guest kernel's resource allocator, and the NVIDIA driver's BAR validation creates a complex failure surface that requires deep expertise to navigate.

Documentation of the vIOMMU Distinction

The assistant's clarification about vIOMMU vs. host IOMMU is valuable knowledge. Many administrators conflate these concepts, and the assistant's warning that vIOMMU "adds IOMMU emulation overhead inside the VM and won't help with P2P" prevents a potentially costly misconfiguration.

The Broader Significance

This message is more than just a troubleshooting episode—it's a window into the challenges of GPU virtualization at scale. As large language models and other AI workloads demand ever more GPU memory and compute, the ability to virtualize multiple high-end GPUs in a single VM becomes increasingly important. Yet the virtualization stack—from the hypervisor to the chipset emulation to the guest kernel to the device drivers—is still catching up to these requirements.

The assistant's diagnostic approach in [msg 332] demonstrates the kind of multi-layered reasoning required to navigate this complexity. It reads raw kernel logs, traces allocation failures through multiple levels of PCI hierarchy, calculates address space budgets, and distinguishes between host-side and guest-side configuration options. It also shows intellectual honesty in its moments of uncertainty and self-correction.

Conclusion

Message [msg 332] is a remarkable piece of technical analysis. In response to a catastrophic failure—six of eight GPUs invisible after a critical VM reconfiguration—the assistant systematically diagnoses a PCI BAR allocation problem, identifies the root cause in the interaction between the Q35 chipset emulation and the Linux kernel's resource allocator, and prescribes a targeted fix. The message is notable for its transparent reasoning process, its careful arithmetic, and its clear separation of diagnosis from remediation.

The story does not end here. The user's next message ([msg 333]) shows them attempting the fix and encountering further complications. But the assistant's analysis in this message provides the foundation for everything that follows. It transforms a baffling failure—"my GPUs disappeared!"—into a tractable engineering problem with a known solution path. For anyone working at the intersection of GPU computing and virtualization, this message is a case study in how to think about PCI resource allocation, how to read kernel logs for diagnostic clues, and how to methodically narrow down a complex system failure.