Splitting the Blackwells: A PCI Mapping Surgery on Proxmox

Introduction

In the course of a sprawling machine learning infrastructure session spanning dozens of segments and thousands of messages, a single seemingly mundane operation stands out as a critical inflection point: the moment when an assistant reaches into the Proxmox host's PCI mapping configuration and surgically splits eight NVIDIA RTX PRO 6000 Blackwell GPUs across two distinct ownership domains. Message 6058 is that moment. It is a brief, almost mechanical act — a heredoc piped into a configuration file — yet it encapsulates hours of prior investigation, embodies a fundamental architectural decision about resource partitioning, and sets the stage for an entirely new phase of the deployment.

This article examines message 6058 in depth: why it was written, what decisions it encodes, the assumptions it rests on, and the knowledge it both consumes and produces. To understand this message is to understand how modern AI infrastructure is assembled — not through grand architectural blueprints, but through a chain of small, precise interventions, each building on the last.

The Context: From 8 GPUs to a Divided House

The session up to this point had been a relentless optimization campaign. The assistant and user had been deploying large language models — first GLM-5-NVFP4, then Qwen3.5-397B-A17B-NVFP4, then Qwen3.5-122B BF16 — across eight RTX PRO 6000 Blackwell GPUs connected via PCIe. They had wrestled with flash-attn compilation, CUDA version mismatches, NCCL P2P DMA corruption under SEV-SNP IOMMU, and a cascade of backend configuration issues. The machine was a Proxmox hypervisor, and the GPUs were being passed through to an LXC container running SGLang for model serving.

But the user had a new requirement. In message 6034, they said: "I now want to reconfigure the proxmox host such that the lxc container only has 4 GPUs and the other 4 are available for another VM / VMs through passthrough (nvidia driver to 4 on one numa, other 4 to vfio)."

This was not a trivial request. The eight GPUs were physically distributed across two NUMA nodes — four on NUMA 0 (PCI addresses 01:00.0, 11:00.0, 61:00.0, 71:00.0) and four on NUMA 1 (81:00.0, 91:00.0, e1:00.0, f1:00.0). The existing Proxmox PCI mapping, named pro6000, listed all eight GPUs in a single group, and the LXC container had bind-mount entries for all eight /dev/nvidia* device files. The SGLang server was configured for tensor parallelism across all eight GPUs (TP=8).

The assistant's response to the user's request was methodical. It first mapped the topology, discovering the NUMA split and the stray ASPEED BMC VGA controller at 52:00.0 that could be mistaken for a ninth GPU. It stopped the SGLang server and the LXC container. It unbond the four NUMA 1 GPUs from the nvidia driver and rebond them to vfio-pci — the kernel driver that enables VFIO passthrough to virtual machines. It edited the LXC configuration to remove the bind-mount entries for nvidia4 through nvidia7. And then, in message 6058, it updated the PCI mapping.

What Message 6058 Actually Does

The message is a single bash command executed on the Proxmox host (root@10.1.2.6):

cat > /etc/pve/mapping/pci.cfg << 'EOF'
nv3080
	map id=10de:2204,iommugroup=72,node=kpro4,path=0000:43:00,subsystem-id=1043:87d5

nv3090
	map id=10de:2204,iommugroup=20,node=kpro5,path=0000:53:00,subsystem-id=1043:87d5

nv3090-2
	map id=10de:2204,iommugroup=24,node=kpro5,path=0000:57:00,subsystem-id=1043:87d5

nv3090-3
	map id=10de:2204,iommugroup=50,node=kpro5,path=0000:d6:00,subsystem-id=1043:87d5

nva6000
	map id=10de:2230,iommugroup=16,node=kpro5,path=0000:4f:00,subsystem-id=10de:1459

nva6000-2
	map id=10de:2230,iommugroup=43,node=kpro5,path=0000:ce:00,subsystem-id=10de:1459

pro6000
	map id=10de:2bb5,iommugroup=25,node=kpro6,path=0000:01:00.0,subsystem-id=10de:204e
	map id=10de:2bb5,iommugroup=47,node=kpro6,path=0000:11:00.0,subsystem-id=10de:204e
	map id=10de:2bb5,iommugroup=21,node=kpro6,path=0000:61:00.0,subsystem-id=10de:204e
	map id=10de:2bb5,iommugroup=2,node=kpro6,path=0000:71:00.0,subsystem-id=10de:204e

pro6000-vm
	map id=10de:2bb5,iommugroup=74,node=kpro6,path=0000:81:00.0,subsystem-id=10de:204e
	map id=10de:2bb5,iommugroup=93,node=kpro6,path=0000:91:00.0,subsystem-id=10de:204e
	map id=10de:2bb5,iommugroup=70,node=kpro6,path=0000:e1:00.0,subsystem-id=10de:204e
	map id=10de:2bb5,iommugroup=51,node=kpro6,path=0000:f1:00.0,subsystem-id=10de:204e
EOF
echo "PCI mapping updated"

This overwrites the entire /etc/pve/mapping/pci.cfg file. The file is Proxmox's registry of PCI device mappings — named groups of physical GPUs that can be assigned to virtual machines through the hostpci configuration key. Each mapping entry specifies:

The device ID (10de:2bb5 for the RTX PRO 6000 Blackwell)
The IOMMU group number (critical for VFIO isolation)
The Proxmox node name (kpro6)
The PCI bus address
The subsystem ID (10de:204e) The operation does three things in one atomic write: 1. Preserves all existing mappings for other GPU types on the same hypervisor (the nv3080, nv3090, nva6000 series, which belong to other nodes kpro4 and kpro5) 2. Truncates the pro6000 mapping from eight entries to four, keeping only the NUMA 0 GPUs 3. Creates a brand new mapping pro6000-vm containing the four NUMA 1 GPUs, now bound to vfio-pci and ready for VM passthrough## The Reasoning: Why This Message Exists Message 6058 is the culmination of a chain of reasoning that begins with the user's explicit requirement: split the 8 GPUs so that 4 stay with the LXC container (running SGLang for model serving) and 4 become available for a new VM (likely for a different workload, possibly a different model or a development environment). But beneath that surface-level instruction, there are deeper motivations. The NUMA topology is the key. The eight GPUs are split across two NUMA nodes, and NUMA boundaries matter enormously for GPU performance. Cross-NUMA memory access is slower than local NUMA access. By assigning the LXC container the four GPUs on NUMA 0 and the VM the four on NUMA 1, the assistant ensures that each workload's GPUs are on the same NUMA node as the CPU cores they'll be scheduled on. This avoids the performance penalty of remote memory access. The assistant's earlier investigation (message 6036) explicitly checked the NUMA assignment of each GPU, and this topology-informed split is the direct result. Furthermore, the split respects IOMMU group boundaries. Each GPU is in its own IOMMU group (groups 25, 47, 21, 2 for NUMA 0; groups 74, 93, 70, 51 for NUMA 1), meaning they can be individually assigned without needing to pass through a whole PCIe root port. This was verified in message 6049, where the assistant checked each IOMMU group and confirmed that only the GPU endpoint (plus its PCIe bridge, which doesn't need rebinding) was present. This granularity is essential for the split to work cleanly. The message also represents a deliberate architectural choice about persistence. The PCI mapping file is part of Proxmox's cluster filesystem (/etc/pve/), which is synchronized across all nodes in the cluster. By writing the mapping here, the assistant ensures that the GPU assignment survives reboots and can be managed through Proxmox's standard tools (the pvesh API and the web interface). The earlier step of unbinding GPUs from nvidia and binding to vfio-pci via sysfs was a runtime operation that would not survive a reboot — the PCI mapping file is the persistent counterpart.

The Decisions Encoded in the Configuration

Every line of the new pci.cfg encodes a decision. Consider the choice of which GPUs go where. The assistant could have split by any criterion — first four GPUs by index, last four, a mix. Instead, it split strictly by NUMA node. This is a performance-aware decision that prioritizes locality over simplicity.

The naming convention is also significant. The existing mapping was simply pro6000. The assistant chose pro6000-vm for the new mapping, using a suffix convention that makes the relationship between the two mappings obvious. This is a small but important design choice — it makes the configuration self-documenting and reduces the chance of confusion when someone reads this file months later.

The decision to overwrite the entire file rather than append to it is worth noting. The assistant could have used sed to modify just the pro6000 section and then appended the new pro6000-vm section. Instead, it chose to rewrite the whole file from a template. This is safer: it avoids the risk of partial edits, corrupted state, or leftover entries. The template includes all the other GPU mappings verbatim, preserving the existing configuration for other nodes' GPUs. This "atomic replacement" strategy is a defensive programming practice — it ensures the file is always in a consistent state.

Assumptions and Their Validity

The message makes several assumptions, most of which are well-founded but worth examining.

Assumption 1: The IOMMU group numbers will remain stable across reboots. IOMMU group assignment is determined by the PCIe topology and the kernel's IOMMU implementation. On a given hardware platform with a fixed BIOS configuration, IOMMU groups are deterministic. However, if the user later changes BIOS settings (e.g., enabling PCIe ACS, adjusting ARI, or modifying IOMMU coverage — all of which the assistant investigates later in the segment), the IOMMU groups could shift. This is a reasonable assumption for the current configuration but a potential fragility point.

Assumption 2: The pro6000 mapping was originally intended for the LXC container. This is implicit in the assistant's decision to keep the name pro6000 for the NUMA 0 set. The original mapping was used by the LXC container (via lxc.mount.entry bind mounts, not through Proxmox's VM PCI mapping system). By retaining the name, the assistant preserves backward compatibility with any scripts or documentation that reference it.

Assumption 3: The subsystem ID 10de:204e is correct for all eight GPUs. The assistant derived this from the existing mapping entries. This is a safe assumption — all eight GPUs are the same model (RTX PRO 6000 Blackwell Server Edition) and should have the same subsystem ID. But if there were a mix of GPU variants (e.g., different OEM branding), this could be wrong. The earlier nvidia-smi output (message 6041) confirmed all eight have the same UUID prefix pattern, lending confidence.

Assumption 4: The node name kpro6 is correct. This was taken from the existing mapping and is the Proxmox node hostname. This is a trivial assumption — if the node name changed, the mapping would fail silently.## Input Knowledge: What You Need to Understand This Message

To fully grasp message 6058, a reader needs knowledge spanning several domains:

Proxmox VE internals. The file /etc/pve/mapping/pci.cfg is not a standard Linux configuration file — it is specific to Proxmox's PCI mapping subsystem, used to define named groups of PCI devices that can be assigned to virtual machines via the hostpci configuration key. The format is idiosyncratic: a mapping name followed by map lines with key-value pairs. Understanding that id=10de:2bb5 is the PCI vendor:device ID (NVIDIA's RTX PRO 6000 Blackwell), that iommugroup refers to the kernel's IOMMU group number, and that node is the Proxmox cluster node name, is essential.

PCIe and IOMMU concepts. The IOMMU group determines which devices can be isolated for VFIO passthrough. Devices in the same IOMMU group must be passed through together. The assistant's earlier verification that each GPU is in its own group (messages 6049-6050) is the prerequisite that makes this split possible. Without that understanding, the mapping file looks like arbitrary numbers.

NUMA topology awareness. The split along NUMA boundaries is intentional and performance-critical. A reader who doesn't understand NUMA might wonder why the assistant didn't simply take the first four GPUs by index. The answer lies in the earlier discovery (message 6036) that GPUs 0-3 are on NUMA 0 and GPUs 4-7 are on NUMA 1.

The session's history. Message 6058 is meaningless without the context of the preceding 23 messages (6014-6057), which include the topology discovery, the driver rebinding, the LXC config edit, and the verification steps. The assistant's todo list (visible in messages 6043, 6047, 6053, 6056) tracks the progression through these steps, and message 6058 is explicitly the "Create PCI mapping for VM passthrough" item being checked off.

Output Knowledge: What This Message Creates

Message 6058 produces both tangible and intangible outputs.

Tangible: A new version of /etc/pve/mapping/pci.cfg on the Proxmox host. This file now has two mappings for the Blackwell GPUs: pro6000 (4 GPUs on NUMA 0, still bound to the nvidia driver) and pro6000-vm (4 GPUs on NUMA 1, bound to vfio-pci). The VM administrator can now assign pro6000-vm to a new VM via hostpci0: mapping=pro6000-vm in the VM configuration file.

Intangible: A documented architectural boundary. The mapping file serves as a form of infrastructure documentation — it records the decision to split the GPUs by NUMA node, the IOMMU group assignments, and the intended ownership (LXC vs. VM). Anyone who reads this file later can reconstruct the topology and the reasoning behind it.

Also intangible: A checkpoint in the session's progress. With the PCI mapping in place, the assistant can proceed to start the LXC container, reconfigure SGLang for TP=4, and begin serving the Qwen3.5-122B model on the reduced GPU set. The VM passthrough path is now open for future work.

The Thinking Process: What the Assistant's Actions Reveal

Although message 6058 does not contain explicit reasoning text (it is a straightforward bash command), the assistant's thinking is visible in the sequence of actions leading up to it. The pattern is classic systems engineering:

Discover the current state (messages 6035-6042: enumerate GPUs, check NUMA, check IOMMU groups, verify driver bindings)
Plan the transformation (message 6043: todo list with 6 steps)
Execute in dependency order (stop server → stop container → rebind GPUs → edit LXC config → update PCI mapping)
Verify each step (message 6051: confirm driver bindings; message 6052: confirm nvidia-smi sees only 4 GPUs)
Persist the configuration (message 6058: write the mapping file) The assistant's choice to use a heredoc with cat > rather than an editor or incremental modification is telling. It reflects a preference for deterministic, scriptable operations over interactive editing. The entire mapping file is reconstructed from a template, ensuring that the result is exactly what was intended, with no risk of partial edits or leftover artifacts from previous configurations. The message also reveals the assistant's awareness of the Proxmox cluster filesystem. The path /etc/pve/mapping/pci.cfg is a special path — it is not a regular file on disk but a file in Proxmox's PMXCFS cluster filesystem, which synchronizes across all nodes. Writing to it triggers cluster-wide updates. The assistant does not treat this with any special ceremony, indicating a comfortable familiarity with Proxmox's internals.

Potential Mistakes and Risks

The message is clean, but several risks are worth noting.

Risk of file corruption during write. The cat > operation truncates the file before writing. If the command is interrupted mid-write (e.g., network timeout, disk failure), the file could be left in a partial state, potentially breaking PCI mappings for all other GPU types on the system. A safer approach would be to write to a temporary file and then atomically rename it, but Proxmox's cluster filesystem may not support that pattern reliably.

Risk of stale IOMMU group numbers. The IOMMU group numbers were verified at runtime, but they are hardware-dependent and can change with BIOS updates or kernel version changes. If the system is rebooted with different firmware settings, the mapping file could reference nonexistent or incorrect IOMMU groups.

Risk of naming collision. The name pro6000-vm is descriptive but could conflict with a future mapping created by another administrator who chooses the same convention. This is a minor risk but worth noting in a multi-admin environment.

No validation after write. The assistant does not verify that the new mapping file is syntactically valid or that Proxmox accepts it. The echo "PCI mapping updated" is a confirmation of the write, not of the file's correctness. A subsequent step (creating the VM with pro6000-vm) would reveal errors, but by then the misconfiguration could be confusing.

Conclusion

Message 6058 is a small but pivotal moment in a complex infrastructure deployment. It represents the transition from a unified 8-GPU serving setup to a partitioned architecture where different workloads can coexist on the same physical hardware. The message itself is simple — a single heredoc writing a configuration file — but it encodes hours of investigation, a deep understanding of NUMA topology and IOMMU isolation, and a careful sequence of preparatory steps.

In the broader narrative of the session, this message marks the point where the assistant stops optimizing a single model deployment and starts building a multi-tenant infrastructure. The split enables the user to run both the LXC-based SGLang server (for production inference) and a new VM (for development, experimentation, or a different model) on the same machine, each with dedicated GPUs on their own NUMA node.

The article has examined the reasoning, decisions, assumptions, and knowledge embedded in this message. What emerges is a picture of infrastructure engineering as a craft of precision: each line of configuration is a decision, each decision rests on verified facts, and each fact was uncovered through methodical investigation. Message 6058 is not just a configuration update — it is the crystallization of a system's understanding into a persistent, actionable form.