The NUMA Discovery: A Single Command That Unlocks GPU Topology Splitting

Introduction

In the sprawling, multi-session journey of deploying and optimizing large language models across eight NVIDIA RTX PRO 6000 Blackwell GPUs, there are moments of quiet revelation that stand apart from the more dramatic episodes of kernel compilation, CUDA version upgrades, and NCCL tuning. Message [msg 6036] is one such moment. It is deceptively simple: a single bash command piped through a loop, executed on a Proxmox hypervisor host. Yet this command encapsulates a critical decision point in the system architecture, one that determines how the entire GPU topology will be partitioned between two fundamentally different virtualization environments. Understanding why this message exists, what knowledge it presupposes, and what it unlocks for subsequent work reveals the deep interplay between hardware topology awareness and practical system administration in high-performance AI infrastructure.

The Message

The assistant executes the following command on the Proxmox host at 10.1.2.6:

ssh root@10.1.2.6 'for dev in 01:00.0 11:00.0 61:00.0 71:00.0 81:00.0 91:00.0 e1:00.0 f1:00.0; do numa=$(cat /sys/bus/pci/devices/0000:$dev/numa_node); echo "$dev -> NUMA $numa"; done'

The output reveals the NUMA node assignment for each GPU:

01:00.0 -> NUMA 0
11:00.0 -> NUMA 0
61:00.0 -> NUMA 0
71:00.0 -> NUMA 0
81:00.0 -> NUMA 1
91:00.0 -> NUMA 1
e1:00.0 -> NUMA 1
f1:00.0 -> NUMA 1

Four GPUs on NUMA 0, four on NUMA 1 — a perfect symmetric split.

Why This Message Was Written: The Reasoning and Motivation

The immediate trigger for this message is the user's instruction in [msg 6034]: "I now want to reconfigure the proxmox host such that the lxc container only has 4 GPUs and the other 4 are available for another VM / VMs through passthrough (nvidia driver to 4 on one numa, other 4 to vfio)."

This request is not arbitrary. It reflects a fundamental tension in the system's design. Up to this point, all eight GPUs had been exclusively assigned to a single LXC container running SGLang for large language model inference. The container was using all eight GPUs with tensor parallelism (TP=8) to serve the Qwen3.5-397B-A17B-NVFP4 model, achieving impressive throughput of over 2,100 tokens per second at high concurrency. However, the user now wants to share these expensive Blackwell GPUs between two workloads: the LXC container (for continued inference serving) and a separate VM (for other GPU-accelerated tasks).

The user's specification is precise: "nvidia driver to 4 on one numa, other 4 to vfio." This means four GPUs should remain bound to the NVIDIA driver (accessible to the LXC container), while the other four should be bound to the vfio-pci driver (for VM passthrough). Crucially, the user wants this split aligned with NUMA boundaries — one NUMA node's GPUs stay with NVIDIA, the other NUMA node's GPUs go to VFIO.

The assistant's first response ([msg 6035]) is to gather information. It runs lspci to list all NVIDIA devices, confirming the eight GPU PCI addresses. But lspci alone does not reveal NUMA topology. The assistant needs to know which PCI devices belong to which NUMA node to implement the user's request correctly. This is the precise motivation for [msg 6036].

Input Knowledge Required

To understand this message, one must possess several layers of knowledge:

Linux PCI device enumeration. The device paths (0000:01:00.0, 0000:11:00.0, etc.) follow the standard PCI domain:bus:device.function notation. The assistant already obtained these addresses from the lspci output in the previous message.

NUMA topology awareness. NUMA (Non-Uniform Memory Access) is a computer memory design used in multiprocessor systems where the memory access time depends on the memory location relative to the processor. On modern multi-socket systems, PCIe slots are typically assigned to specific NUMA nodes. A GPU on NUMA 0 has faster access to memory on NUMA 0 than to memory on NUMA 1. For GPU-intensive workloads, keeping GPU and CPU memory accesses within the same NUMA node is critical for performance.

Linux sysfs interface for PCI devices. The Linux kernel exposes hardware topology through the sysfs virtual filesystem. The path /sys/bus/pci/devices/0000:$dev/numa_node is a standard interface for reading the NUMA node assignment of a PCI device. This is not a widely known path — it requires familiarity with the Linux PCI subsystem's sysfs layout.

Proxmox virtualization architecture. The assistant is working with a Proxmox VE host that manages both LXC containers (lightweight OS-level virtualization) and KVM-based VMs (full hardware virtualization). GPU passthrough to VMs uses the vfio-pci driver, which takes over the PCI device from the host kernel. The user's request to split GPUs between "nvidia driver" and "vfio" reflects this dual virtualization strategy.

The specific hardware platform. The system has 8× NVIDIA RTX PRO 6000 Blackwell GPUs connected via PCIe Gen5. The assistant already knows from earlier work that these GPUs are spread across two NUMA nodes, but the exact mapping needs to be confirmed.

Output Knowledge Created

This message creates a concrete, actionable piece of knowledge: the NUMA-to-GPU mapping for all eight devices. The output reveals a perfectly symmetric topology:

How Decisions Were Made

The decision-making process in this message is subtle but important. The assistant faces a choice: how to determine the NUMA topology? There are several possible approaches:

  1. Parse lspci -v output, which sometimes includes NUMA information in verbose mode.
  2. Use numactl --hardware, which shows CPU and memory NUMA topology but not PCI device assignments.
  3. Query sysfs directly, reading the numa_node attribute for each PCI device.
  4. Use nvidia-smi topo -m, which shows NVIDIA GPU topology including NUMA affinity. The assistant chooses option 3 — the sysfs approach. This is a deliberate choice that reveals the assistant's system administration philosophy. The sysfs path is the most direct, lowest-level interface to the kernel's NUMA assignment for PCI devices. It requires no additional tools (unlike nvidia-smi, which requires the NVIDIA driver stack to be loaded and functioning). It works even if the GPUs are not yet bound to any driver, making it robust for the current state where the assistant is still planning the reconfiguration. The command structure is also revealing. The assistant uses a simple shell for loop over the eight PCI device addresses, reading each NUMA node assignment and printing it. This is not an optimized or elegant approach — it does not use xargs or parallel execution — but it is clear, debuggable, and produces human-readable output. The assistant prioritizes clarity over efficiency, which makes sense for a one-time discovery operation.

Assumptions Made

Several assumptions underpin this message:

The PCI device addresses are correct and stable. The assistant assumes that the eight addresses obtained from lspci in the previous message are the correct and complete set of NVIDIA GPUs. It does not verify that each address actually corresponds to an NVIDIA device before querying its NUMA node. If any address were wrong (e.g., a typo or a device that had been removed), the cat would fail, and the loop would print an error message.

The NUMA node information is available and accurate. The assistant assumes that the numa_node file exists for each device and contains a valid integer. On some systems, this file may contain -1 for devices not yet associated with a NUMA node, or it may not exist at all on non-NUMA systems. The assistant implicitly trusts that the kernel has correctly populated this information.

The NUMA split will be clean. The assistant does not yet know whether the GPUs are evenly distributed across NUMA nodes. It could discover an uneven split (e.g., 5 on NUMA 0 and 3 on NUMA 1), which would complicate the user's request. The command is designed to discover this fact, whatever it may be.

The user's NUMA-based split is feasible. The assistant assumes that the user's request to split by NUMA node is technically achievable — that is, that the LXC container can be configured to use only GPUs from one NUMA node, and the VM can use GPUs from the other. This requires Proxmox PCI passthrough configuration that respects PCI bus addresses, which is standard but not trivial.

Mistakes and Incorrect Assumptions

There are no outright mistakes in this message, but there are some notable omissions and potential issues:

No validation of device existence. The loop does not check whether each PCI device actually exists before reading its NUMA node. If any device address were incorrect, the cat command would produce a "No such file or directory" error, but the loop would continue. The assistant would need to examine the output carefully to spot such errors. In practice, all eight devices exist and respond correctly, so this is not a problem here.

No error handling for NUMA-unaware systems. If the system did not have NUMA enabled (e.g., a single-socket system or NUMA disabled in BIOS), the numa_node file might contain -1 or might not exist. The assistant does not handle this case. Again, this is not an issue on the actual hardware, which is a multi-socket server platform.

The command is executed over SSH with a quoted string. The entire command is passed as a single quoted argument to ssh. This works because the command contains no nested quotes or shell metacharacters that would cause parsing issues. However, it means the command runs on the remote host's default shell (likely bash), and the assistant has no direct control over the remote environment. This is standard practice and not a mistake, but it does mean the assistant is somewhat blind to the remote shell's state.

No verification of NUMA node 0 vs 1 meaning. The assistant does not verify which physical CPU socket corresponds to NUMA 0 and which to NUMA 1. For the purpose of splitting GPUs, this may not matter — the user only cares about keeping each workload's GPUs on a single NUMA node, not about which specific node. However, if one NUMA node has significantly better memory bandwidth or CPU proximity, the choice could affect performance. The assistant does not investigate this.

The Thinking Process Visible in Reasoning

The reasoning chain leading to this message is visible when we examine the conversation flow:

  1. [msg 6034]: The user states the goal: split 8 GPUs between LXC (4 GPUs, nvidia driver) and VM (4 GPUs, vfio-pci), aligned with NUMA boundaries.
  2. [msg 6035]: The assistant's first action is reconnaissance. It runs lspci to enumerate all NVIDIA devices on the Proxmox host. This confirms the eight GPU PCI addresses. But the assistant does not stop there — it recognizes that lspci alone does not provide NUMA information. The command output shows the eight addresses, and the assistant now needs to map them to NUMA nodes.
  3. [msg 6036]: The assistant executes the NUMA discovery command. The thinking here is: "I have the PCI addresses. I need to know which NUMA node each GPU is on. The Linux kernel exposes this via sysfs. I'll write a loop that reads the numa_node file for each device." The assistant does not ask the user for clarification about which GPUs to assign where. It does not propose an alternative split (e.g., by PCIe slot position or by GPU index). It simply executes the discovery step needed to implement the user's request exactly as specified. The choice of sysfs over alternatives like nvidia-smi topo -m is also a thinking signal. The assistant may have considered that nvidia-smi requires the NVIDIA driver to be loaded, and at this point in the reconfiguration, the driver state is uncertain. The sysfs approach works regardless of driver state, making it more robust.

Broader Context and Significance

This message sits at a pivotal moment in the infrastructure lifecycle. Up to this point, the entire system has been optimized for a single workload: serving the Qwen3.5-397B model on all eight GPUs. The user's decision to split the GPUs represents a shift from dedicated inference serving to multi-tenant GPU sharing. This is a common evolution in AI infrastructure: a system starts as a single-purpose inference server, then grows to support multiple workloads as its value is recognized.

The NUMA discovery is the first technical step in this transition. Without knowing which GPUs share a NUMA node, any split would be arbitrary and potentially performance-degrading. A workload whose GPUs are split across NUMA nodes would suffer from increased latency and reduced bandwidth for GPU-to-GPU communication (via PCIe), which is critical for tensor-parallel inference.

The symmetric split revealed by this command — 4 GPUs on NUMA 0, 4 on NUMA 1 — is the best possible outcome. It allows a clean, natural division that preserves NUMA locality for both workloads. If the split had been uneven (e.g., 5-3 or 6-2), the assistant would have needed to consult with the user about how to proceed, potentially requiring one workload to accept cross-NUMA GPUs or reducing the number of GPUs assigned to one workload.

Conclusion

Message [msg 6036] is a masterclass in targeted system discovery. It is not flashy — no kernel patches, no CUDA upgrades, no benchmark records. It is a single shell command that reads eight files from the Linux sysfs filesystem. Yet this command unlocks the entire GPU reconfiguration that follows. It answers the fundamental question that must be answered before any split can proceed: "Which GPUs are on which NUMA node?"

The command's elegance lies in its directness. It goes straight to the kernel's source of truth for NUMA topology, bypassing higher-level tools that might add interpretation layers or require specific driver states. It produces clear, unambiguous output that maps directly to the user's requirement. And it does so in a single SSH invocation, without requiring any changes to the system state.

In the broader narrative of this coding session, this message represents the transition from optimization to reconfiguration. The assistant has finished tuning the eight-GPU inference setup and now pivots to a new challenge: partitioning the hardware for multi-tenant use. The NUMA discovery is the first step in this new direction, and it is executed with the same precision and systems-level thinking that characterized the earlier work.