A Single Sentence That Saved a Benchmark: NUMA Awareness in Multi-GPU Inference

In the middle of an intensive benchmarking session on an 8-GPU machine, a single, six-word user message stopped a running experiment cold:

"it's loading on gpu 1-5, should be on 0-4 to be on one numa"

This message, sent as the assistant's TP4 benchmark was warming up ([msg 11327]), is a masterclass in concise, high-value feedback. It corrects a subtle but performance-critical configuration error that would have silently invalidated the entire TP4 benchmark phase. To understand why these few words matter so much, we must unpack the hardware architecture, the assistant's assumptions, and the deep system knowledge that the user brought to bear in an instant.

The Scene: A Benchmark in Progress

The context leading up to this message is a methodical benchmarking campaign on the CT200 machine — a Proxmox LXC container with 8× NVIDIA RTX PRO 6000 Blackwell GPUs. The assistant had just completed an extensive TP1 (single-GPU) benchmark of the Qwen3.6-27B model with DFlash and DDTree speculative decoding, producing strong results showing DDTree with budget 15 as the clear winner, achieving up to 6.5× speedup over autoregressive decoding ([msg 11325]).

Buoyed by these results, the assistant moved to TP4 — 4-way tensor parallelism across GPUs 1 through 4. The assistant's reasoning, as visible in the preceding message ([msg 11326]), was focused on token counts and context lengths: verifying that a 25,000-token prompt would fit within the 32,768 context window. The assistant launched the TP4 phase with the command python3 bench_runner.py tp4, and the benchmark began warming up.

Then the user aborted it.

What the User Saw

The user observed that the model was loading on GPUs 1 through 5 — five GPUs instead of the intended four. More importantly, they recognized that the correct GPU range should be 0 through 4, because those five GPUs (0, 1, 2, 3, 4) span two NUMA nodes on this machine, while GPUs 0 through 3 would stay within a single NUMA node.

Wait — the user said "0-4" which is five GPUs (0, 1, 2, 3, 4), but the assistant intended TP4 (4 GPUs). The user's correction is actually: "if you're going to use 4 GPUs for tensor parallelism, use GPUs 0-3 (all on NUMA node 0), not GPUs 1-4 (which straddle NUMA nodes)." The "0-4" phrasing is slightly loose — the user likely means "start at 0, use 4 GPUs" (i.e., 0, 1, 2, 3) rather than literally GPUs 0 through 4 inclusive. The key insight is NUMA locality.

NUMA: The Hidden Performance Trap

Non-Uniform Memory Access (NUMA) is a fundamental architectural feature of multi-socket server systems. The CT200 machine has two CPU sockets (NUMA nodes 0 and 1), each with its own memory controller and PCIe root complex. GPUs physically connected to the same socket's PCIe lanes can communicate with each other and with host memory on that socket at full speed. When communication crosses the inter-socket interconnect (Intel UPI or AMD Infinity Fabric), latency increases and bandwidth decreases.

The assistant's subsequent check of the NUMA topology ([msg 11329]) confirmed the layout:

| GPU | NUMA Node | CPU Affinity | |-----|-----------|-------------| | 0 | 0 | 0-31 | | 1 | 0 | 0-31 | | 2 | 0 | 0-31 | | 3 | 0 | 0-31 | | 4 | 1 | 32-63 | | 5 | 1 | 32-63 | | 6 | 1 | 32-63 | | 7 | 1 | 32-63 |

GPUs 0–3 are all on NUMA node 0, connected via NVLink/NVSwitch within the same socket. GPUs 4–7 are on NUMA node 1. The assistant's original choice of GPUs 1–4 placed three GPUs on NUMA node 0 (1, 2, 3) and one GPU on NUMA node 1 (GPU 4). Every tensor-parallel AllReduce operation would have required cross-NUMA communication, adding significant latency to every collective operation.

The Assistant's Assumptions

The assistant made two incorrect assumptions:

  1. That GPUs are interchangeable. The assistant chose GPUs 1–4 without considering NUMA topology, treating all GPUs as equivalent. On a single-socket machine or a system with a unified PCIe fabric, this would be correct. But on a dual-socket server, GPU index directly determines which CPU socket and memory controller the GPU is associated with.
  2. That the benchmark runner's default GPU assignment was reasonable. The assistant had configured the TP4 phase to use GPUs 1–4, likely as an arbitrary offset from the TP1 phase (which used GPU 1). There was no deliberate NUMA-aware placement strategy. The user's correction reveals an additional subtlety: even if the assistant had used GPUs 0–4 (five GPUs), that would still cross NUMA boundaries. The correct interpretation is that the user wants the 4 GPUs to be within a single NUMA node — either GPUs 0–3 or GPUs 4–7. The assistant correctly understood this and, after verifying the topology, fixed the device mappings to use GPUs 0–3 ([msg 11330]).

Knowledge Required and Created

To understand this message, one needs:

The Broader Lesson

This message exemplifies a pattern that recurs throughout high-performance computing: the most impactful feedback is often the most concise. A user who understands the hardware at the NUMA level can spot configuration errors in seconds that might take hours to surface through benchmarking artifacts. The assistant's willingness to immediately stop, verify, and correct — rather than argue or rationalize — turned a potential data-quality disaster into a minor course correction.

The corrected TP4 benchmarks went on to reveal that TP4 was faster than TP8 for single requests due to PCIe cross-NUMA overhead, and that DDTree b15 achieved an aggregate throughput of 1270.8 tok/s at 8 concurrent requests. These results would have been contaminated if the NUMA issue had gone uncorrected.

In the end, six words saved an entire benchmark phase. That is the power of deep system knowledge applied at precisely the right moment.