The 10th Agent: Uncovering NUMA Memory Imbalance in a Multi-GPU Inference Stack

Introduction

In the high-stakes world of large language model inference, a 30x performance gap between theoretical maximum throughput and actual observed performance is a crisis demanding systematic investigation. This article examines a single message from an opencode coding session — message index 1269 — in which an AI assistant dispatches the tenth of ten parallel exploration agents to investigate memory and NUMA (Non-Uniform Memory Access) allocation on a dual-socket AMD EPYC server running eight NVIDIA Blackwell GPUs. While seemingly a routine diagnostic command, this message represents the culmination of a carefully orchestrated parallel reconnaissance effort, and its results would contribute to a comprehensive system audit that ultimately identified multiple critical misconfigurations throttling GPU inference performance.

The Message

The subject message is an assistant response containing a single tool call — a task invocation that spawns a subagent to investigate memory and NUMA allocation:

[assistant] [task] {"description":"Check memory/NUMA allocation","prompt":"SSH into both the Proxmox host (`ssh root@10.1.2.6`) and the LXC container (`ssh root@10.1.230.174`) to investigate memory allocation, NUMA distribution, and potential issues:\n\nOn the HOST:\n1. `numastat -m` - NUMA memory statistics\n2. `nu...
task_id: ses_3884ac296ffepkr58axwJc739W (for resuming to continue this task if needed)

<task_result>
Here is the complete analysis of all findings:

---

## HOST (Proxmox - 10.1.2.6) Findings

### 1. NUMA Memory Distribution

| Metric | Node 0 (252 GB) | Node 1 (252 GB) | Total (504 GB) |
|---|---|---|---|
| **MemTotal** | 257,545 MB | 257,968 MB | 515,513 MB |
| **MemFree** | 2,276 MB (0.9%) | 22,193 MB (8.6%) | 24,469 MB |
| **MemUsed** | 255,269 MB | 235,775 MB | 491,044 MB |
| **FilePages ...

The message is deceptively simple: a single task dispatched to a subagent. But understanding its significance requires reconstructing the investigative context that led to this moment.

The Investigative Context

To appreciate why this message was written, one must understand what preceded it. The session had been engaged in a multi-day effort to deploy and optimize the GLM-5-NVFP4 model — a large Mixture-of-Experts (MoE) language model using FP4 quantization — on a server equipped with eight NVIDIA RTX PRO 6000 Blackwell GPUs. After extensive kernel tuning, server parameter optimization, and benchmarking, the assistant had computed the theoretical maximum single-stream throughput for this configuration: approximately 309 tokens per second. The actual measured performance was a mere 10.36 tokens per second — a staggering 3.4% efficiency ratio.

This revelation triggered a fundamental shift in the investigation. Rather than continuing to tune software parameters, the assistant and user pivoted to a comprehensive system audit. The user's instruction was direct and strategic: "Start explore agents to 1. learn about potential issues observed in configurations for UEFI/Linux kernel (runtime knobs as well as kernel version)/LXC settings/Nvidia, Cuda driver versions/etc. and have those agents check the state of the machine we're working with. Use 10 parallel agents" ([msg 1259]).

The assistant responded by launching ten parallel task agents, each dedicated to a specific subsystem. Messages 1260 through 1268 dispatched the first nine agents, covering:

1. UEFI/BIOS settings — IOMMU status, ACS control, PCIe AER
2. LXC container configuration — cgroup settings, device permissions, memory limits
3. NVIDIA driver and CUDA configuration — driver version, CUDA toolkit compatibility, known issues
4. Kernel runtime knobs — CPU frequency governor, NUMA balancing, C-states, preemption
5. Kernel version compatibility — Proxmox kernel 6.8.12 vs. AMD EPYC Turin (Zen 5) and Blackwell GPUs
6. GPU/CUDA memory configuration — memory allocation, GPU state, ECC status
7. PCIe MaxReadReq size — PCIe configuration, read request size, payload size
8. Power and thermal configuration — CPU power states, GPU power limits, thermal throttling
9. NCCL/P2P bandwidth — GPU-to-GPU communication bandwidth measurements Message 1269, the subject of this article, dispatched the tenth and final agent: memory and NUMA allocation.

Why This Message Was Written: Reasoning and Motivation

The decision to investigate NUMA memory allocation was not arbitrary — it was driven by a deep understanding of how memory topology affects GPU compute performance in multi-socket systems. The server in question has two AMD EPYC 9335 processors (32 cores each), arranged across two NUMA nodes. Each NUMA node has its own local memory controller, and GPUs are connected to PCIe root complexes that are local to one NUMA node or the other.

The critical insight is this: when a GPU driver allocates memory for GPU use (e.g., for DMA buffers, CUDA contexts, or pinned memory), the allocation is served by the NUMA node local to the CPU core that initiates the allocation. If memory is severely imbalanced — with one NUMA node nearly exhausted while the other has ample free memory — GPU allocations could be forced to use remote memory, incurring cross-NUMA latency penalties. For a bandwidth-sensitive inference workload processing tokens at millisecond timescales, even small cross-NUMA penalties could compound into significant throughput degradation.

The assistant's motivation was therefore to rule out (or confirm) memory imbalance as a contributor to the 30x performance gap. This was especially important because the system was running an LXC container with 460 GB of memory allocated — nearly all of the host's 504 GB. If the container's memory was pinned to one NUMA node, or if the host's memory allocation was heavily skewed, it could explain why GPU operations were slower than expected.

How Decisions Were Made

The decision to launch ten parallel agents, rather than investigating sequentially, reflects a strategic choice about parallelism and efficiency. The assistant recognized that these investigations were largely independent — checking the kernel version does not require knowing the PCIe configuration first. By launching all ten agents simultaneously, the assistant could gather a complete system profile in roughly the time it would take to run the single slowest check.

Within this parallel strategy, the NUMA agent was designed to cover both the host and the LXC container. The prompt instructs the subagent to SSH into both machines and run a battery of commands: numastat -m for memory statistics, numastat -p for per-process NUMA policies, checks for NUMA balancing, and verification of NUMA-aware memory allocation for the container. This dual-scope approach was essential because NUMA issues can manifest at multiple layers — the hypervisor's memory allocation strategy, the container's memory binding, and the application's memory access patterns.

Assumptions Made

The assistant made several assumptions in designing this investigation:

1. NUMA configuration could be a significant factor. This was a reasonable assumption for a dual-socket system running memory-intensive GPU workloads. However, the subsequent analysis would show that NUMA imbalance, while real, was not the primary bottleneck — the FP4 GEMM kernel overhead dwarfed all other concerns.
2. The LXC container's memory allocation could be NUMA-pinned. The assistant checked whether the container was bound to a specific NUMA node via its configuration, which would constrain memory flexibility.
3. The tools available (numastat, numactl, lscpu) would provide sufficient diagnostic data. These are standard Linux tools for NUMA analysis, and they were indeed adequate for the initial assessment.
4. The subagent would correctly interpret the raw data. The assistant trusted the subagent to run the commands and produce a coherent analysis, which the task result confirms it did.

Input Knowledge Required

Understanding this message requires knowledge of several domains:

• NUMA architecture: The concept that in multi-socket systems, memory access latency depends on which CPU socket (NUMA node) the memory is physically connected to. Accessing memory from a remote NUMA node incurs higher latency than local access.
• GPU memory allocation: How GPU drivers (NVIDIA's in particular) allocate host memory for DMA transfers, pinned memory, and CUDA contexts, and how these allocations interact with NUMA topology.
• LXC container memory management: How container runtimes allocate and bind memory, and how cgroup memory limits interact with NUMA nodes.
• Linux diagnostic tools: The purpose and output format of numastat, numactl, and related utilities.
• The broader performance context: The 30x gap between theoretical and actual throughput, and the ongoing effort to identify its root causes.

Output Knowledge Created

The task result produced several valuable findings:

1. NUMA memory distribution on the host: Node 0 had only 0.9% free memory (2,276 MB out of 257,545 MB), while Node 1 had 8.6% free (22,193 MB out of 257,968 MB). This imbalance meant that any new memory allocation on Node 0 would likely trigger swapping or cross-NUMA allocation.
2. The LXC container's memory allocation: The container's 460 GB allocation was distributed across both NUMA nodes, but the imbalance reflected the host's overall memory pressure.
3. NUMA balancing status: Whether automatic NUMA balancing was enabled (which can cause page migration overhead).
4. Per-process NUMA policies: How the sglang server process was distributing its memory across NUMA nodes. This output fed directly into the broader system audit. While the NUMA imbalance was noted as a potential concern, the parallel agents investigating other subsystems would uncover more critical issues: a suboptimal CPU governor (acpi-cpufreq instead of amd_pstate), an outdated kernel (6.8.12), enabled NUMA balancing, deep CPU C-states, and PCIe MaxReadReq stuck at 512 bytes instead of 4096. These findings collectively painted a picture of a system that was fundamentally misconfigured for high-performance GPU compute.

The Thinking Process

The assistant's thinking process, visible in the structure of the investigation, reveals a methodical and systematic approach. Rather than jumping to conclusions about the performance gap, the assistant chose to gather comprehensive data across all relevant subsystems before making any diagnosis. This is the mark of a mature troubleshooting methodology — "measure, don't guess."

The choice to investigate NUMA specifically reflects an understanding that memory topology is a frequently overlooked performance factor in multi-GPU systems. Many practitioners focus exclusively on GPU compute capabilities, neglecting the host memory subsystem that feeds data to the GPUs. The assistant's inclusion of NUMA in the audit demonstrates a holistic systems-level perspective.

Furthermore, the dual-scope investigation (host and container) shows awareness that performance issues can originate at multiple layers of abstraction. A NUMA imbalance on the host might not directly affect the container if the container's memory is properly distributed, but a container pinned to a single NUMA node could starve its GPUs of local memory bandwidth.

Conclusion

Message 1269, while appearing as a simple task dispatch, represents the tenth and final prong of a comprehensive parallel investigation into a massive inference performance gap. The NUMA memory allocation agent it spawned would reveal a significant memory imbalance between the two NUMA nodes on the host system — a finding that, while not the primary bottleneck, contributed to the complete system picture that would ultimately guide the optimization effort.

This message exemplifies the power of systematic, parallelized troubleshooting. By launching ten independent investigations simultaneously, the assistant was able to produce a holistic system assessment in minimal time. The NUMA agent, like its nine siblings, was a necessary piece of the puzzle — and only by assembling all ten pieces could the team identify the true bottlenecks and begin the work of closing the gap between theoretical and actual performance.