The System Audit That Changed Everything: How 10 Parallel Agents Diagnosed a 30x Performance Gap

Introduction

In the high-stakes world of large language model inference serving, a 30x gap between theoretical maximum performance and actual throughput is not merely a disappointment—it is a crisis demanding systematic investigation. When the GLM-5-NVFP4 model, deployed across eight NVIDIA RTX PRO 6000 Blackwell GPUs, delivered a meager 10.36 tokens per second at single-stream concurrency against a theoretical ceiling of 309 tok/s, the disparity signaled that something fundamental was wrong with the system configuration. The model was not compute-bound in the way one might expect; rather, the entire hardware-software stack was operating far below its potential.

Message 1273 in this conversation represents a watershed moment: the consolidation of findings from ten parallel exploration agents dispatched to audit every layer of the system, from UEFI firmware settings to kernel runtime parameters, LXC container configuration, NVIDIA driver stack, PCIe topology, and memory management. This single message transformed a vague suspicion of "something wrong" into a precise, prioritized action plan with sixteen concrete findings, nine high-impact runtime fixes, and two medium-impact changes requiring reboot. It is a masterclass in systematic performance debugging at scale.

The Context: A Performance Crisis Demands Answers

To understand why message 1273 was written, one must appreciate the events immediately preceding it. The conversation had been tracking the performance of GLM-5-NVFP4—a Mixture-of-Experts model with 256 experts running in FP4 precision—through multiple optimization cycles. The assistant had already experimented with FlashInfer CUTLASS MoE autotuning, expert parallelism (EP8), piecewise CUDA graphs, and various server parameter tunings. Each optimization yielded incremental gains, but the single-stream performance remained stubbornly around 10 tok/s.

The theoretical maximum analysis, computed just before the audit, was devastating: the GPUs should have been capable of 309 tok/s for a single stream given their 2.3 PFLOPS of FP4 compute and the model's 8.2 billion active parameters per forward pass. The actual performance represented a mere 3.4% utilization of theoretical peak. Something was systematically wrong—not with the model or the serving software, but with the environment itself.

The user's instruction in message 1259 was precise and urgent: "Start explore agents to 1. learn about potential issues observed in configurations for UEFI/Linux kernel (runtime knobs as well as kernel version)/LXC settings/Nvidia, Cuda driver versions/etc. and have those agents check the state of the machine we're working with. Use 10 parallel agents." This directive recognized that the problem likely lay not in the application layer but in the system configuration stack beneath it.

The Architecture of the Audit: Ten Agents, One Comprehensive Picture

The assistant's response to this challenge was architecturally elegant. Rather than performing a sequential investigation—checking one subsystem at a time—it launched ten parallel task agents, each responsible for a specific domain of inquiry. This parallelism was crucial: the agents could run concurrently without blocking each other, and the parent session would only resume once all ten had completed. The task tool in opencode spawns subagent sessions that run to completion independently, and multiple task calls in the same round execute their subagents in parallel.

The ten agents covered:

1. UEFI/BIOS settings — IOMMU mode, ACS (Access Control Services), ReBAR (Resizable BAR), PCIe error reporting
2. LXC container configuration — cgroup limits, device access, AppArmor, memory locking
3. NVIDIA driver version and CUDA configuration — driver version, CUDA toolkit compatibility, GPU compute mode
4. Kernel runtime knobs — CPU frequency governor, NUMA balancing, scheduler settings, C-states
5. Kernel version compatibility — Proxmox kernel version, AMD EPYC Turin (Zen 5) support, Blackwell GPU compatibility
6. CUDA/GPU memory configuration — GPU memory allocation, fragmentation, pinned memory limits
7. PCIe MaxReadReq size — Per-GPU PCIe configuration, MaxReadRequest size, MaxPayload size
8. Power and thermal configuration — CPU power states, GPU power caps, thermal headroom, PSU capacity
9. NCCL/P2P bandwidth — GPU-to-GPU communication bandwidth, NVLink presence, P2P topology
10. SGLang/Python runtime configuration — Python version, package versions, environment variables, ulimits This division of labor was not arbitrary. Each agent targeted a known category of performance bottlenecks in GPU-accelerated ML serving. The PCIe agent, for instance, was motivated by the well-documented impact of MaxReadReq on GPU DMA performance—a setting that defaults to a conservative 512 bytes on many systems but can be increased to 4096 for substantial bandwidth gains. The kernel runtime knobs agent targeted CPU frequency scaling and NUMA balancing, both of which can silently degrade GPU kernel dispatch latency. The LXC agent checked for containerization overhead that might add latency to CUDA operations.

What Was Found: The Good, the Bad, and the Actionable

The consolidated report in message 1273 is structured with remarkable clarity. It begins with a "Confirmed Good" section—an equally important finding that many performance audits neglect. Sixteen system attributes were verified as optimal: IOMMU passthrough mode, PCIe Gen5 x16 links with no degradation, hardware ACS per GPU, ReBAR enabled at 128GB, GPU thermals well within operating range (28-31°C idle), full 600W TDP available, GPU clocks at 96-97% of max boost, ECC enabled with zero errors, sufficient PSU capacity, performance CPU governor, and correct PCIe Relaxed Ordering/NoSnoop/10-bit Tags/ECRC settings.

Perhaps the most definitive finding was the resolution of the NVLink question. Earlier in the conversation, there had been speculation about whether the RTX PRO 6000 Blackwell GPUs had NVLink interconnects. A mystery device at PCI address 0000:52:00.0 had been suspected of being an NVSwitch. Agent 10 definitively identified it as an ASPEED AST2600 BMC/IPMI VGA controller—a standard server management controller, completely unrelated to NVIDIA interconnects. The verdict was unambiguous: "NO NVLink. This hardware does NOT have NVLink capability." This finding had profound implications for optimization strategy, as it meant that custom allreduce fusion—a technique that requires NVLink for efficient GPU-to-GPU communication—would never work on this hardware. The assistant correctly classified this as a "Key Negative Result (NOT Issue)" rather than a fixable problem.

The actionable findings were ranked by impact into three tiers. The P0 (high impact, runtime-fixable) items included:

• nvidia_peermem module not loaded: This kernel module enables GPUDirect Peer-to-Peer (P2P) communication between GPUs, bypassing host memory for faster data transfer. Its absence meant that inter-GPU communication was going through system memory, adding latency to every allreduce operation during model inference.
• PCIe MaxReadReq stuck at 512 bytes: The GPU hardware supports a MaxReadRequest size of 4096 bytes, but the system had it configured at 512 bytes. This limits the amount of data the GPU can request in a single PCIe read transaction, reducing effective bandwidth for GPU-to-CPU and GPU-to-GPU transfers. The fix—using setpci to write 5937 to the PCIe capability register—is a low-level hardware configuration operation that requires precise knowledge of the PCIe register layout.
• NUMA balancing enabled: The kernel's automatic NUMA balancing (kernel.numa_balancing=1) periodically migrates memory pages between NUMA nodes to optimize access latency. For GPU workloads, where memory is pinned and allocated with specific NUMA affinity, this migration introduces unnecessary overhead and can cause performance degradation.
• CPU C-states allowing C2 (100μs wake latency): When CPUs enter deep sleep states (C2 and beyond), waking them to dispatch GPU kernels adds latency. For inference serving, where kernel dispatch happens at millisecond granularity, even 100μs of wake latency represents a significant fraction of total decode time.
• Container RLIMIT_MEMLOCK at ~63GB: CUDA pinned memory allocations are charged against the memlock limit. With eight GPUs each holding large model weights and KV caches, the 63GB limit could constrain memory allocation and cause allocation failures under load. The P1 (medium impact, reboot-required) items identified two systemic issues: the CPU scaling driver was using acpi-cpufreq instead of amd_pstate, which meant the AMD EPYC Turin processors were not using their hardware-guided frequency control path. The kernel version (6.8.12-9-pve) was also nine patches behind the latest in its series, and the report flagged that a 6.14 kernel could yield ~14% performance improvement for Turin processors.

The Thinking Process: From Data to Decision

What makes message 1273 particularly instructive is not just the data it presents but the reasoning structure embedded in its organization. The assistant did not simply dump raw findings from ten agents. It synthesized, prioritized, and contextualized every result.

The ranking of fixes by impact level demonstrates a sophisticated understanding of the performance landscape. P0 items were chosen because they could be applied immediately without reboot—critical when the goal was to improve throughput in a running production system. The assistant knew that asking an engineer to reboot a machine with active GPU workloads is a high-cost operation, so it separated runtime-fixable items from reboot-required items.

The inclusion of "Key Negative Results (NOT Issues)" is equally important. The assistant anticipated that the engineer might wonder about MaxPayload size (found to be 256 bytes, a hardware limit of Blackwell GPUs) or the disabled custom allreduce (expected for PCIe-only GPU configurations). By explicitly calling these out as non-issues, the assistant prevented wasted effort on dead-end investigations.

The note about the concurrency=10 benchmark anomaly—where throughput jumped from 38 tok/s in the previous session to 66 tok/s in the fresh baseline—demonstrates scientific rigor. Rather than celebrating the apparent improvement, the assistant flagged it as suspicious: "this needs investigation (different random seed? warmup state?)." This intellectual honesty is crucial in performance analysis, where measurement artifacts can easily mislead.

Assumptions and Their Validation

The entire audit rested on a critical assumption: that the 30x performance gap between theoretical and actual throughput was caused by system configuration issues rather than fundamental architectural limitations of the model or GPU. This assumption was reasonable given the magnitude of the gap—a 3.4% utilization rate is so low that it almost certainly indicates a systemic problem rather than a algorithmic one.

A secondary assumption was that the issues would be found in the "traditional" performance trouble spots: PCIe configuration, CPU frequency management, NUMA settings, and kernel parameters. This assumption proved partially correct—several significant issues were found in exactly these areas. However, the audit also revealed that many aspects of the configuration were already optimal, which was valuable negative information.

The assumption that NVLink might be present was definitively disproven. Earlier in the conversation, there had been hope that the RTX PRO 6000 Blackwell cards might support NVLink, which would enable GPU-to-GPU direct communication at much higher bandwidth than PCIe. The definitive identification of the mystery PCI device as a BMC controller closed this line of inquiry permanently.

The Knowledge Created: A System Blueprint

Message 1273 created a comprehensive knowledge artifact that served as the foundation for all subsequent optimization work. Before this message, the team had been operating with incomplete information—tuning server parameters and trying different MoE backends without understanding the systemic constraints. After this message, every optimization decision could be evaluated against a known baseline of system health.

The fresh baseline benchmarks included in the message provided a reference point for measuring the impact of subsequent fixes. With concurrency-1 throughput at 10.34 tok/s and TPOT at 95.21ms, any improvement from the P0 fixes could be quantified precisely. The message also documented the CuteDSL MoE backend comparison, showing it was slower than CUTLASS at high concurrency—a finding that prevented wasted effort on a dead-end optimization path.

Perhaps most importantly, the message created a shared mental model of the system's health across all participants. The "Confirmed Good" list was as valuable as the "Actionable Fixes" list because it prevented future investigators from chasing phantom issues. When someone later wondered "could it be a PCIe link issue?" they could refer back to this audit and see that all eight GPUs were confirmed at Gen5 x16 with zero errors.

The Broader Significance

Message 1273 exemplifies a methodology that extends far beyond this specific ML inference project. It demonstrates how to systematically audit a complex hardware-software system when performance falls dramatically short of expectations. The key principles are:

1. Parallelize investigation: Ten agents working in parallel can cover more ground than a sequential investigation in the same wall-clock time.
2. Separate confirmed-good from actionable-fix: Knowing what is NOT broken is as valuable as knowing what is.
3. Rank by impact and cost: Not all fixes are worth implementing. The P0/P1/P2 ranking considers both the potential performance impact and the operational cost (reboot vs. runtime).
4. Flag anomalies honestly: When measurements contradict expectations, acknowledge the uncertainty rather than forcing a narrative.
5. Document negative results explicitly: Preventing future dead-end investigations saves more time than it costs to document. The message also reveals the assistant's deep understanding of the full system stack—from PCIe register layout (CAP_EXP+0x08.w=5937 is not a command one writes without understanding PCIe capability structures) to kernel memory management (kernel.numa_balancing, vm.swappiness) to NVIDIA driver internals (nvidia_peermem module). This cross-layer expertise is what made the audit effective: each agent could dive deep into its domain while the parent session synthesized the results into a coherent whole.

Conclusion

Message 1273 is the turning point in a long optimization journey. Before it, the team was making incremental gains through application-level tuning—adjusting server parameters, trying different MoE backends, experimenting with expert parallelism. After it, the team had a roadmap of systemic fixes that could potentially unlock an order of magnitude improvement.

The message itself is a model of technical communication: structured, prioritized, evidence-based, and honest about uncertainties. It transformed ten parallel investigations into a single coherent action plan, creating knowledge that would guide every subsequent decision. In the high-pressure environment of production ML inference optimization, where every minute of downtime costs compute budget and engineering time, this kind of systematic audit is not a luxury—it is the only rational response to a 30x performance gap.