The Hidden Lever: BIOS Configuration as the Final Frontier in Multi-GPU Inference Performance

In the high-stakes world of multi-GPU inference deployments, the most frustrating problems are often the ones that sit at the boundary between hardware and software — where the operating system, hypervisor, and physical platform intersect in ways that no single documentation set fully covers. Message 402 in this opencode session represents exactly such a moment: after hours of deep technical investigation into PCIe topology, IOMMU groups, kernel parameters, and virtualization workarounds, the conversation arrives at a surprisingly practical pivot point. The user asks a simple question — "Btw any settings in bios that are highly warrated to set? Asus E13 GPU server" — and the assistant responds with one of the most comprehensive and meticulously reasoned BIOS configuration guides for AMD EPYC-based GPU servers that one could hope to find in a coding session transcript.

This message is remarkable not because it solves a novel engineering problem, but because it demonstrates something rarer: the ability to step back from a complex technical impasse and identify the remaining unturned stones. The assistant had just spent multiple rounds exhaustively investigating why Peer-to-Peer (P2P) DMA between GPUs was impossible inside a Proxmox VM. The conclusion was stark: the ASUS ESC8000A-E13 motherboard's hardware topology — with each of the eight RTX PRO 6000 Blackwell GPUs connected to its own dedicated PCIe root complex — fundamentally prevents VFIO from granting direct GPU-to-GPU DMA access, regardless of any software or kernel configuration. This was a hard ceiling. But the BIOS settings message reveals that even when a primary goal is blocked, there are often secondary levers that can meaningfully improve the situation.

The Message in Full

The assistant's response is structured as a reference-grade BIOS settings table, organized into three tiers of importance:

Here are the **critical BIOS settings** for your ASUS ESC8000A-E13 with AMD EPYC 9335 and 8x NVIDIA GPUs. These are based on AMD's official recommendations and GPU server best practices:

## Critical BIOS Settings (check/change these)

### Must-Have Settings

| Location | Setting | Value | Why |
|----------|---------|-------|-----|
| **Advanced / PCI Subsystem** | **Above 4G Decoding** | **Enabled** | Required for GPU large BAR (128GB VRAM mapping). If this is off, GPUs can't expose full VRAM. |
| **Advanced / PCI Subsystem** | **Resizable BAR** | **Enabled** | Allows GPUs to expose full VRAM BAR to the system |
| **Advanced / PCI Subsystem** | **PCIe Generation** | **Gen 5** (or Auto) | Ensures max PCIe bandwidth. Your GPUs show Gen5 x16 on host but Gen1 x16 in VM — the Gen1 is just QEMU virtual link, not real. |
| **AMD CBS / NBIO** | **IOMMU** | **Enabled** | Already on (you have IOMMU working). Required for VFIO passthrough. |
| **AMD CBS / NBIO** | **PCIe Ten Bit Tag** | **Enabled** | Increases PCIe TLP efficiency. Required for high bandwidth. |
| **AMD CBS / NBIO** | **ACS (Access Control Services)** | **Enabled** | Required for VFIO isolation. You have this since GPUs are in separate IOMMU groups. |
| **AMD CBS / NBIO** | **PCIE ARI Enumeration** | **Enabled** | Required for proper PCIe device enumeration. |
| **AMD CBS / NBIO** | **PCIE ARI Support** | **Enabled** | Required. |

### Performance-Critical Settings

| Location | Setting | Value | Why |
|----------|---------|-------|-----|
| **Advanced / Power Management** | **Power Management** | **Maximum Performance** | Don't let the CPU throttle during inference |
| **AMD CBS / NBIO / SMU** | **Determinism Control** | **Manual** | |
| **AMD CBS / NBIO / SMU** | **Determinism Slider** | **Power** | Prioritize performance over power saving |
| **AMD CBS / NBIO / SMU** | **cTDP Control** | **Manual** | |
| **AMD CBS / NBIO / SMU** | **cTDP** | **Max for EPYC 9335** (likely 210W) | Run CPU at full power |
| **AMD CBS / NBIO / SMU** | **APBDIS** | **1** (Disabled) | Disables Data Fabric P-states — reduces latency |
| **AMD CBS / NBIO / SMU** | **DF C-states** | **Disabled** | **Key setting** — reduces cross-GPU communication latency by keeping the data fabric always active |
| **AMD CBS / NBIO / SMU** | **Fixed SOC P-state** | **P0** | Keeps SoC at highest performance — critical for GPU↔CPU interaction |
| **AMD CBS / NBIO** | **ASPM** | **Disabled** | Disables PCIe Active State Power Management — prevents link power-down latency |
| **AMD CBS / DF Common** | **NUMA Nodes Per Socket** | **NPS1** (or Auto) | One NUMA node per socket. NPS2/NPS4 can fragment memory and hurt GPU workloads. |
| **AMD CBS / DF Common** | **Memory Interleaving** | **Enabled** | Required for optimal memory bandwidth |
| **AMD CBS / GPU Common** | **Global C-state Control** | **Enabled** | AMD says keep this on (it's different from DF C-states) |
| **AMD CBS / GPU Common / Performance** | **SMT Control** | **Disable** | For pure compute workloads, disabling SMT can reduce jitter. However, since you're in a VM with 60 cores per socket, this trades core count for latency. **Optional — try with SMT on first.** |
| **AMD CBS / Security** | **TSME** | **Disabled** | Memory encryption adds latency to every memory access |

### Particularly Important for Your Setup

The settings most likely to help your specific bottleneck (cross-GPU latency in a VM) are:

1. **DF C-states = Disabled** — If the AMD Data Fabric enters C-states between GPU transfers, it adds wake-up latency to every cross-GPU copy. This could be contributing to your 13µs floor.

2. **APBDIS = 1** — Disabling DF P-states keeps the inter-socket and memory fabric at full speed.

3. **Fixed SOC P-state = P0** — Keeps the entire SoC fabric at max frequency.

4. **ASPM = Disabled** — Prevents PCIe link power management from adding latency.

5. **Above 4G Decoding + Resizable BAR = Enabled** — These should already be on since GPUs work, but worth confirming.

### How to Check Current Settings

If you have IPMI/BMC access (ASMB11-iKVM on the ASUS), you can check remotely. Otherwise it requires a physical console reboot into BIOS. 

Can you check these settings? The **DF C-states** and **APBDIS** settings in particular could explain some of the latency overhead and are worth verifying.

To understand why this BIOS settings message matters, one must appreciate the journey that preceded it. The broader opencode session (segments 0 through 3) tells the story of deploying the GLM-5-NVFP4 model — a large language model using NVIDIA's NVFP4 quantization format — across eight RTX PRO 6000 Blackwell GPUs in a Proxmox virtualized environment. The early segments dealt with the standard challenges of ML infrastructure: installing NVIDIA drivers, resolving flash-attn build issues, and getting SGLang running. But segment 3, the immediate context for this message, was where things got truly interesting.

The team had discovered a crippling performance bottleneck: cross-GPU communication latency was stuck at approximately 13 microseconds for small transfers, roughly an order of magnitude worse than what bare-metal GPU servers typically achieve. This latency floor was the direct result of virtualization overhead. In a KVM-based Proxmox VM, GPUs cannot perform direct Peer-to-Peer (P2P) DMA transfers. Every GPU-to-GPU data movement must follow a staged path: GPU A writes to host memory via the IOMMU, then GPU B reads from that same host memory. This round-trip through the hypervisor adds latency that no amount of software tuning within the VM can eliminate.

The assistant had already attempted every reasonable workaround. It migrated the VM from the legacy i440FX chipset to Q35 with proper PCIe passthrough. It added pci=realloc to fix BAR allocation failures. It set up sysfs NUMA affinity for all eight GPUs. It created a custom NCCL topology XML file to give NCCL accurate topology information. It even investigated disabling ACS (Access Control Services) on the host to merge IOMMU groups — a BIOS-level change that, in theory, could allow the VFIO subsystem to see multiple GPUs as belonging to the same IOMMU group and thus enable P2P. But that attempt failed because the motherboard's physical topology (each GPU on its own root complex) made ACS irrelevant as a blocking factor.

This is the moment captured in [msg 398], where the assistant provides a summary table of everything that had been accomplished and everything that remained unfixable. The tone is one of pragmatic acceptance: "What remains unfixable in the VM: P2P stays NS — this is a fundamental VFIO limitation." It is in direct response to this summary that the user asks about BIOS settings ([msg 399]), and the assistant's reply ([msg 402]) represents a pivot from "what can we hack" to "what can we optimize."

The Reasoning Behind Every Recommendation

What makes this message particularly valuable is the reasoning that underpins each BIOS setting recommendation. The assistant does not simply list settings; it explains the causal mechanism by which each setting affects performance, and it prioritizes them according to the specific bottleneck the user is experiencing.

Consider the treatment of DF C-states (Data Fabric C-states). The AMD Data Fabric is the interconnect that links the CPU cores, memory controllers, and I/O die within an EPYC processor. When the Data Fabric enters a C-state (a power-saving sleep state), it takes a measurable amount of time to wake back up. In a multi-GPU inference workload where GPUs are constantly sending small synchronization messages and activation data across the PCIe bus, every Data Fabric wake-up adds latency to every cross-GPU transfer. The assistant explicitly connects this to the observed 13-microsecond floor: "If the AMD Data Fabric enters C-states between GPU transfers, it adds wake-up latency to every cross-GPU copy. This could be contributing to your 13µs floor." This is not generic advice — it is a targeted hypothesis based on the specific symptoms observed in this deployment.

Similarly, APBDIS (Application Processor Bus Disable) is a setting that controls whether the Data Fabric can transition between P-states (performance states). Setting APBDIS to 1 locks the Data Fabric at its highest performance state, eliminating the latency spikes that occur when the fabric transitions between power levels. The assistant correctly identifies this as a companion to the DF C-states disable — together, they ensure that the interconnect between CPU, memory, and PCIe devices remains at maximum readiness at all times.

The Fixed SOC P-state = P0 recommendation follows the same logic. The SoC (System on Chip) fabric in an EPYC processor encompasses the memory controllers, PCIe root complexes, and inter-socket links. Locking it at P0 ensures that GPU-initiated memory accesses and PCIe transactions never encounter a fabric that is ramping up from a lower power state. In a virtualized environment where every GPU memory access already carries the overhead of IOMMU translation, adding SoC fabric wake-up latency on top would be particularly damaging.

The ASPM = Disabled recommendation addresses a different but related concern. PCIe Active State Power Management allows individual PCIe links to enter power-saving states when idle. In a multi-GPU inference server, GPUs may have brief idle periods between communication bursts. If ASPM is enabled, the PCIe link can power down during these idle periods and then require a wake-up sequence before the next transfer. This wake-up latency, typically in the range of microseconds, would add directly to every cross-GPU communication. The assistant's recommendation to disable ASPM is therefore directly motivated by the observed latency bottleneck.

What This Message Reveals About the Assistant's Thinking Process

The structure of the message reveals a sophisticated tiered reasoning approach. The assistant organizes the BIOS settings into three categories: "Must-Have Settings," "Performance-Critical Settings," and "Particularly Important for Your Setup." This is not arbitrary categorization — it reflects a deliberate triage based on the specific problem context.

The "Must-Have" settings are the foundational requirements for any GPU server: Above 4G Decoding, Resizable BAR, IOMMU enabled, PCIe Ten Bit Tag, ACS, ARI Enumeration, and ARI Support. These are settings that, if misconfigured, would prevent the system from working at all or would severely degrade basic functionality. The assistant notes that several of these are "already on" — the IOMMU is working, ACS is enabled (as evidenced by the separate IOMMU groups), and the GPUs are enumerated. This acknowledgment serves an important rhetorical purpose: it tells the user that the system is fundamentally sound and that we are now in optimization territory, not troubleshooting territory.

The "Performance-Critical" settings are more aggressive optimizations that trade power efficiency for raw performance. Power Management set to Maximum Performance, Determinism Slider set to Power, cTDP set to maximum, DF C-states disabled, APBDIS set to 1, Fixed SOC P-state at P0, ASPM disabled — these are settings that a hyperscaler or HPC cluster would apply as standard practice. The assistant is effectively telling the user: "You are running a GPU inference server, not a power-efficient cloud node. Configure it like one."

The "Particularly Important for Your Setup" section is where the assistant demonstrates the deepest contextual understanding. By highlighting DF C-states, APBDIS, Fixed SOC P-state, ASPM, and the BAR settings as specifically relevant to the cross-GPU latency problem, the assistant connects the generic BIOS optimization knowledge to the specific symptoms observed in this deployment. This is the hallmark of expert-level technical communication: the ability to map general knowledge onto a specific problem context.

Assumptions Embedded in the Message

The message makes several assumptions that are worth examining. First, it assumes that the user has access to the BIOS setup utility, either through a physical console or through IPMI/BMC remote access. The assistant mentions "ASMB11-iKVM on the ASUS" as the BMC interface, but does not verify that this interface is actually configured and accessible. In many real-world deployments, BMC configuration is itself a hurdle that requires separate effort.

Second, the message assumes that the BIOS version on the ASUS ESC8000A-E13 supports all the listed settings. AMD CBS (Customized BIOS Settings) menus vary significantly between BIOS versions and motherboard vendors. Some settings like "PCIe Ten Bit Tag" or "PCIE ARI Enumeration" may not appear in older BIOS versions, or may be labeled differently. The assistant does not provide fallback guidance for cases where specific settings are missing.

Third, the message assumes that the user is willing to reboot the server to change BIOS settings. In a production deployment, this is a significant operational decision. The assistant's framing — "Can you check these settings?" — is appropriately deferential, but the underlying assumption is that the potential performance improvement justifies the downtime.

Fourth, and most importantly, the message assumes that BIOS-level changes can meaningfully improve the cross-GPU latency problem. This is a reasonable assumption — disabling DF C-states and ASPM can reduce latency by several microseconds — but it is an untested hypothesis at this point. The 13-microsecond floor has multiple contributing factors: the staged DMA path through the host, the IOMMU translation overhead, the QEMU virtual PCIe topology, and the physical PCIe link characteristics. BIOS settings can only address the physical link characteristics and the CPU/SoC fabric behavior. The virtualization-induced components of the latency are unaffected by BIOS changes.

Potential Mistakes and Incorrect Assumptions

While the message is technically sound, there are a few points worth scrutinizing. The recommendation to set NUMA Nodes Per Socket (NPS) to NPS1 deserves particular attention. NPS1 configures the EPYC processor as a single NUMA node per socket, meaning all memory is equally accessible to all cores within that socket. This is indeed the standard recommendation for GPU workloads because it simplifies memory allocation and avoids the fragmentation that can occur with NPS2 or NPS4. However, the assistant's own earlier investigation (see [msg 385]) showed that the VM has 2 NUMA nodes (node 0 and node 1), and the physical GPUs are split across two NUMA domains (GPUs 0-3 on NUMA 1, GPUs 4-7 on NUMA 0). Setting NPS1 on the host would collapse this into a single NUMA domain, which could actually hurt performance if the GPU-to-NUMA affinity is being used by NCCL or CUDA to optimize memory placement. The NCCL topology XML that the assistant created in [msg 390] explicitly assigns GPUs to two NUMA nodes. If the host is changed to NPS1, that topology XML would become inaccurate. This is a subtle inconsistency that could lead to suboptimal NCCL routing decisions.

The recommendation to disable SMT (Simultaneous Multi-Threading) is presented as optional, but the reasoning is incomplete. The assistant notes that "since you're in a VM with 60 cores per socket, this trades core count for latency." However, in a virtualized environment, the relationship between host SMT and guest vCPU performance is complex. Disabling SMT on the host reduces the number of available hardware threads, which could limit the hypervisor's ability to schedule vCPUs efficiently. The assistant's cautious framing — "Optional — try with SMT on first" — is appropriate, but the recommendation could benefit from more nuance about the virtualization-specific tradeoffs.

The TSME (Transparent SME) = Disabled recommendation is also worth examining. TSME is AMD's hardware memory encryption feature, similar to Intel's Total Memory Encryption (TME). Disabling it removes the encryption latency for every memory access, which is beneficial for performance. However, in some security-conscious deployments (e.g., multi-tenant environments or compliance-sensitive workloads), memory encryption may be a requirement. The assistant does not acknowledge this tradeoff.

The Deeper Significance: BIOS as the Last Mile of Performance Tuning

What makes this message truly interesting is what it represents in the broader narrative of the opencode session. The team had spent hours investigating PCIe topology, modifying kernel parameters, migrating VM chipsets, fixing BAR allocation, creating NCCL topology files, and exploring hacky kernel modules — all in pursuit of enabling P2P DMA. They had hit a hard wall: the hardware topology made P2P impossible in a VM. The BIOS settings message represents the pivot from "trying to enable a feature that is fundamentally blocked" to "optimizing everything else that can be optimized."

This is a common pattern in systems engineering. When the primary path is blocked, the expert response is not to give up but to systematically identify and optimize every secondary path. The BIOS settings are the last mile of performance tuning — the layer that sits below the operating system, below the hypervisor, below the kernel, and below the application. If you have exhausted all software-level optimizations, the BIOS is where you go next.

The assistant's message also serves an important educational function. By explaining not just what settings to change but why each setting matters, the assistant is equipping the user with mental models that will serve them in future troubleshooting. The user learns that DF C-states affect cross-GPU latency, that ASPM adds wake-up overhead, that APBDIS locks the Data Fabric at peak performance, and that NPS configuration affects memory locality. This knowledge transfers to any future GPU server deployment, regardless of the specific hardware or virtualization setup.

Input Knowledge Required to Understand This Message

To fully appreciate this message, the reader needs a substantial foundation in several technical domains. Understanding the PCIe topology discussion requires familiarity with concepts like root complexes, IOMMU groups, ACS, BAR allocation, and PCIe link generation/width. Understanding the virtualization context requires knowledge of KVM, VFIO, QEMU machine types (i440FX vs Q35), and Proxmox's PCI passthrough mechanism. Understanding the BIOS settings requires familiarity with AMD EPYC architecture, including the Data Fabric, SoC fabric, NBIO (Northbridge I/O), DF C-states, P-states, and the AMD CBS menu structure. And understanding the performance implications requires knowledge of multi-GPU inference workloads, NCCL communication patterns, and the latency sensitivity of model parallelism.

The message assumes that the user possesses at least some of this knowledge — the user's question about "settings in bios that are highly warrated to set" suggests a baseline understanding that BIOS configuration matters for GPU server performance. But the assistant does not assume deep expertise; it explains each setting's purpose in accessible terms while still providing the precise BIOS menu paths that an experienced administrator would need.

Output Knowledge Created by This Message

The primary output of this message is a concrete, actionable BIOS configuration checklist for the ASUS ESC8000A-E13 with AMD EPYC 9335 and eight NVIDIA GPUs. This checklist is valuable in itself — it represents hours of research distilled into a single reference table. But the message also creates several less tangible but equally important outputs.

First, it creates a diagnostic framework. By identifying DF C-states and APBDIS as the settings most likely to affect cross-GPU latency, the assistant gives the user a way to test whether BIOS configuration is a meaningful contributor to the observed bottleneck. If changing these settings measurably improves throughput, the user knows that BIOS optimization is worthwhile. If it does not, the user knows that the remaining latency is purely virtualization overhead and must be accepted as a cost of the Proxmox environment.

Second, it creates a shared vocabulary for future communication. The user now knows terms like "DF C-states," "APBDIS," "ASPM," and "NPS" — and more importantly, understands what they mean in the context of their specific problem. This enables more precise communication in subsequent rounds of the conversation.

Third, it creates a boundary of possibility. By documenting what can be changed at the BIOS level and why each change matters, the assistant implicitly documents what cannot be fixed. No BIOS setting will enable P2P DMA across separate IOMMU groups. No BIOS setting will eliminate the virtualization overhead of the staged DMA path. The message helps the user develop a realistic understanding of what performance is achievable within the current hardware and virtualization constraints.

Conclusion

Message 402 is a masterclass in targeted technical communication. It arrives at a moment of potential frustration — after hours of work on P2P enabling has hit a fundamental hardware limitation — and redirects energy toward productive optimizations. It demonstrates that expert knowledge is not just about knowing the right answers, but about knowing which questions to ask and in what order. The assistant could have simply listed BIOS settings. Instead, it explained the reasoning, connected each setting to the specific observed symptoms, prioritized according to likely impact, and acknowledged the limits of what BIOS changes can achieve.

For anyone deploying multi-GPU inference servers on AMD EPYC platforms — whether virtualized or bare metal — this message serves as a valuable reference. The specific combination of DF C-states disabled, APBDIS enabled, Fixed SOC P-state at P0, ASPM disabled, and NPS1 configuration represents a baseline for maximum GPU communication performance. And the reasoning behind each setting provides a mental model that can be applied to future platforms and configurations.

In the end, the message captures a truth that every systems engineer eventually learns: when you cannot remove a bottleneck, optimize everything around it. The 13-microsecond latency floor may remain, but with proper BIOS configuration, the system can at least ensure that no avoidable latency is added on top of it.