The 30x Performance Gap: A Systematic Investigation of GPU Inference Bottlenecks

Introduction

When deploying a large language model for inference, the gap between theoretical peak performance and actual throughput can be a source of profound frustration—or a roadmap to deep system understanding. In this opencode coding session, a team deploying the GLM-5-NVFP4 model across eight NVIDIA RTX PRO 6000 Blackwell GPUs discovered that their system was achieving only 10.36 tokens per second against a theoretical maximum of 309 tokens per second—a staggering 3.4% efficiency ratio. This article synthesizes the work across an entire chunk of that session, tracing the investigation from the initial shock of the performance gap through a multi-layered diagnostic effort that spanned system audits, kernel upgrades, container forensics, and custom profiling tools.

The chunk represents a critical inflection point in the broader deployment effort. Rather than accepting the low throughput or guessing at causes, the team embarked on a methodical elimination of potential bottlenecks, working through layers of the system stack from the CPU governor and kernel version down to the LXC container configuration and the individual GPU kernel operations. The investigation is a masterclass in systematic performance debugging, demonstrating how to combine theoretical analysis, system auditing, container forensics, and targeted benchmarking to isolate the root cause of a massive performance shortfall.

The Performance Gap: Computing the Theoretical Ceiling

The investigation began with a fundamental question: what should this system be capable of? Before any optimization can begin, one must establish a realistic upper bound on performance. The team computed the theoretical maximum single-stream throughput by analyzing the hardware's capabilities: eight RTX PRO 6000 Blackwell GPUs, each with substantial compute capacity, connected via PCIe to a dual AMD EPYC 9335 processor with 128 threads and 460 GB of RAM.

The calculation yielded 309 tokens per second—a figure derived from the GPU's peak FLOPs, the model's arithmetic intensity, and the memory bandwidth available for parameter loading. Against this ceiling, the actual 10.36 tok/s represented a utilization rate so low that it demanded immediate investigation. A 30x gap cannot be explained by any single factor; it must be the cumulative result of multiple bottlenecks across the system stack.

This theoretical computation itself was a significant achievement. It required detailed knowledge of the GPU architecture (the Blackwell generation's tensor core performance), the model's characteristics (GLM-5-NVFP4's parameter count, layer structure, and precision format), and the system's memory hierarchy (HBM bandwidth, PCIe bandwidth, and host memory bandwidth). The result served as both a wake-up call and a north star: any optimization that closed the gap would be measured against this 309 tok/s target.

The System Audit: Ten Agents in Parallel

With the performance gap quantified, the team launched a comprehensive parallel system audit. Ten independent agents were dispatched simultaneously, each probing a different aspect of the system configuration. This parallel approach was essential—with so many potential causes, sequential investigation would have taken days. The agents checked everything from CPU power management to PCIe configuration, from kernel parameters to NUMA balancing settings.

The audit uncovered a cascade of critical misconfigurations:

Suboptimal CPU governor. The system was running the acpi-cpufreq CPU frequency scaling driver instead of the modern amd_pstate driver. For AMD EPYC processors, amd_pstate provides more granular frequency control and better performance under sustained loads. The acpi-cpufreq driver, while functional, cannot match the responsiveness and efficiency of the platform-specific driver.

Outdated kernel. The system was running kernel 6.8.12, which, while recent, lacked important optimizations and fixes for the AMD EPYC 9335 platform. More critically, it may have lacked the necessary support for the amd_pstate driver and other performance features.

Enabled NUMA balancing. The kernel's automatic NUMA balancing was active, which can cause memory pages to migrate between NUMA nodes during execution. For GPU inference workloads, where memory access patterns are stable and predictable, NUMA balancing introduces unnecessary overhead and can degrade performance by moving pages that should remain pinned to a specific node.

Deep CPU C-states. The processor was allowed to enter deep idle states (C-states), which introduce significant latency when the CPU must wake up to service GPU interrupts or dispatch new work. For inference workloads with frequent GPU kernel launches, even microsecond-scale wake-up latencies can accumulate into significant overhead.

Suboptimal PCIe MaxReadReq. The PCIe Maximum Read Request size was stuck at 512 bytes instead of the optimal 4096 bytes. This is a classic performance trap: a small MaxReadReq limits the amount of data the GPU can read from host memory in a single PCIe transaction, increasing the number of transactions needed and reducing effective bandwidth. For a model that requires frequent host-to-device transfers, this setting alone could account for a significant fraction of the performance gap.

Each of these findings was individually impactful, but their combination was devastating. The system was essentially operating with multiple layers of performance sandbagging, each layer adding its own tax on throughput.

The Kernel Upgrade: From 6.8.12 to 6.14.11

The most impactful remediation was a major kernel upgrade. The team moved from kernel 6.8.12 to 6.14.11, a significant leap that brought with it the amd_pstate=active driver, improved NUMA handling, and numerous other performance optimizations. The upgrade was not undertaken lightly—kernel upgrades on production systems carry risk, especially when NVIDIA drivers and CUDA toolkits must remain compatible.

The new kernel was booted with specific parameters designed to maximize inference performance:

• amd_pstate=active to enable the modern CPU frequency scaling driver
• processor.max_cstate=1 to prevent deep CPU idle states
• numa_balancing=disable to stop automatic NUMA page migration These parameters represented a targeted configuration for latency-sensitive GPU workloads. The max_cstate=1 setting, in particular, was a deliberate trade-off: it prevents the CPU from entering deep sleep states, which increases idle power consumption but eliminates the wake-up latency that can stall GPU kernel launches.

The Post-Reboot Crisis: CUDA Fails in the Container

The kernel upgrade required a full system reboot—always a tense moment in infrastructure operations. When the system came back online, a critical problem emerged: CUDA had stopped working inside the LXC container. The NVIDIA GPUs were visible on the host, but inside the container, CUDA initialization failed.

The root cause was subtle but instructive. The new kernel had reassigned the major device numbers for NVIDIA devices. In Linux, device nodes are identified by major and minor numbers; the NVIDIA driver registers character devices with specific major numbers (typically 195 for nvidiactl, and higher numbers for individual GPU devices). When the kernel was upgraded, the device major numbers changed, but the LXC container's cgroup configuration still referenced the old numbers.

The LXC container configuration file (/etc/pve/lxc/129.conf) contained lxc.cgroup2.devices.allow entries that explicitly permitted access to character devices with specific major numbers. After the kernel upgrade, these numbers no longer matched the actual NVIDIA devices, so the container's cgroup device allowlist effectively blocked GPU access.

The fix was straightforward once the root cause was identified: update the device major numbers in the container configuration to match the new kernel's device map. This restored full GPU functionality inside the container. The incident served as a valuable lesson in the fragility of containerized GPU setups—the device passthrough configuration is not abstracted away by the container runtime; it must be explicitly maintained and updated when the host kernel or driver changes.

The LXC Container Forensics: A Deep Dive

With the system-level fixes applied and CUDA operational again, the team turned their attention to the container configuration itself. A dedicated subagent session was spawned to perform a forensic audit of the LXC container's configuration, examining every setting that could affect GPU compute performance [1][2][3][4].

The investigation was exhaustive, covering 13 distinct areas of concern:

1. Container configuration file (/etc/pve/lxc/129.conf): The canonical source of truth for the container's definition, including CPU and memory allocation, device passthrough rules, and security settings.
2. Cgroup settings: Both host-side and container-side cgroup limits were checked to ensure no artificial constraints on memory, CPU, or device access.
3. Privileged vs. unprivileged mode: The container was confirmed to be privileged (unprivileged: 0), which is essential for full GPU device access. Unprivileged containers have severely restricted device access that can interfere with NVIDIA driver operations.
4. Seccomp profile: The seccomp (secure computing mode) filter was examined to ensure it wasn't blocking syscalls required by the NVIDIA driver. The container showed Seccomp: 2 (filtered mode) with one filter active, but the default Proxmox seccomp profile was confirmed to be permissive enough for CUDA operations [3].
5. AppArmor profile: AppArmor was found to be in "unconfined" mode for the container's processes, meaning no additional restrictions were being enforced beyond the standard LXC profile [2][3].
6. Seccomp status from inside the container: Cross-validated against the host-side check to ensure consistency.
7. Cgroup memory and CPU limits from inside: Confirmed that memory.max and cpu.max were both set to "max" (no hard limits beyond the container's allocation).
8. Ulimits: The ulimit -a output revealed a critical finding: the maximum locked memory (RLIMIT_MEMLOCK) was capped at approximately 63 GB [3]. While generous, this could become a constraint for large model inference with eight GPUs requiring extensive pinned memory for DMA transfers.
9. Capabilities: The full capability mask was decoded from hex (0x000001fcfdfcffff) into named capabilities, confirming that CAP_IPC_LOCK (essential for GPU pinned memory) was present [4].
10. Hugepages: HugePages_Total was 0 and Transparent Hugepages was in madvise mode—assessed as neutral for GPU workloads, since NVIDIA drivers don't typically require explicit hugepage reservations [3].
11. NUMA topology: The container could see both NUMA nodes, but a significant memory imbalance was discovered: NUMA node 0 had only 2.2 GB free out of 257 GB total, while node 1 had 22.5 GB free [3]. This imbalance could cause DMA buffer allocations to spill to the remote NUMA node, incurring a 3.2× latency penalty.
12. Cpuset restrictions: The cpuset effective masks showed no restrictions—the container could use all available CPUs and memory nodes.
13. NVIDIA device permissions: All eight GPUs were passed through with crw-rw-rw- (666) permissions—wide open for any process inside the container [3]. The container audit produced a ranked list of issues: the memlock limit was flagged as HIGH severity, the NUMA memory imbalance as MEDIUM, and the open files limit (1024) as LOW. Specific remediations were provided for each, including adding lxc.prlimit.memlock: unlimited to the container configuration [3].

The Diagnostic Tool: Measuring Decode Latency Components

With the system and container configurations addressed, the team turned to the most challenging question: what was actually consuming the 95 milliseconds per decode step? They built a custom diagnostic tool to measure the latency of individual decode components, breaking down the 95ms into its constituent parts.

The tool simulated specific operations to isolate their costs:

• BF16 GEMMs (general matrix multiplications in bfloat16 precision)
• AllReduce communication operations
• FP4 GEMM kernel overhead
• MoE routing latency
• Attention computation The results were revelatory. The simulated BF16 GEMMs and AllReduce operations accounted for only 8.9 milliseconds of the total 95ms decode time. This meant that the overwhelming majority of the latency—over 86 milliseconds—was being consumed by other components. The diagnostic tool pointed the finger squarely at three culprits: 1. FP4 GEMM kernel overhead: The custom FP4 (4-bit floating point) kernels used for the GLM-5-NVFP4 model's quantized operations were the primary suspect. These kernels, while necessary for the model's novel precision format, appeared to have significantly lower utilization than the standard BF16 kernels. 2. MoE routing: The Mixture-of-Experts routing logic, which determines which expert networks to activate for each token, was consuming a disproportionate share of the decode time. This is a common bottleneck in MoE models, as the routing computation involves top-k selection and load balancing that doesn't map cleanly to GPU tensor cores. 3. Attention: The attention mechanism, particularly for long sequences, was another significant contributor. The quadratic complexity of attention means that even with optimizations like FlashAttention, it can dominate decode time for large models. This finding was a critical turning point. The team had systematically ruled out system-level misconfigurations (CPU governor, kernel version, C-states, PCIe settings), container-level issues (cgroup limits, seccomp, AppArmor, device permissions), and communication overhead (AllReduce). What remained was the model's own computational profile—the kernels and algorithms that define the GLM-5-NVFP4 architecture.

Synthesis: The Layered Investigation Strategy

The work in this chunk represents a masterclass in systematic performance debugging. The investigation strategy can be understood as a series of concentric layers, each eliminating a class of potential bottlenecks:

Layer 1: Theoretical ceiling. Compute the maximum possible performance based on hardware specifications. This establishes the target and quantifies the gap.

Layer 2: System configuration. Audit CPU governor, kernel version, NUMA settings, C-states, PCIe parameters. Fix all identified issues. This layer addresses the "is the OS configured correctly?" question.

Layer 3: Container configuration. Audit LXC settings, cgroup limits, seccomp, AppArmor, capabilities, device permissions, NUMA topology. Fix all identified issues. This layer addresses the "is the container configured correctly?" question.

Layer 4: Communication overhead. Measure AllReduce and other inter-GPU communication costs. This layer addresses the "is the network/PCIe the bottleneck?" question.

Layer 5: Kernel efficiency. Measure individual kernel latencies (GEMMs, attention, routing). This layer addresses the "are the GPU kernels running at peak efficiency?" question.

By Layer 5, the investigation had narrowed the bottleneck to the FP4 GEMM kernels, MoE routing, and attention—the model's own computational core. The team concluded the chunk by creating a deeper analysis tool specifically designed to measure these remaining bottlenecks with greater precision.

Lessons for ML Infrastructure Engineers

Several broader lessons emerge from this investigation:

Theoretical ceilings matter. Without knowing the upper bound, you cannot distinguish between a 30% performance gap (annoying) and a 30x gap (crisis). Computing theoretical throughput forces you to understand your hardware's capabilities and your model's requirements at a deep level.

Systematic elimination works. The layered approach—ruling out entire categories of bottlenecks before diving deeper—is far more efficient than chasing individual symptoms. Each layer that passes inspection narrows the search space dramatically.

Container configuration is a first-class concern. The LXC container audit revealed that even with GPUs visible and CUDA initializing, subtle configuration issues (memlock limits, NUMA imbalance, seccomp filters) can silently degrade performance. Container configuration should be treated as a performance-critical parameter, not an operational afterthought [1][2][3][4].

Kernel upgrades have consequences. The post-reboot CUDA failure, caused by stale NVIDIA device major numbers in the LXC cgroup configuration, is a cautionary tale. Any change to the host kernel or driver stack requires re-verification of container device passthrough configurations.

Custom kernels need custom profiling. The FP4 GEMM kernel overhead was invisible to standard profiling tools because it wasn't a communication bottleneck or a system configuration issue—it was a kernel efficiency problem specific to the model's novel precision format. Custom diagnostic tools were required to isolate it.

Conclusion

The chunk of work synthesized in this article represents a turning point in the GLM-5-NVFP4 deployment effort. The team began with a shocking 30x performance gap and, through methodical investigation across five layers of the system stack, narrowed the bottleneck to the model's own computational core. System-level fixes—CPU governor, kernel upgrade, PCIe configuration—closed some of the gap. Container configuration adjustments addressed another portion. Communication profiling ruled out AllReduce as a primary culprit. And finally, custom diagnostic tools identified the FP4 GEMM kernels, MoE routing, and attention as the remaining targets for optimization.

The investigation is not complete—the deeper analysis tool created at the end of this chunk will need to be deployed and its results interpreted. But the trajectory is clear: the team has transformed a vague "our inference is slow" problem into a precise, measurable, and actionable set of hypotheses about specific kernel operations. That transformation—from mystery to measurement—is the essence of performance engineering.

For anyone managing GPU inference infrastructure, the methodology demonstrated here is directly applicable. Compute your theoretical ceiling. Audit every layer of your system. Build tools that measure what matters. And never stop asking: what is consuming the time, and why?