The 30x Gap: From Theoretical Maximum to Kernel-Level Bottleneck in Blackwell FP4 Inference

Introduction

In the high-stakes world of large language model inference optimization, few moments are as sobering as computing the theoretical maximum performance of your system and discovering that reality is a mere 3.4% of that number. This is precisely the situation the assistant and user found themselves in during Segment 10 of this opencode session. The GLM-5-NVFP4 model, a massive 744-billion-parameter Mixture-of-Experts (MoE) architecture deployed via SGLang across eight NVIDIA RTX PRO 6000 Blackwell Server Edition GPUs, was achieving approximately 10.36 tokens per second in single-stream inference — against a theoretical ceiling of 309 tok/s. The gap was not a factor of two or three; it was a factor of thirty.

What followed was one of the most intensive system-level optimization campaigns in the entire session. The assistant launched a comprehensive parallel audit via ten subagents, uncovered a litany of misconfigurations, executed a major kernel upgrade from 6.8.12 to 6.14.11, survived a post-reboot CUDA crisis caused by stale device major numbers in the LXC container's cgroup configuration, and ultimately built diagnostic tools that identified the true bottleneck: FP4 GEMM kernel overhead on the Blackwell SM120 architecture. This article synthesizes that journey, tracing the arc from shock to diagnosis to the hard-won understanding that some bottlenecks cannot be fixed by system tuning alone.

The Theoretical Wake-Up Call

The journey began with a user's pointed question: "for this model on this machine, gen5 pcie, 2 sockets, what's the maximum possible perf in theory, for single stream?" ([msg 1189]). The assistant had attempted to compute this earlier, but a Python script written inline on the command line had failed due to a zsh shell escaping issue — parentheses in print statements were interpreted as glob patterns, producing the cryptic error zsh:1: no matches found: (TP8): ([msg 1195]). This seemingly trivial technical hiccup delayed the analysis and forced a more robust approach: writing the script as a file rather than passing it inline ([msg 1199]).

When the calculation finally succeeded ([msg 1200]), the results were devastating. The theoretical maximum single-stream throughput was 309 tok/s, derived from a careful accounting of the model's weight footprint (2.86 GB per GPU after tensor-parallel sharding), HBM bandwidth (~1,800 GB/s per GPU), and AllReduce communication overhead. Against this ceiling, the actual measured performance was a mere 10.36 tok/s ([msg 1201]). The assistant's response captured the moment perfectly: "Fascinating results — a shocking 3.4% efficiency."

This number reframed the entire optimization problem. The gap was not 10% or 20% — it was 30x. No amount of incremental tuning at the application level could close such a chasm. Something fundamental was wrong with the system configuration, and finding it would require looking far beyond the SGLang server parameters and MoE backend selections that had occupied the team's attention for the preceding nine segments.

The Ten-Agent System Audit

The assistant's response to the 3.4% efficiency revelation was architecturally ambitious. Rather than investigating one subsystem at a time, it launched ten parallel exploration agents, each responsible for a specific domain of system configuration ([msg 1260] through [msg 1271]). This parallelism was enabled by opencode's task tool, which spawns independent subagent sessions that run to completion concurrently. The parent session would only resume once all ten had returned their findings.

The ten agents covered: UEFI/BIOS settings, LXC container configuration, NVIDIA driver version and CUDA configuration, kernel runtime knobs (CPU governor, NUMA balancing, C-states), kernel version compatibility, CUDA/GPU memory configuration, PCIe MaxReadReq size, power and thermal configuration, NCCL/P2P bandwidth, and SGLang/Python runtime configuration. Each agent was a deep dive into its domain, running diagnostic commands and interpreting the results against known best practices for GPU-accelerated ML serving.

The consolidated findings, delivered in [msg 1273], were remarkably structured. The assistant organized them into three categories: Confirmed Good (16 items verified as optimal, including IOMMU passthrough mode, PCIe Gen5 x16 links with no degradation, ReBAR enabled at 128GB, healthy GPU thermals at 28-31°C idle, and correct PCIe Relaxed Ordering/NoSnoop settings), P0 Actionable Fixes (high-impact changes that could be applied at runtime), and P1 Actionable Fixes (medium-impact changes requiring reboot).

The P0 findings included a litany of misconfigurations: the nvidia_peermem kernel module was not loaded (disabling GPUDirect Peer-to-Peer communication), PCIe MaxReadReq was stuck at 512 bytes instead of the optimal 4096, NUMA balancing was enabled (causing unnecessary page migration overhead), CPU C-states were allowing C2 sleep with 100μs wake latency, and the container's RLIMIT_MEMLOCK was capped at ~63GB, potentially constraining CUDA pinned memory allocations.

The P1 findings identified two systemic issues requiring reboot: the CPU scaling driver was using acpi-cpufreq instead of the superior amd_pstate driver, and the kernel version (6.8.12-9-pve) was outdated. The audit flagged that a 6.14 kernel could yield approximately 14% performance improvement for the AMD EPYC Turin processors powering the system.

Perhaps the most definitive finding was the resolution of a long-standing mystery about NVLink. Earlier in the conversation, a mystery PCI device at address 0000:52:00.0 had been suspected of being an NVSwitch. Agent 10 definitively identified it as an ASPEED AST2600 BMC/IPMI VGA controller — a standard server management controller, completely unrelated to NVIDIA interconnects. The verdict was unambiguous: NO NVLink. This finding had profound implications for optimization strategy, as it meant that custom allreduce fusion techniques requiring NVLink would never work on this hardware.

From Diagnosis to Action: Applying Runtime Fixes

The user's response to the audit findings was a model of concise engineering direction ([msg 1274]): "write down in system-improve.md, apply the changes runtime only for now, check that p2p works with some manual micro bench, then rerun inference benchmark." This 24-word instruction encapsulated a complete workflow: document, apply, validate, measure.

The assistant followed this instruction precisely. It wrote the comprehensive system-improve.md documentation ([msg 1276]), then applied all nine runtime fixes in a systematic blitz ([msg 1279]). The changes included disabling NUMA balancing, disabling C2 deep sleep states, disabling the NMI watchdog, disabling sched_autogroup, reducing swappiness to 10, tuning dirty page ratios, increasing network buffer sizes, and — crucially — bumping PCIe MaxReadReq from 512 to 4096 bytes on all eight GPUs and their root ports using the setpci command. Each fix was verified ([msg 1280], [msg 1281]), and the assistant confirmed that 9 of 10 identified issues had been addressed at runtime.

The verification of the MaxReadReq change proved unexpectedly challenging. When the assistant attempted to run a P2P bandwidth microbenchmark via SSH ([msg 1284]), the command failed with zsh:1: bad pattern — another shell escaping issue. The Python script used f-strings with curly brace format specifiers ({...:>12}), which Zsh on the remote machine interpreted as brace expansion patterns. This failure, while frustrating, revealed that the remote container used Zsh as its default shell — information that would inform future command construction. The assistant eventually ran the benchmark by writing the script to a file and executing it directly ([msg 1287]), confirming that P2P bandwidth was healthy at ~50 GB/s same-NUMA and ~37 GB/s cross-NUMA.

The Benchmark That Broke the Hypothesis

With the runtime fixes applied and verified, the assistant restarted the SGLang server and ran a comprehensive benchmark sweep ([msg 1293]). The results were devastating in their clarity: single-stream throughput was 10.30 tok/s — essentially identical to the pre-tuning baseline of 10.36 tok/s. The Mean TPOT (Time Per Output Token) was 95.59 ms, unchanged from before. After hours of system auditing, nine runtime fixes, and careful verification, the performance had not budged.

This negative result was itself a form of knowledge. It told the team that the bottleneck was not in the platform layer — not in CPU scheduling, not in PCIe read size, not in NUMA balancing, not in C-state wake-up latency. The runtime tuning had zero measurable effect because these system-level parameters were not the primary constraint. The bottleneck must be elsewhere: in the FP4 GEMM kernel itself, in the MoE routing logic, in the attention mechanism.

The user's response was immediate and strategic ([msg 1294]): "Try new kernel and the reboot-requiring fixes, feel free to reboot that proxmox node too." This directive authorized the most invasive change possible — upgrading the kernel on a production Proxmox virtualization host and rebooting it. The assistant formulated a plan ([msg 1295]): kernel upgrade to 6.14.11, persistent sysctl configuration, amd_pstate=active on the kernel command line, processor.max_cstate=1 to disable deep C-states, and a full reboot.

The Kernel Upgrade: A Multi-Step Orchestration

The kernel upgrade was one of the most meticulously orchestrated operations in the entire session, spanning messages 1295 through 1313. The first step was reconnaissance. The assistant checked the current kernel (6.8.12-9-pve), surveyed available kernels via apt-cache search proxmox-kernel-6.14, and confirmed that the latest available version was 6.14.11-5-bpo12-pve. It also verified the boot mechanism — Proxmox uses proxmox-boot-tool with systemd-boot on UEFI systems, with kernel cmdline stored in /etc/kernel/cmdline — ensuring that any boot-time parameters would be correctly applied.

The kernel installation itself revealed a critical dependency chain. When the assistant installed the proxmox-kernel-6.14.11-5-bpo12-pve-signed package, the DKMS (Dynamic Kernel Module Support) auto-build for the NVIDIA driver was skipped because kernel headers for the new kernel were not installed. This was a near-miss: if the assistant had not noticed the warning and proceeded directly to reboot, the system would have booted without GPU drivers, rendering all eight GPUs inaccessible. The assistant immediately installed the headers package (proxmox-headers-6.14.11-5-bpo12-pve) and manually triggered the DKMS build and install for NVIDIA driver version 590.48.01.

With the driver built, the assistant turned to persistent configuration. The kernel cmdline was updated to include amd_pstate=active, processor.max_cstate=1, and nmi_watchdog=0. Runtime sysctl parameters — NUMA balancing disabled, scheduler autogroup disabled, swappiness reduced, dirty ratios tightened, network buffers increased — were written to /etc/sysctl.d/99-gpu-compute.conf for persistence across reboots. The PCIe MaxReadReq fix, which resets on driver reload, was encoded in a systemd oneshot service (gpu-pcie-tuning.service) that fires after nvidia-persistenced.service at boot.

The assistant then performed a series of verifications. It checked the kernel boot order via proxmox-boot-tool kernel list, confirming that the 6.14 kernel was listed first and would be the default. It mounted the EFI partitions and inspected the actual boot entry files, confirming that the new cmdline had been correctly propagated. It verified DKMS status one final time, listing the five NVIDIA kernel modules (nvidia.ko, nvidia-uvm.ko, nvidia-drm.ko, nvidia-modeset.ko, nvidia-peermem.ko) built for the new kernel. And crucially, it regenerated the initramfs for the 6.14 kernel to include the uvm_disable_hmm=1 modprobe configuration — a critical workaround for a known CUDA initialization issue with the NVIDIA UVM driver's Heterogeneous Memory Management feature.

The Reboot and the Post-Reboot Crisis

The reboot command was issued in message 1313, accompanied by a summary of the six major changes about to take effect. The assistant then waited — a 150-second polling loop that checked SSH connectivity every 10 seconds, with a 5-second connection timeout and a 10-minute maximum wait. The host came back cleanly, and the verification in message 1316 confirmed that every tuning parameter had taken effect: kernel 6.14.11, amd-pstate-epp driver active, only C0 and C1 C-states available, all eight GPUs detected in P0 state, MaxReadReq set to 4096, and all sysctls applied.

But when the assistant turned to the LXC container that hosted the actual ML workload, disaster struck. The container had not auto-started — the assistant's polling loop in message 1318 ran for 300 seconds without success. The user intervened with a three-word correction: "need to start it manually" ([msg 1319]). The assistant checked the container status via pct list, found VMID 129 in "stopped" state, and started it with pct start 129.

With the container running, the assistant attempted to run a P2P bandwidth benchmark — and it crashed immediately with a CUDA initialization error. The error was CUDA_ERROR_UNKNOWN (error code 999), a different and more opaque failure than the error 3 (CUDA_ERROR_NOT_INITIALIZED) that had been resolved earlier by disabling HMM. The assistant's debugging process was a masterclass in systematic troubleshooting.

The first hypothesis — that the uvm_disable_hmm modprobe parameter had not applied on the new kernel — was tested and rejected when /sys/module/nvidia_uvm/parameters/uvm_disable_hmm returned Y. The assistant then confirmed that nvidia-smi worked inside the container (ruling out a complete driver failure) but that PyTorch's torch.cuda.is_available() returned False. A direct cuInit(0) call via the CUDA Python bindings confirmed the failure at the lowest driver API level.

The breakthrough came when the assistant compared the host's /proc/devices output against the LXC container's cgroup configuration. The kernel upgrade had changed the device major numbers for NVIDIA drivers: nvidia-uvm had moved from major 504 to 509, and nvidia-caps from 507 to 237. But the container's cgroup rules — stored in /etc/pve/lxc/129.conf — still referenced the old numbers. The cgroup v2 device controller was silently blocking the container's access to the UVM and caps devices, causing cuInit to fail with a generic unknown error.

The fix was straightforward once the root cause was identified. The assistant stopped the container, edited the configuration to replace the stale cgroup rules with the correct major numbers (509 for nvidia-uvm, 237 and 238 for nvidia-caps), and restarted the container. The verification in message 1341 confirmed success: torch.cuda.is_available() returned True, torch.cuda.device_count() returned 8, and a tensor could be created on cuda:0.

This post-reboot CUDA failure is a textbook example of the hidden dependencies that can break containerized GPU workloads after system updates. The device major numbers for NVIDIA drivers are not standardized — they are assigned dynamically by the kernel at module load time. A kernel upgrade can change these numbers, and any container configuration that hard-codes them will break. The assistant's ability to diagnose and fix this issue quickly was essential to keeping the optimization campaign on track.

Building Diagnostic Tools: Decomposing the 97ms Decode

With CUDA restored and the system fully tuned, the assistant turned to the next phase of the investigation. The P2P benchmark was re-uploaded (the original had been lost when the container's /tmp was cleared during restart) and executed. The results showed healthy P2P bandwidth of approximately 50 GB/s between same-NUMA GPUs, confirming that the PCIe and NVLink infrastructure was working correctly.

But the fundamental question remained: why was single-stream throughput stuck at 10 tok/s when the theoretical maximum was 309 tok/s? The assistant had eliminated system-level misconfigurations as the primary cause. The kernel was modern, the CPU governor was optimal, C-states were restricted, PCIe was tuned, and P2P bandwidth was healthy. The bottleneck had to be elsewhere.

The user redirected the investigation toward single and dual-stream performance, and the assistant built a diagnostic tool to measure the latency of individual decode components. This tool decomposed the ~95ms per-token decode time into its constituent parts: simulated BF16 GEMMs, AllReduces, and the remaining overhead. The results were striking: the simulated BF16 GEMMs and AllReduces accounted for only 8.9ms of the 95ms decode time. The remaining ~86ms — over 90% of the latency — was attributable to FP4 GEMM kernel overhead, MoE routing, and attention.

This was the critical finding. The bottleneck was not in the system configuration, the CPU governor, the PCIe tuning, or any of the changes made during the kernel upgrade. It was in the FP4 GEMM kernels themselves, which were inefficient on the SM120 architecture of the Blackwell GPUs. The theoretical maximum of 309 tok/s was unattainable not because of system misconfiguration, but because the CUDA kernels for FP4 matrix multiplication were not optimized for this hardware generation.

Earlier benchmark data had shown that FP4 GEMMs at the small sizes characteristic of per-expert computation (M=1, N=256, K=6144) achieved only 0.3 TFLOPS, or 0.02% of the GPU's 1,850 TFLOPS peak. This is the fundamental architectural mismatch: the Blackwell SM120 architecture, with its 100KB shared memory limit and lack of advanced features like TMEM, cannot efficiently execute the tiny matrix multiplications that MoE models require. The session concluded with the creation of a deeper analysis tool specifically designed to measure these remaining bottlenecks — the FP4 GEMM kernel overhead, MoE routing latency, and attention cost. This tool would provide the granular breakdown needed to determine whether the 30x gap could be closed through software optimization or whether it represented a fundamental limitation of the hardware-software combination.

The Broader Lessons

This chunk of the optimization campaign offers several enduring lessons for anyone working on GPU-accelerated ML inference.

First, theoretical maximums matter. The 309 tok/s calculation was not an academic exercise — it was a diagnostic forcing function that revealed the true scale of the problem. Without it, the team might have continued optimizing within their existing framework, celebrating incremental gains while remaining 30x from the physical limit.

Second, systematic auditing pays dividends. The ten-agent parallel audit was architecturally ambitious but methodologically sound. It covered every layer of the system simultaneously, producing a comprehensive picture that no sequential investigation could have matched. The "Confirmed Good" list was as valuable as the "Actionable Fixes" list — knowing what is NOT broken prevents wasted effort on phantom issues.

Third, negative results are knowledge. The benchmark that showed no improvement after runtime tuning ([msg 1293]) was not a failure — it was a crucial piece of evidence that redirected the investigation toward the kernel level. The assistant's willingness to confront and report this negative result honestly is a hallmark of scientific rigor.

Fourth, the bottleneck is often where you least expect it. The team spent days optimizing communication (AllReduce, MSCCLPP, SBO), scheduling (CUDA graphs, continuous decode steps), and system configuration (CPU governor, C-states, PCIe parameters). In the end, the bottleneck was in the FP4 GEMM kernel itself — the fundamental matrix multiplication operation that the model spends most of its time on. The 8.9ms measured for BF16 GEMMs + AllReduce versus the 95ms total decode time made this unambiguous.

Fifth, containerized GPU workloads have hidden fragility. The post-reboot CUDA failure, caused by stale device major numbers in the LXC cgroup configuration, is a failure mode that is rarely documented but frequently encountered in practice. The assistant's ability to diagnose and fix this issue — tracing from "CUDA doesn't work" to "device major numbers changed" to "update LXC config" — demonstrates the cross-layer expertise required to manage GPU-accelerated infrastructure.

Sixth, kernel upgrades on GPU-accelerated systems require meticulous preparation. The near-miss with DKMS auto-build skipping due to missing kernel headers is a classic pitfall. The assistant's careful verification — checking DKMS status, regenerating initramfs, confirming boot order, and testing each component — prevented what could have been a catastrophic boot failure. The lesson is clear: never assume DKMS auto-build will work; always verify that the NVIDIA kernel modules are built and installed for the new kernel before rebooting.

Conclusion

The arc from theoretical maximum calculation to kernel-level bottleneck identification represents a complete diagnostic cycle. It began with a number — 309 tok/s — that seemed impossibly far from reality. It proceeded through systematic investigation, ruling out one potential cause after another. It survived a kernel upgrade, a post-reboot crisis, and the disappointment of runtime fixes that produced no measurable improvement. And it culminated in a precise understanding: the 30x gap is not in communication, not in system configuration, not in scheduling overhead, but in the FP4 GEMM kernel's inability to efficiently execute the small matrix multiplications that MoE inference requires.

This understanding is itself a form of progress. The team now knows exactly where to focus their optimization efforts: on the FP4 GEMM kernel, on increasing the effective batch size per expert, on finding ways to make SM120's limited shared memory and compute resources work efficiently for the tiny GEMM shapes that 256-expert MoE models demand. The 30x gap is not a wall — it is a target, and the diagnostic work documented in this segment has provided the map for reaching it.

The journey also underscores a deeper truth about ML infrastructure optimization: the most impactful discoveries often come not from applying a known fix, but from the systematic process of eliminating possibilities until the true bottleneck reveals itself. The assistant's methodical approach — compute the theoretical maximum, audit every layer, apply fixes, measure impact, iterate — transformed a shocking 3.4% efficiency into a precise, actionable diagnosis. That diagnosis may not have closed the gap, but it has pointed the way forward with clarity and confidence.