Segment 10
This sub-session began with computing the theoretical maximum single-stream throughput (309 tok/s) and comparing it to actual performance (10.36 tok/s), revealing a 3.4% efficiency gap. To address this, a comprehensive system audit was launched via 10 agents, uncovering several misconfigurations including suboptimal CPU governor (acpi-cpufreq instead of amd_pstate), outdated kernel (6.8.12), enabled NUMA balancing, deep CPU C-states, and suboptimal PCIe MaxReadReq (512 bytes instead of 4096). All runtime fixes were applied and a major kernel upgrade to 6.14.11 with amd_pstate=active and processor.max_cstate=1 was performed, requiring a reboot. Post-reboot, CUDA failed in the LXC container due to stale NVIDIA device major numbers in the cgroup configuration, which was quickly fixed by updating to match the new kernel's device map. Initial benchmarks on the new kernel were interrupted as the user refocused on low single/dual-stream performance. A diagnostic tool was built to measure latency of individual decode components, revealing that simulated BF16 GEMMs and AllReduces accounted for only 8.9ms of the 95ms decode time, pointing to FP4 GEMM kernel overhead, MoE routing, and attention as the primary bottlenecks. A deeper analysis tool was then created to specifically measure these remaining bottlenecks.
The 30x Gap: From Theoretical Maximum to Kernel-Level Bottleneck in Blackwell FP4 Inference