Segment 7
In this sub-session, the assistant first confirmed that the GLM-5-NVFP4 model is compute-bound rather than communication-bound by benchmarking TP4+PP2, which was 2× slower than TP8, ruling out allreduce latency as the primary bottleneck. A deep investigation into FP4 GEMM kernel efficiency on SM120 revealed that GPUs draw only ~235W out of 600W TDP, CUTLASS kernels plateau at ~1,300 TFLOPS only for large matrices, and the 99KB shared memory limit prevents larger tile configurations. Through systematic tuning—raising `--max-running-requests` to 2048 and setting `--num-continuous-decode-steps 8`—the assistant achieved a 28% throughput improvement at 2048 concurrency, reaching 2,095 output tok/s and 4,151 total tok/s. Multiple research agents explored expert parallelism, piecewise CUDA graphs, MSCCLPP allreduce, single-batch overlap, L2 cache pinning, persistent grouped GEMM kernels, and FP4 structured sparsity. The assistant synthesized these findings into a ranked optimization plan and began documenting each approach as `glb5improvement-xx.md` files, starting with piecewise CUDA graphs.
From 235 Watts to a Ranked Roadmap: The GLM-5-NVFP4 Optimization Campaign on Blackwell GPUs