Segment 6
In this sub-session, the assistant boosted GLM-5-NVFP4 throughput on 8x RTX PRO 6000 GPUs from ~880 tok/s to ~3,740 tok/s at 1024 concurrency by patching sglang's model_runner.py to include flashinfer_cutlass in the MoE autotune list, raising --max-running-requests to 1024, and enabling --disable-cuda-graph and --disable-radix-cache. However, GPU power draw stayed at ~250W out of 600W TDP, indicating underutilization. Investigation revealed that FlashInfer's allreduce fusion is disabled on SM120 because the underlying TRT-LLM kernels only support SM90/SM100. An attempt to patch flashinfer to add SM120 support led to poor performance (236 tok/s) due to synchronization issues, so the changes were reverted. The assistant then explored NCCL tuning and --num-continuous-decode-steps 4 without gains, and ended the session trying the flashinfer_trtllm MoE backend as an alternative path to better utilization.
The Blackwell Divide: From 880 to 3,740 tok/s and the Allreduce Fusion Wall