Segment 2
In this sub-session, the assistant resolved the critical NaN crash during decode by using `--nsa-decode-backend trtllm` and `--nsa-prefill-backend trtllm`, which produced coherent output on the SM120 GPUs. Baseline benchmarks with 64 concurrent requests showed ~225 output tok/s and ~516 total tok/s. The assistant then tuned the server by increasing `--mem-fraction-static` to 0.92 and enabling CUDA graphs, which captured successfully without OOM, though throughput remained similar (~210–247 output tok/s). Various MoE runner backends were tested with comparable results. The user raised the possibility of expert parallelism to improve PCIe-bound performance, but analysis concluded full replication was impossible and EP8 offered no advantage due to small hidden size and similar communication volume. Investigation into cross-GPU latency from the Proxmox VM environment confirmed the system is a KVM/QEMU VM with no direct GPU peer-to-peer support, forcing all cross-GPU transfers through host memory. A bandwidth test showed ~32 GB/s for large transfers but only ~1 GB/s for small messages typical of all-reduce, indicating that virtualization overhead significantly contributes to the latency-limited throughput.