Segment 1
In this sub-session, the assistant verified that all 8 RTX PRO 6000 Blackwell GPUs were visible and focused on deploying the GLM-5-NVFP4 model with sglang. Initial installation of sglang 0.5.8.post1 was replaced by a build from the main branch to include the critical SM120 shared memory fix (PR #14311). The model's glm_moe_dsa architecture required upgrading Transformers to 5.2.0. The server was launched with recommended parameters (tensor parallelism 8, FP4 quantization, flashinfer attention backends). Despite successful model loading and CUDA graph capture, the server repeatedly crashed during decode with a device-side assert triggered error caused by NaN/Inf values in the probability tensor. The assistant traced this to two likely sources: a warning about DeepGemm being enabled with an incompatible checkpoint scale format (ue8m0), and a Transformers 5.2.0 warning about potential RoPE parameter incompatibilities. Multiple configuration attempts were made—switching attention backends (triton, flashmla_sparse), forcing --fp8-gemm-backend cutlass, disabling CUDA graphs, and trying trtllm NSA backends—but the NaN crash persisted. The assistant consulted the local research repository (FINDINGS.md), which documented previous successful NVFP4 deployments on the same hardware and highlighted the DeepGemm scale format issue as a known problem on Blackwell. The latest attempt uses --kv-cache-dtype auto, --disable-cuda-graph, and explicit NSA backends (trtllm), and was still loading at the end of the sub-session.
Through the Blackwell Storm: Deploying GLM-5-NVFP4 from NaN Crashes to Production Benchmarking