Segment 39
This sub-session focused on deploying Qwen3.5-397B-A17B-NVFP4 on 8× RTX PRO 6000 Blackwell GPUs with a production-quality, accuracy-first configuration. The assistant upgraded the entire stack to nightly builds (PyTorch 2.12.0.dev20260307+cu130, flashinfer 0.6.5, SGLang latest main) and built sgl-kernel from source with SM120 support by applying catid's patches for CMake policy guards, CUDA 13 cccl include paths, and FA3 fallback, compiled with TORCH_CUDA_ARCH_LIST=12.0a. Extensive backend testing identified that flashinfer_cutlass for MoE and flashinfer_cudnn for FP4 GEMM produce correct output on SM120, while flashinfer_trtllm and flashinfer_cutedsl crash or produce NaN/garbage. A critical accuracy issue was resolved by forcing BF16 KV cache (--kv-cache-dtype bf16) instead of the checkpoint's default FP8 KV cache without proper scaling factors, ensuring high-precision long-context inference. The final production configuration was codified into a systemd service, achieving ~172 tok/s at single-request concurrency and over 2100 tok/s aggregate at high concurrency (C=32), with built-in MTP speculative decoding (NEXTN) showing no throughput gain on synthetic benchmarks.