Chunk 39.0

This chunk focused on upgrading the entire stack to nightly builds and deploying Qwen3.5-397B-A17B-NVFP4 with a production-quality, accuracy-first configuration. The assistant followed the user's directive to "update all to nightly," upgrading PyTorch to `2.12.0.dev20260307+cu130`, flashinfer to `0.6.5`, and SGLang to the latest main branch. The critical technical achievement was building `sgl-kernel` from source with SM120 (Blackwell) support by applying catid's patches for CMake policy guards, CUDA 13 cccl include paths, and FA3 fallback, compiled with `TORCH_CUDA_ARCH_LIST=12.0a` to enable the required FP4 kernels. Extensive backend testing was conducted to find a configuration that produced correct output on SM120. The `flashinfer_trtllm` and `flashinfer_cutedsl` backends were found to be incompatible (crashing or producing NaN/garbage), while `flashinfer_cutlass` for MoE and `flashinfer_cudnn` for FP4 GEMM worked correctly. Built-in MTP speculative decoding (`NEXTN`) was successfully loaded but showed no throughput gain on synthetic benchmarks, confirming the baseline performance was already optimal. A critical accuracy issue was identified and resolved: the checkpoint's default FP8 KV cache was being applied without proper scaling factors, which would degrade long-context agentic coding tasks. The KV cache was explicitly forced to BF16 (`--kv-cache-dtype bf16`), providing ~1.57M tokens of high-precision cache—more than sufficient for the intended workload. The final production configuration was codified into the systemd service, achieving ~172 tok/s at single-request concurrency and over 2100 tok/s aggregate at high concurrency (C=32). The overarching themes of this session were aggressive, hands-on optimization (forking/modifying source code, exhaustive backend testing) balanced with a strong emphasis on output correctness over performance hacks. The assistant proactively addressed the FP8 KV cache accuracy risk, demonstrating a commitment to production-quality inference suitable for demanding agentic coding workflows on the 8× RTX PRO 6000 Blackwell setup.

The Nightly Pivot: Deploying Qwen3.5-397B-A17B-NVFP4 on Blackwell GPUs Through Strategic Stack Upheaval 3484 words

Message Articles

Subagent Sessions