The Nightly Pivot: A Calculated Risk to Unlock Blackwell FP4 Inference

In the sprawling, multi-session effort to deploy large language models on an 8× NVIDIA RTX PRO 6000 Blackwell GPU setup, few individual messages carry as much strategic weight as the assistant's response at message index 5859. This is the moment where the entire software stack is deliberately destabilized — and rebuilt — in pursuit of a working Qwen3.5-397B-A17B-NVFP4 deployment. The message is short, but it encodes a critical technical decision that would determine the success or failure of the entire deployment.

The Context: A Model That Produces Garbage

To understand why this message matters, we must first understand the crisis that precipitated it. The assistant had just finished a grueling multi-week effort deploying Kimi-K2.5 INT4 with EAGLE-3 speculative decoding, achieving impressive throughput numbers. The team then pivoted to Qwen3.5-397B-A17B-NVFP4, a newer, smaller (223 GB vs 547 GB) NVFP4-quantized model that promised faster loading and Blackwell-native performance.

The first attempt was a disaster. When the assistant started the server and sent a test prompt, the model responded with repeated exclamation marks — pure garbage output (see [msg 5842]). A second test returned null content with a finish_reason of "NaN happened" ([msg 5843]). The FP4 kernels were producing numerical garbage on the SM120 (Blackwell) hardware.

The root cause was identified via catid's gist ([msg 5853]): the default auto MoE runner backend selected a kernel incompatible with SM120, producing NaN values. The fix required explicitly pinning --moe-runner-backend flashinfer_cutlass and --fp4-gemm-runner-backend flashinfer_cudnn. But there was a deeper concern: the pre-built sgl-kernel 0.3.21+cu130 wheel might not contain SM120-specific FP4 kernels at all.

The User's Directive: "Update All to Nightly"

The user's instruction at [msg 5857] was concise and decisive: "Update all to nightly, the patches are more recent than our build." This was followed by [msg 5858]: "Continue if you have next steps, or stop and ask for clarification if you are unsure how to proceed."

This instruction is remarkable for what it implies. The assistant had just spent considerable effort documenting a carefully curated stable stack: PyTorch 2.9.1+cu130, sgl-kernel 0.3.21+cu130, flashinfer 0.6.4. The assistant had even explicitly warned that "nightly 2.12.0+cu130 has ABI incompatibility with sgl-kernel 0.3.21" (see [msg 5856]). The user, recognizing that catid's working gist was built against nightly PyTorch, overrode this caution and directed a full upgrade.

The Message: A Plan Takes Shape

The assistant's response at [msg 5859] is the acceptance and execution plan for this directive:

I'll update the entire stack to PyTorch nightly and rebuild sgl-kernel from source with SM120 support. The rationale is that the pre-built sgl-kernel 0.3.21+cu130 was built against older torch, and catid's gist (which works) uses nightly. Let me plan this out.

This is followed by a structured todowrite with three high-priority tasks:

  1. Upgrade PyTorch to nightly (2.12.0+cu130 or latest) — priority high
  2. Upgrade flashinfer to latest nightly/compatible version — priority high
  3. Build sgl-kernel from source with SM120 FP4 support (TORCH_CUDA_ARCH_LIST=12.0a) — priority high
  4. Apply catid's CMakeLists.txt patches to sgl-kernel — listed but truncated in the view

The Reasoning: Why This Is the Correct Move

The assistant's reasoning reveals a sophisticated understanding of the dependency chain. The key insight is that the ABI incompatibility between nightly PyTorch and sgl-kernel 0.3.21 exists only because the pre-built wheel was compiled against an older torch version. If you build sgl-kernel from source against the same nightly PyTorch, the ABI mismatch disappears. The constraint was not the torch version itself, but the pre-built binary.

This is a classic systems-engineering tradeoff: stability versus capability. The stable stack (torch 2.9.1) was safe but couldn't run the model correctly. The nightly stack (torch 2.12.0) was risky but was the only path demonstrated to work on SM120. The assistant correctly identifies that the risk can be mitigated by rebuilding sgl-kernel from source.

Assumptions Embedded in the Decision

Several assumptions underpin this plan, each carrying its own risk:

  1. That catid's patches are complete and correct for this hardware. The gist was written for a similar but not identical setup. The assistant assumes the CMake policy guards, CUDA 13 cccl include paths, and FA3 fallback will apply cleanly to the SGLang main branch's sgl-kernel source.
  2. That the nightly PyTorch won't introduce new incompatibilities with SGLang. The SGLang main branch is itself rapidly evolving. The assistant is betting that the latest SGLang code is compatible with the latest PyTorch nightly.
  3. That the build won't exhaust system memory. Earlier in the session (segment 0), the assistant had to reduce MAX_JOBS from 128 to 20 to avoid memory exhaustion during flash-attn compilation. Building sgl-kernel with TORCH_CUDA_ARCH_LIST=12.0a and FP4 support could be similarly memory-intensive on a machine with no swap.
  4. That the FP4 backend flags from catid's gist will work with the newly built sgl-kernel. The flashinfer_cutlass MoE backend and flashinfer_cudnn FP4 GEMM backend were identified as the working combination, but this was not yet verified on this specific hardware.

Input Knowledge Required

A reader fully understanding this message needs familiarity with:

Output Knowledge Created

This message produces:

  1. A clear, prioritized execution plan with four high-level tasks that will guide the next several rounds of tool calls.
  2. A documented rationale for the nightly pivot, preserving the reasoning for future reference.
  3. A structured todowrite that the assistant can use to track progress and report status.
  4. An implicit commitment to the "build from source" approach, which is more complex but more reliable than trying to patch pre-built wheels.

The Thinking Process

The assistant's thinking is visible in the structure and content of the message. The first sentence states the decision and the rationale in one breath: "update the entire stack to PyTorch nightly and rebuild sgl-kernel from source with SM120 support." This is followed by "The rationale is that the pre-built sgl-kernel 0.3.21+cu130 was built against older torch, and catid's gist (which works) uses nightly."

This reveals a two-step logical chain:

What Happened Next

The subsequent messages in this chunk (indices 5860-5993) show the assistant executing this plan. The nightly upgrade succeeded: PyTorch was updated to 2.12.0.dev20260307+cu130, flashinfer to 0.6.5, and sgl-kernel was built from source with SM120 FP4 support using catid's patches. The assistant then conducted extensive backend testing, finding that flashinfer_cutlass for MoE and flashinfer_cudnn for FP4 GEMM produced correct output, while other backends crashed or produced NaN. A critical accuracy issue — the checkpoint's default FP8 KV cache lacking proper scaling factors — was also identified and fixed by forcing BF16 KV cache dtype.

The final production configuration achieved ~172 tok/s at single-request concurrency and over 2100 tok/s at high concurrency (C=32). The nightly pivot, risky as it was, paid off.

Conclusion

Message 5859 is a textbook example of a well-reasoned technical pivot. The assistant correctly identifies that the constraint is not the torch version itself but the pre-built binary, recognizes that building from source eliminates the ABI concern, and commits to a plan that prioritizes capability over stability. The structured todowrite and clear rationale demonstrate a disciplined approach to what could have been a chaotic upgrade. This message is the hinge point on which the entire Qwen3.5 NVFP4 deployment turns — before it, the model produced garbage; after it, the model would go on to achieve production-grade throughput on 8× Blackwell GPUs.