The Nightly Pivot: Upgrading PyTorch to 2.12.0+cu130 for Blackwell FP4 Inference

Introduction

In the long arc of deploying large language models on cutting-edge hardware, there comes a moment when the stable, known-good stack must be abandoned in favor of the bleeding edge. Message [msg 5865] captures precisely such a moment. The assistant, responding to the user's directive to "update all to nightly" ([msg 5857]), executes the first step of a full-stack upgrade from PyTorch 2.9.1+cu130 to the nightly 2.12.0.dev20260307+cu130. This message is the fulcrum upon which the entire Qwen3.5-397B-A17B-NVFP4 deployment pivots from a blocked, garbage-output state to a working production service.

Context: Why Nightly Was Necessary

The story leading up to this message is one of persistent hardware-software friction. The machine in question is an 8× NVIDIA RTX PRO 6000 Blackwell Server Edition system — GPUs with compute capability 12.0 (SM120), the newest Blackwell architecture. The model being deployed is Qwen3.5-397B-A17B-NVFP4, a massive 397-billion-parameter mixture-of-experts model quantized to NVFP4, a 4-bit floating-point format that is only supported on Blackwell hardware.

The problem was stark: when launched with default settings, the model produced garbage output — repeated exclamation marks or NaN values. The root cause, documented in a critical gist by developer catid (<https://gist.github.com/catid/87cca824963f17fe7479a0ed26221397>), was that the default FP4 GEMM and MoE runner backends selected kernels that produced incorrect results on SM120. The fix required explicit backend flags (--moe-runner-backend flashinfer_cutlass and --fp4-gemm-runner-backend flashinfer_cudnn) and, crucially, building sgl-kernel from source with SM120 support enabled.

The existing stack used PyTorch 2.9.1+cu130 with a pre-built sgl-kernel 0.3.21+cu130 wheel. This wheel was compiled without SM120-specific FP4 kernels. Catid's gist, which documented a working configuration, used PyTorch nightly and provided patches for building sgl-kernel from source. The user's directive was therefore strategically sound: move to nightly builds where the patches are designed to apply, rather than trying to backport fixes to an older, incompatible stable release.

The Message: Execution Begins

The subject message opens with a declaration of intent:

Latest nightly is torch==2.12.0.dev20260307+cu130. Let me proceed with the full upgrade. I'll do this step by step:

This single sentence reveals the assistant's reasoning process. The "Latest nightly" conclusion was reached through a dry-run installation in the preceding message ([msg 5864]), where uv pip install --dry-run confirmed that torch==2.12.0.dev20260307+cu130 was available from the PyTorch nightly index. The assistant did not blindly assume a version — it verified availability first, a hallmark of methodical engineering.

The phrase "I'll do this step by step" signals a deliberate, sequenced approach. The assistant recognizes that this is a multi-component upgrade (PyTorch → flashinfer → sgl-kernel) where each step has dependencies on the previous one. Breaking it into steps minimizes the risk of an unrecoverable failure mid-upgrade and allows verification at each stage.

The Command and Its Output

The assistant then executes the actual installation via SSH:

ssh root@10.1.230.174 '~/.local/bin/uv pip install --python ~/ml-env/bin/python3 \
  --index-url https://download.pytorch.org/whl/nightly/cu130 \
  --upgrade --force-reinstall torch torchvision torchaudio 2>&1 | tail -20'

Several technical decisions are encoded in this command:

Using uv over pip: The environment was set up with uv, a fast Python package manager. The command ~/.local/bin/uv pip install invokes uv's pip-compatible interface, ensuring that the existing ml-env virtual environment is used rather than creating a new one. This preserves all existing package dependencies while upgrading the specified ones.

The --index-url flag: Pointing to https://download.pytorch.org/whl/nightly/cu130 is critical. This is the PyTorch nightly build index specifically compiled for CUDA 13.0 (the cu130 suffix). Using the wrong index (e.g., the default PyPI or a different CUDA version) would install incompatible binaries. The assistant's awareness of this distinction reflects deep knowledge of the PyTorch ecosystem's distribution channels.

--upgrade --force-reinstall: This combination ensures that even if torch is already installed, it will be completely replaced. The --force-reinstall flag is particularly important when upgrading across major version boundaries (2.9.1 → 2.12.0.dev) to prevent partial upgrades or version conflicts.

Piping to tail -20: The assistant only captures the last 20 lines of output, focusing on the summary of what changed rather than the full download log. This is a pragmatic choice for a remote SSH session where output volume matters.

The output confirms a substantial upgrade:

 - nvidia-nvjitlink==13.0.39
 + nvidia-nvjitlink==13.0.88
 - nvidia-nvshmem-cu13==3.3.24
 + nvidia-nvshmem-cu13==3.4.5
 - nvidia-nvtx==13.0.39
 + nvidia-nvtx==13.0.85
 - pillow==12.0.0
 + pillow==12.1.0
 - setuptools==70.2.0
 + setuptools==78.1.0
 ~ sympy==1.14.0
 - torch==2.9.1+cu130
 + torch==2.12.0.dev20260307+cu130
 - torchaudio==2.9.1+cu130
 + torchaudio==2.11.0.dev20260307+cu130
 - torchvision==0.24.1+cu130
 + torchvision==0.26.0.dev20260307+cu130
 - triton==3.5.1
 + triton==3.6.0+git9844d...

The ~ prefix on sympy==1.14.0 indicates it was reinstalled at the same version (no change). The + and - prefixes show what was added and removed. Notably, nvidia-nvjitlink jumped from 13.0.39 to 13.0.88 — this is the NVIDIA JIT linker library, essential for CUDA kernel compilation at runtime. The nvidia-nvshmem-cu13 upgrade from 3.3.24 to 3.4.5 brings improved shared memory support, which is relevant for the Blackwell architecture's larger shared memory capacities.

The triton upgrade from 3.5.1 to 3.6.0+git9844d is particularly significant. Triton is the compiler used by PyTorch's inductor and by flashinfer for custom kernel generation. The nightly triton includes Blackwell-specific optimizations that are not present in the stable release.

Assumptions and Risks

The assistant makes several assumptions in this message, some explicit and some implicit:

That nightly PyTorch is compatible with CUDA 13.0.1: The system has CUDA Toolkit 13.0.1 installed at /usr/local/cuda-13.0/. The nightly cu130 wheels are compiled against CUDA 13.0 headers, but there can be subtle ABI incompatibilities between different CUDA 13 subversions. The assistant implicitly trusts that the wheel's CUDA requirements are satisfied.

That the upgrade won't break the Python environment: Upgrading PyTorch from 2.9.1 to 2.12.0.dev is a major version jump. Other packages in the environment (flashinfer, sgl-kernel, vLLM, etc.) were installed against torch 2.9.1's ABI. The assistant knows this is a risk — indeed, in the previous segment ([msg 5856]), the documentation explicitly notes: "torch 2.10.0+cu130 and nightly 2.12.0+cu130 have ABI break with sgl-kernel 0.3.21." The plan is to rebuild sgl-kernel from source against the new torch, mitigating this risk, but other packages may still break.

That the nightly build is stable enough for production inference: Nightly builds are, by definition, not release-quality. They may contain regressions, API changes, or correctness bugs. The assistant is betting that catid's successful deployment on nightly validates this risk.

That the step-by-step approach will succeed: The assistant assumes that upgrading PyTorch first won't break the ability to install flashinfer or build sgl-kernel afterward. This is a reasonable assumption but not guaranteed — if the nightly torch changes a C API that flashinfer depends on, the subsequent flashinfer upgrade could fail.

Input Knowledge Required

To fully understand this message, one needs:

  1. The history of the deployment: The previous attempts to serve Qwen3.5-397B-A17B-NVFP4 produced NaN/garbage output due to incorrect FP4 kernel selection on SM120 Blackwell GPUs.
  2. Catid's gist: The critical reference documenting that sgl-kernel must be built from source with TORCH_CUDA_ARCH_LIST=12.0a and specific CMake flags, and that the MoE and FP4 GEMM backends must be explicitly pinned.
  3. The existing software stack: PyTorch 2.9.1+cu130, sgl-kernel 0.3.21+cu130 (pre-built wheel), flashinfer 0.6.4, CUDA 13.0.1, NVIDIA driver 590.48.01.
  4. The ABI incompatibility note: The documentation explicitly warns that torch 2.10.0+cu130 and nightly 2.12.0+cu130 have an ABI break with sgl-kernel 0.3.21, which is why the sgl-kernel rebuild from source is necessary.
  5. uv package management: The environment uses uv rather than pip, and the assistant must use the correct uv invocation syntax.
  6. SSH remote execution: All commands are executed over SSH to the LXC container at 10.1.230.174.

Output Knowledge Created

This message produces several concrete outputs:

  1. A verified nightly PyTorch installation: The command output confirms that torch==2.12.0.dev20260307+cu130 is now installed, along with updated torchvision, torchaudio, triton, and NVIDIA CUDA libraries.
  2. A confirmed upgrade path: The dry-run in [msg 5864] was speculative; this message confirms the actual installation succeeds without errors.
  3. A visible plan: The assistant's "Step 1" labeling establishes a multi-step plan that the reader (and the user) can track. Subsequent steps will upgrade flashinfer and build sgl-kernel from source.
  4. A baseline for debugging: If subsequent steps fail, the team knows that PyTorch was successfully upgraded and can focus on flashinfer or sgl-kernel issues.

The Thinking Process

The assistant's thinking is visible in the structure of the message. It does not simply execute a command — it first states the conclusion ("Latest nightly is torch==2.12.0.dev20260307+cu130"), then announces the plan ("Let me proceed with the full upgrade. I'll do this step by step"), then labels the first action ("Step 1: Install PyTorch nightly cu130"), and finally executes.

This structure reveals a methodical, plan-oriented mindset. The assistant is not guessing or experimenting — it has gathered information (the nightly version from the dry-run), formulated a strategy (step-by-step upgrade), and is now executing with clear labeling so that progress is trackable. The use of bold headings within the message ("Step 1") serves both as documentation for the current state and as a signal to the user about what to expect next.

The choice to show the full tail -20 output rather than a summary is also telling. The assistant wants to demonstrate transparency — the user can see exactly what packages changed, at what versions. In a debugging scenario where version mismatches are the primary source of errors, this level of detail is valuable.

Conclusion

Message [msg 5865] is a seemingly simple package upgrade command that carries enormous weight in the broader narrative. It represents the moment when the team commits to the nightly build track, abandoning the safety of stable releases for the promise of Blackwell-native FP4 support. The assistant's methodical approach — verify the version, announce the plan, execute step by step, show the output — exemplifies good engineering practice in high-stakes infrastructure work.

The upgrade succeeds cleanly, setting the stage for the subsequent steps: upgrading flashinfer (which will prove more challenging, as [msg 5868] and [msg 5869] reveal), building sgl-kernel from source with SM120 patches, and ultimately achieving a working Qwen3.5-397B-A17B-NVFP4 deployment that delivers ~172 tok/s single-request throughput and over 2100 tok/s at high concurrency. The nightly pivot was the right call, and this message captures the moment the decision was executed.