The Moment the Toolkit Clicked: Installing CUDA 13.0 nvcc to Unlock Blackwell GPUs

Introduction

In the sprawling, multi-day effort to deploy Kimi K2.6 with DFlash speculative decoding on 8× RTX PRO 6000 Blackwell GPUs, there are dramatic moments—benchmark victories, throughput records, architectural breakthroughs. And then there are quiet, almost invisible moments where an infrastructure puzzle finally yields. Message [msg 11442] is one of those quiet pivots. It is a single bash command, executed over SSH, that installs the CUDA 13.0 nvcc compiler via apt on a remote machine. On its surface, it is mundane: apt-get install -y cuda-nvcc-13-0. But to understand why this command was written, and why it represents a genuine breakthrough after a cascade of failures, one must trace the reasoning chain that led to it—a chain that reveals how deeply the assistant understood the CUDA toolchain, how it debugged through layers of abstraction, and how it finally recognized that the simplest fix had been sitting in the package manager all along.

The Problem: FlashInfer Rejects Blackwell

The proximate cause of the crisis was FlashInfer, a JIT-compilation library for CUDA kernels that SGLang uses for sampling operations. The Blackwell RTX PRO 6000 GPUs have compute capability SM120 (major version 12, minor version 0). FlashInfer's compilation_context.py contains a function _normalize_cuda_arch that enforces a hard requirement: "SM 12.x requires CUDA >= 12.9." When FlashInfer detected the system's CUDA toolkit version, it found version 12.8—the version installed at /usr/local/cuda-12.8/bin/nvcc. Since 12.8 < 12.9, the function raised a RuntimeError, which was caught by an except Exception handler that logged a warning and left TARGET_CUDA_ARCHS as an empty set. Downstream, check_cuda_arch() found no eligible architectures and raised a fatal error, blocking the entire service from starting.

The cruel irony is that PyTorch itself was built against CUDA 13.0. The Python runtime reported torch.version.cuda == "13.0", and the CUDA runtime libraries from PyTorch's nvidia/cuda_runtime package were fully capable of targeting SM120. But FlashInfer's get_cuda_version() function (cached via @functools.cache) queries the system nvcc binary, not PyTorch's version string. The system had CUDA 12.8 installed as a full toolkit, with nvcc at /usr/local/cuda-12.8/bin/nvcc. The result was a version mismatch between the runtime (13.0) and the compiler (12.8), and FlashInfer's conservative check blocked progress.

The Cascade of Failed Workarounds

The assistant's first instinct was to work around the check rather than fix the underlying toolkit. In [msg 11436], it attempted to set CUDA_HOME=/dev/null in the systemd service file, hoping that FlashInfer's get_cuda_path() would fail to find nvcc and fall back to torch.version.cuda = 13.0. The service started, but after 420 seconds it failed. The journal logs revealed a JIT compilation error from SGLang's own jit_kernel/utils.py—the load_inline function also needed a working nvcc. The /dev/null/bin/nvcc path was, of course, nonexistent. The workaround was too clever by half: it broke the very toolchain it was trying to preserve.

In [msg 11438], the assistant's reasoning block shows it weighing four options: installing the full CUDA 13.0 toolkit, grabbing just the nvidia-cuda-nvcc package from PyPI, patching FlashInfer's source code to check torch.version.cuda, or trying environment variable tricks with a cached architecture list. It correctly rejected the last option: "option 4 won't work since CUDA 12.8's nvcc can't actually compile for SM120 anyway." This is a crucial insight—even if FlashInfer could be tricked into accepting SM120 as a target, the actual compilation would fail because CUDA 12.8's nvcc doesn't know about SM120. The compiler itself needed to be upgraded.

The assistant then tried the pip route. In [msg 11439], it attempted pip install nvidia-cuda-nvcc-cu13, which failed to build its wheel. In [msg 11440], it tried the alternative naming nvidia-cuda-nvcc-cu130, which didn't exist in any repository. The PyPI approach was a dead end—the nvcc compiler is a large native binary that can't easily be distributed as a pure Python wheel.

The Discovery That Changed Everything

At this point, the assistant had exhausted the clever workarounds. It was stuck. Then, in [msg 11441], it took a step back and ran a simple reconnaissance command: apt list --installed | grep cuda-toolkit. The output was revelatory:

cuda-toolkit-12-8-config-common/unknown,now 12.8.90-1 all [installed,automatic]
cuda-toolkit-12-config-common/unknown,now 12.9.79-1 all [installed,automatic]
cuda-toolkit-config-common/unknown,now 13.2.75-1 all [installed,automatic]

Three config-common packages were installed, for CUDA 12.8, 12.9, and 13.2. The presence of cuda-toolkit-config-common version 13.2.75-1 meant that the system's apt repository was configured to serve CUDA 13.x packages. The CUDA 13.0 toolkit was available—it just hadn't been installed. The assistant had been trying to install CUDA 13.0 nvcc via pip, but the real solution was sitting in the system's own package manager.

This is the moment captured in the subject message. The assistant runs:

ssh -o ConnectTimeout=5 root@10.1.2.200 "
# The config-common packages exist for 12.9 and 13.2, let's try installing CUDA 13 toolkit
apt-get update -qq 2>&1 | tail -3
apt-get install -y cuda-nvcc-13-0 2>&1 | tail -10
"

The output shows a cascade of package installations: libnvptxcompiler-13-0, cuda-driver-dev-13-0, libnvvm-13-0, cuda-cudart-13-0, cuda-cudart-dev-13-0, cuda-crt-13-0, and finally cuda-nvcc-13-0. Critically, the output also shows:

update-alternatives: using /usr/local/cuda-13.0 to provide /usr/local/cuda (cuda) in auto mode
update-alternatives: using /usr/local/cuda-13.0 to provide /usr/local/cuda-13 (cuda-13) in auto mode

The update-alternatives system automatically updated the /usr/local/cuda symlink to point to the newly installed CUDA 13.0 toolkit. This is significant because FlashInfer's get_cuda_path() function checks CUDA_HOME first, then falls back to which nvcc, and then checks common paths like /usr/local/cuda. With the symlink updated, which nvcc would now find /usr/local/cuda-13.0/bin/nvcc, which reports version 13.0—satisfying the >= 12.9 requirement for SM120 support.

Assumptions, Mistakes, and Lessons

The assistant made several assumptions during this debugging chain, some correct and some not. The correct assumption was that the core problem was a version mismatch between the CUDA compiler and the GPU architecture—FlashInfer's check was legitimate, not a bug. The assistant correctly recognized that patching the version check without upgrading nvcc would only defer the failure to actual kernel compilation.

The key mistake was assuming that installing CUDA 13.0 nvcc would be difficult. The assistant spent multiple rounds trying pip-based approaches (which failed) before checking the apt repository. The assumption that "if CUDA 12.8 is installed, CUDA 13.0 probably isn't available via apt" was implicit and wrong. In reality, the machine's NVIDIA apt repository was configured to serve multiple CUDA toolkit versions, and the config-common packages for 12.9 and 13.2 were already installed as part of the driver setup. The solution was a single apt-get install command away.

Another implicit assumption was that the CUDA_HOME=/dev/null trick would only affect FlashInfer's version check, not SGLang's JIT compilation. This was a failure to fully trace the dependency graph—both systems use nvcc for compilation, and breaking the nvcc path broke everything.

Input and Output Knowledge

Input knowledge required to understand this message includes: familiarity with the CUDA toolkit versioning scheme and the SM architecture numbering (SM120 = Blackwell); understanding of FlashInfer's JIT compilation pipeline and its architecture detection logic; knowledge of the update-alternatives mechanism for managing multiple CUDA installations; and awareness that NVIDIA's apt repository can serve multiple toolkit versions simultaneously.

Output knowledge created by this message is significant. The installation of cuda-nvcc-13-0 unblocks the entire downstream pipeline. FlashInfer can now detect CUDA >= 12.9, accept SM120 as a valid target architecture, compile sampling kernels for Blackwell, and allow SGLang to serve the K2.6 model. The message also implicitly documents a reliable procedure for adding CUDA 13.0 nvcc to a system that already has CUDA 12.8 installed—simply install the cuda-nvcc-13-0 package from the existing apt repository. This is a repeatable, auditable fix rather than a fragile workaround.

The Thinking Process

The assistant's reasoning across messages [msg 11436] through [msg 11442] follows a clear arc: from clever workaround, to failed workaround, to diagnosis of why the workaround failed, to reconnaissance, to the correct fix. The reasoning in [msg 11438] is particularly instructive. The assistant explicitly enumerates four options and evaluates each one. It correctly identifies that "CUDA 12.8's nvcc can't actually compile for SM120 anyway," which means any solution that doesn't upgrade the compiler is doomed. It then tries pip (fails), tries an alternative pip package name (fails), and finally checks apt (succeeds). The progression is methodical and shows a deep understanding of the CUDA toolchain's layered architecture: runtime libraries, compiler, and JIT infrastructure.

Broader Significance

This message is a microcosm of the entire infrastructure-debugging genre in machine learning engineering. The problem was not a bug in FlashInfer, SGLang, or PyTorch—it was a configuration gap. The system had a CUDA 12.8 toolkit installed (probably from an earlier setup phase) and a PyTorch built against CUDA 13.0. The two were inconsistent, and the inconsistency only manifested when Blackwell GPUs (requiring CUDA >= 12.9) were introduced. The fix was not to patch code but to align the toolchain. This is a lesson that applies broadly: when JIT compilation fails on new hardware, suspect the compiler version before the library code.

The message also demonstrates the value of system-level reconnaissance. The assistant could have spent hours trying to patch FlashInfer's source code, override version checks, or build nvcc from source. Instead, a single apt list --installed | grep cuda-toolkit revealed that the solution was already in the repository. The simplest fix was the right one all along.