The Critical Link: Installing nvcc to Unlock SGLang's CUDA Graph Capture on Blackwell SM120

In the sprawling effort to deploy a Qwen3.6-27B model on an 8× RTX PRO 6000 Blackwell cluster for large-scale batch inference, one message stands out as the quiet resolution of a multi-layered debugging saga. Message [msg 9496] is deceptively simple: a single bash command that installs the nvidia-cuda-nvcc package via uv pip install inside a Python virtual environment on a Proxmox LXC container. Yet this command represents the culmination of a chain of reasoning spanning half a dozen prior messages, each peeling back a different layer of incompatibility between SGLang, its kernel library, and the novel SM120 GPU architecture.

The Message

The command itself is straightforward:

ssh -o ConnectTimeout=10 root@10.1.2.6 "pct exec 200 -- bash -c 'export PATH=/root/.local/bin:\$PATH && uv pip install --python /root/venv/bin/python3 nvidia-cuda-nvcc 2>&1 | tail -5'" 2>&1

And the result:

Prepared 3 packages in 1.04s
Installed 3 packages in 6ms
 + nvidia-cuda-crt==13.2.78
 + nvidia-cuda-nvcc==13.2.78
 + nvidia-nvvm==13.2.78

Three packages installed in under seven seconds: nvidia-cuda-crt, nvidia-cuda-nvcc, and nvidia-nvvm, all version 13.2.78. The installation is clean, fast, and uneventful — the hallmark of a fix that required extensive detective work to reach.

The Reasoning Chain: Why This Command Was Necessary

To understand why this message exists, one must trace the debugging trail that preceded it. The agent had been attempting to launch an SGLang inference server on a machine equipped with NVIDIA RTX PRO 6000 Blackwell GPUs, which use the SM120 compute architecture. This architecture is the desktop/workstation variant of Blackwell, distinct from the SM100 (datacenter Blackwell/B200) architecture that SGLang's pre-built kernel wheels targeted.

The first barrier emerged in [msg 9482]: SGLang crashed on startup because sgl_kernel could not load any architecture-specific operations. Inspection revealed that the installed sglang-kernel==0.4.2.post2 wheel only contained kernels for SM90 (Hopper) and SM100 (datacenter Blackwell) — not SM120. The agent attempted various workarounds: switching to the cu130 variant of the wheel, checking for SM120 support, and even considering a full source build of sgl-kernel (which the wiki indicated would take ~2.5 hours).

The breakthrough came when the agent realized that SM100 kernels might actually run on SM120 hardware, since both are Blackwell-family architectures. The real problem was an ABI mismatch: the pre-compiled .so files had been built against PyTorch 2.11, but the environment had PyTorch 2.12. Downgrading PyTorch to 2.11 in [msg 9488] fixed the ABI issue, only to reveal a missing libnvrtc.so.13 library. Setting LD_LIBRARY_PATH to include the pip-installed CUDA libraries in [msg 9491] finally allowed sgl_kernel to import successfully.

But the victory was short-lived. When the agent launched SGLang in [msg 9492], it failed again — this time because CUDA graph capture requires the CUDA compiler (nvcc) for JIT compilation. The error message was clear: "Could not find nvcc and default cuda_home='/usr/local/cuda' doesn't exist."

The Decision: Install nvcc via Pip vs. Disable CUDA Graphs

At this juncture, the agent faced a genuine architectural decision, visible in the reasoning of [msg 9494]. Two paths lay open:

Path A: Disable CUDA graphs via --disable-cuda-graph. This would bypass the need for nvcc entirely, but at a severe performance cost. The agent correctly reasoned that CUDA graphs are critical for decode throughput — each decode step incurs kernel launch overhead, and CUDA graphs amortize this by capturing a sequence of kernel launches into a single graph that can be replayed. For a batch inference job generating 598K completions, disabling CUDA graphs could mean a 30–50% throughput reduction, extending the job from hours to days.

Path B: Install nvcc. This would preserve full performance but required finding a working nvcc package compatible with the existing CUDA runtime (installed via pip as part of the PyTorch ecosystem).

The agent chose Path B, but the first attempt stumbled. Installing nvidia-cuda-nvcc-cu13 — the versioned package name — failed because that package is deprecated. The error message helpfully pointed to the correct package: nvidia-cuda-nvcc. This is where message [msg 9496] enters.

Assumptions and Their Validity

The command in [msg 9496] rests on several assumptions, most of which proved correct:

  1. That nvidia-cuda-nvcc would install cleanly via pip/uv. This was not guaranteed — CUDA toolchain packages distributed via PyPI have historically been unreliable, with version mismatches and missing dependencies. The assumption held: the package resolved and installed in under two seconds.
  2. That the installed nvcc (version 13.2.78) would be compatible with the existing CUDA runtime (version 13.x). The agent did not verify version alignment explicitly, but the pip resolver handled dependency constraints. The co-installation of nvidia-cuda-crt==13.2.78 (the CUDA runtime library) alongside nvcc suggests the resolver ensured consistency.
  3. That SGLang would find nvcc without setting CUDA_HOME or adding to PATH. This assumption is implicit — the command does not set CUDA_HOME or modify PATH to include the nvcc binary. The agent likely assumed that pip installs nvcc into the venv's bin/ directory, which is already on PATH when using the venv's Python interpreter. This is a reasonable assumption given that the SGLang launch command uses /root/venv/bin/python3.
  4. That CUDA graph capture would work once nvcc was available. The agent assumed the remaining error was purely about missing nvcc, not about deeper incompatibilities between the SM100-compiled kernels and SM120 hardware during JIT compilation. This assumption would be tested in the subsequent SGLang launch attempt.

Input Knowledge Required

To understand this message fully, one needs knowledge spanning several domains:

Output Knowledge Created

This message produces several concrete pieces of knowledge:

  1. That nvidia-cuda-nvcc version 13.2.78 installs successfully via uv/pip on Python 3.12 in an Ubuntu 24.04 LXC container, pulling in nvidia-cuda-crt and nvidia-nvvm as dependencies.
  2. That the installation is fast — under 7 seconds for three packages totaling unknown size — making it a practical solution even in time-sensitive workflows.
  3. That the deprecated nvidia-cuda-nvcc-cu13 package is indeed broken (it fails to build from source), confirming the deprecation notice and validating the switch to the unversioned package.
  4. That pip-distributed CUDA toolchain packages can coexist with a CUDA runtime installed via PyTorch's dependencies, without manual path configuration — at least for this specific version combination.

The Broader Significance

This message is a textbook example of a class of problems that increasingly plague ML engineering: the gap between GPU hardware availability and software support. The RTX PRO 6000 Blackwell (SM120) was a brand-new architecture at the time, and the pre-built wheels for SGLang's kernel library had not yet been compiled for it. The agent's journey — from trying pre-built wheels, to downgrading PyTorch for ABI compatibility, to fixing library paths, to installing nvcc — illustrates the layered nature of deployment debugging in the fast-moving ML ecosystem.

The installation of nvcc was the final piece of the puzzle, but it was only meaningful because of the correct diagnosis that preceded it. Each prior fix (PyTorch downgrade, LD_LIBRARY_PATH, kernel fallback) had to be in place for nvcc to be the last missing link. Had the agent chosen to disable CUDA graphs instead, the inference job would have run significantly slower — a hidden tax that would have compounded across 598K completions. The decision to invest in finding the right nvcc package, rather than accepting the performance penalty, reflects a sophisticated understanding of where optimization effort is best spent.

In the end, message [msg 9496] is a reminder that the most impactful commands are often the simplest — once the hard work of diagnosis is done. The seven-second install was the payoff for a chain of reasoning that stretched across multiple rounds of tool calls, each eliminating one more obstacle between the software and the silicon.