The Deprecation Trap: When pip Package Names Lie

In the high-stakes world of deploying large language models on cutting-edge hardware, the smallest dependency management error can cascade into hours of lost productivity. Message [msg 9495] captures one such moment — a seemingly trivial pip install command that fails because the package name carries a deprecation notice that the package manager cannot gracefully handle. This message, buried within a lengthy session of provisioning an 8× RTX PRO 6000 Blackwell GPU machine for batch inference, is a vivid case study in the fragility of Python's package ecosystem when operating at the frontier of hardware support.

The Road to nvcc

To understand why this message exists, we must trace the chain of reasoning that led to it. The assistant had been wrestling with SGLang, a high-throughput inference engine, on an SM120 GPU — the compute architecture of the desktop Blackwell RTX PRO 6000. The journey had already been arduous. First, the pre-built sglang-kernel wheel shipped with kernels for SM90 (Hopper) and SM100 (datacenter Blackwell/B200), but not SM120 (desktop Blackwell). The assistant discovered that SM100 kernels could run on SM120 hardware through forward compatibility, but only after resolving an ABI mismatch by downgrading PyTorch from 2.12 to 2.11 ([msg 9488]), and then fixing a missing libnvrtc.so.13 library path issue ([msg 9490], [msg 9491]).

With sgl_kernel finally importing successfully, the assistant launched the SGLang server ([msg 9492]). But the server crashed with a new error: "Could not find nvcc and default cuda_home='/usr/local/cuda' doesn't exist" ([msg 9493]). The CUDA graph capture feature — a critical optimization for decode throughput — required the NVIDIA CUDA compiler (nvcc) for JIT compilation. Without it, SGLang could not initialize its model worker.

The assistant's reasoning in [msg 9494] reveals a careful trade-off analysis. Two options existed: disable CUDA graphs entirely via --disable-cuda-graph, or install nvcc. The assistant initially leaned toward the simpler flag-based approach, noting that "for batch inference with many concurrent requests the overhead gets distributed, so it might be acceptable." But then a deeper consideration shifted the decision: "CUDA graphs are critical for decode throughput since each decode step incurs kernel launch overhead. For generating 598K completions, disabling them could mean a 30-50% performance hit, making the job take significantly longer." This quantitative reasoning — weighing a 30-50% throughput penalty against the effort of installing a compiler — led the assistant to choose the installation path.

The Subject Message: A Failed Installation

The subject message executes exactly this decision:

[assistant] [bash] ssh -o ConnectTimeout=10 root@10.1.2.6 "pct exec 200 -- bash -c 'export PATH=/root/.local/bin:\$PATH && uv pip install --python /root/venv/bin/python3 nvidia-cuda-nvcc-cu13 2>&1'" 2>&1
Using Python 3.12.3 environment at: venv
Resolved 1 package in 1ms
   Building nvidia-cuda-nvcc-cu13==0.0.1
  × Failed to build `nvidia-cuda-nvcc-cu13==0.0.1`
  ├─▶ The build backend returned an error
  ╰─▶ Call to `setuptools.build_meta.build_wheel` failed (exit status: 1)

      [stderr]
      ⚠️ THIS PROJECT 'nvidia-cuda-nvcc-cu13' IS DEPRECATED.
      Please use 'nvidia-cuda-nvcc' instead.

      To install the correct package, use:

          pip install nvidia-cuda-nvcc

The command is straightforward: SSH into the Proxmox container (ID 200) on the remote host, set the PATH to include uv's binary directory, and use uv pip install to install nvidia-cuda-nvcc-cu13 into the Python 3.12 virtual environment. The package name follows a pattern that the assistant had seen work for other CUDA libraries — nvidia-cublas-cu13, nvidia-cudnn-cu13, nvidia-cuda-runtime-cu13 — all of which use the -cu13 suffix to indicate the CUDA 13 toolkit version they target. It was a reasonable inference: if the runtime libraries use this naming convention, the compiler package should too.

But the package nvidia-cuda-nvcc-cu13 is a trap. It exists as a version 0.0.1 stub that does nothing except print a deprecation warning and tell the user to install nvidia-cuda-nvcc instead. The build fails because the package's setup.py or pyproject.toml is intentionally broken — it is a redirect package, not a functional one. The error message is clear: "THIS PROJECT 'nvidia-cuda-nvcc-cu13' IS DEPRECATED. Please use 'nvidia-cuda-nvcc' instead."

Why This Happened

The NVIDIA CUDA pip packages underwent a naming reorganization. Earlier CUDA 12.x versions used the -cuXX suffix pattern consistently across all packages. For CUDA 13, NVIDIA appears to have standardized on a simpler naming scheme for the compiler package — dropping the CUDA version suffix entirely. The nvidia-cuda-nvcc package (without suffix) is the canonical name, and it installs the correct version (13.2.78 in this case) regardless of the CUDA toolkit version.

The assistant's mistake was an understandable one: applying a pattern that worked for CUDA runtime libraries to a package that had been renamed. The -cu13 suffix was a red herring — a deprecated naming convention that the NVIDIA packaging team had abandoned for this particular package.

The Resolution

The very next message ([msg 9496]) shows the fix. The assistant, learning from the error message, simply drops the -cu13 suffix:

[assistant] [bash] ssh -o ConnectTimeout=10 root@10.1.2.6 "pct exec 200 -- bash -c 'export PATH=/root/.local/bin:\$PATH && uv pip install --python /root/venv/bin/python3 nvidia-cuda-nvcc 2>&1 | tail -5'" 2>&1
Prepared 3 packages in 1.04s
Installed 3 packages in 6ms
 + nvidia-cuda-crt==13.2.78
 + nvidia-cuda-nvcc==13.2.78
 + nvidia-nvvm==13.2.78

The installation succeeds instantly, pulling in three packages: nvidia-cuda-crt, nvidia-cuda-nvcc, and nvidia-nvvm. The entire detour — the failed build, the confusion, the time spent diagnosing — was caused by a single deprecated package name.

Broader Lessons

This message illuminates several important dynamics in modern ML infrastructure work. First, it demonstrates how operating at the frontier of hardware support means constantly encountering edge cases in package management. The RTX PRO 6000 Blackwell GPU (SM120 architecture) was so new that pre-built wheels didn't include its compute capability, and the CUDA pip packages were still undergoing naming standardization. Every component in the stack — PyTorch, SGLang, sglang-kernel, CUDA packages — was at a bleeding-edge version where conventions were fluid.

Second, the message reveals the hidden cost of deprecated packages in the Python ecosystem. A package that exists solely to say "I'm deprecated, use this other one" creates a failure mode that wastes developer time. The error message is helpful — it tells you exactly what to do — but the build failure itself is unnecessary friction. A well-designed deprecation path would either install the replacement automatically or provide a no-op package that installs successfully while printing a warning. Instead, the build backend raises an error, which uv pip install treats as a failure, which stops the entire pipeline.

Third, the assistant's reasoning in the preceding message ([msg 9494]) demonstrates a sophisticated cost-benefit analysis. The decision to install nvcc rather than disable CUDA graphs was grounded in a quantitative estimate of throughput impact (30-50%) against the scale of the generation task (598K completions). This kind of reasoning — weighing engineering effort against performance impact — is the essence of effective ML infrastructure work. The fact that the installation itself hit a trivial naming snag is almost anticlimactic, but it underscores how even well-reasoned decisions can be derailed by ecosystem quirks.

Conclusion

Message [msg 9495] is a small moment in a much larger story — a single failed pip install in a session spanning dozens of commands across multiple machines. But it captures something essential about the nature of frontier ML engineering: the constant negotiation between what should work (based on patterns and conventions) and what actually works (based on the messy reality of package ecosystems in transition). The assistant's response to the failure — reading the error message, understanding the deprecation notice, and immediately applying the correction — is a model of effective debugging. The message stands as a reminder that in infrastructure work, the smallest details matter, and that a deprecated package name can be just as impactful as a misconfigured kernel or an incompatible CUDA version.