The Whack-a-Mole of CUDA Dependencies: A Case Study in Package Ecosystem Assumptions

Introduction

In the course of debugging a machine learning training pipeline, an AI assistant executed a single bash command that, on its surface, appears trivial: installing two Python packages (nvidia-nccl-cu13==2.28.9 and nvidia-nvshmem-cu13==3.4.5) into a virtual environment. Yet this moment captures a profound lesson in systems debugging — the moment when a hypothesis about environmental contamination collides with the reality of shared library dependencies, and the assistant is forced to systematically reinstall everything it just removed.

The Message

[assistant] [bash] ssh -o ConnectTimeout=10 root@10.1.2.6 'pct exec 200 -- bash -c "export PATH=/root/.local/bin:\$PATH && source /root/venv/bin/activate && uv pip install nvidia-nccl-cu13==2.28.9 nvidia-nvshmem-cu13==3.4.5 2>&1"' 2>&1
Using Python 3.12.3 environment at: venv
Resolved 2 packages in 68ms
Installed 2 packages in 7ms
 + nvidia-nccl-cu13==2.28.9
 + nvidia-nvshmem-cu13==3.4.5

The command is executed over SSH into a Proxmox host (10.1.2.6), targeting an LXC container with ID 200 via pct exec. It activates a Python virtual environment at /root/venv and uses uv pip install — the uv package manager — to install two packages pinned to exact versions. The operation succeeds in 7 milliseconds, resolving two packages in 68ms. It's fast, clean, and seemingly unremarkable.

But to understand why this message exists at all, we must trace the reasoning chain that led here — a chain that reveals how easily a well-intentioned cleanup can cascade into a dependency crisis.

The Context: A Training Pipeline in Crisis

The broader session involves training a DFlash drafter model — a speculative decoding architecture that uses multiple GPU workers to generate draft tokens in parallel. The system had been running at approximately 20 Ktok/s (thousand tokens per second) before a dataset expansion and environment change caused throughput to collapse to 12.8 Ktok/s. The assistant was tasked with diagnosing and fixing this regression.

The investigation had taken a specific turn: the assistant hypothesized that leftover CUDA 13 packages — installed during an earlier attempt to use SGLang with CUDA 13.0 — were interfering with the PyTorch CUDA 12.8 installation. The reasoning was plausible: if nvidia-cudnn-cu13, nvidia-nccl-cu13, and similar packages were present, they might intercept CUDA library calls and cause version conflicts, degrading performance.

The Cleanup That Broke Everything

Acting on this hypothesis, the assistant removed ten packages in a single command ([msg 9744]): nvidia-cudnn-cu13, nvidia-cusparselt-cu13, nvidia-nccl-cu13, nvidia-nvshmem-cu13, plus SGLang, flashinfer, and flash-attn packages. The compile cache was also cleared. The environment was now "clean" — or so it seemed.

The first sign of trouble came immediately. When the assistant tried to import PyTorch ([msg 9747]), it crashed with:

ImportError: libcudnn.so.9: cannot open shared object file: No such file or directory

This was the first domino. The assistant had assumed that nvidia-cudnn-cu12 (version 9.19.0.56, already installed) would provide the cuDNN shared library. But investigation revealed a critical fact: the nvidia-cudnn-cu12 package at version 9.19.0.56 is a metadata-only package — it contains no .so files. It exists solely to declare dependencies and metadata. The actual shared libraries were provided exclusively by the nvidia-cudnn-cu13 package that had just been deleted.

This is a subtle but crucial architectural detail in NVIDIA's Python packaging. Starting around the 9.x releases, NVIDIA split its CUDA Python packages into two categories: some ship actual shared objects (.so files), while others are lightweight metapackages that only declare dependencies. The cu13 variants happened to be the ones carrying the actual binaries, while the cu12 variants were hollow shells.

The Whack-a-Mole Sequence

What followed was a systematic, package-by-package restoration:

  1. Missing libcudnn.so.9 → Reinstall nvidia-cudnn-cu13==9.19.0.56 ([msg 9758])
  2. Missing libcusparseLt.so.0 → Reinstall nvidia-cusparselt-cu13==0.8.0 ([msg 9761])
  3. Missing libnccl.so.2 → Reinstall nvidia-nccl-cu13==2.28.9 (<msg id=9763, the subject message>)
  4. Missing libnvshmem → Also reinstalled in the same command as step 3 Each time the assistant fixed one missing library, torch's import would progress further through its initialization sequence before hitting the next missing dependency. The subject message represents the final step in this chain — installing the NCCL (NVIDIA Collective Communications Library) and NVSHMEM (NVIDIA Shared Memory) packages. After this command, the next message ([msg 9764]) shows torch importing successfully: 2.11.0+cu128 12.8 with 8 GPUs available. The environment was functional again.

Assumptions and Their Consequences

This episode reveals several critical assumptions — and their flaws:

Assumption 1: "cu13 packages are contaminants." The assistant assumed that packages tagged cu13 would cause version conflicts with a cu128 PyTorch build. In reality, these packages are version-agnostic shared library providers. The cu13 in the package name refers to the CUDA version the package was built against, not the CUDA version it forces torch to use. PyTorch 2.11.0+cu128 links against its own CUDA 12.8 runtime; the cu13 packages merely provide .so files that torch loads at initialization.

Assumption 2: "cu12 equivalent packages exist and work." The assistant expected that nvidia-cudnn-cu12 would provide the same functionality as nvidia-cudnn-cu13. But NVIDIA's packaging split meant the cu12 variant was a metapackage, not a binary package. This asymmetry is not obvious from package names alone.

Assumption 3: "Removing packages will fix performance." The deeper assumption was that environmental contamination caused the throughput regression. While plausible, this hypothesis was never validated before acting on it. The cleanup created a new crisis (a broken environment) without necessarily solving the original problem.

The Thinking Process Visible in Reasoning

The assistant's reasoning blocks throughout this sequence reveal a methodical debugging approach. Each error message is read carefully, the missing library name is extracted, and the corresponding package is identified and reinstalled. The assistant shows awareness of the irony — in [msg 9765] it notes:

"OK so torch works now with cu128 and the cu13 library packages reinstalled (they just provide .so files, the actual CUDA version used by torch is still 12.8)."

And then the crucial realization:

"Now the issue is: the cu13 libraries were present in the previous 12.8K tok/s run too. So reinstalling them doesn't fix the throughput problem."

This is the moment of insight: the entire cleanup was based on a faulty premise. The cu13 packages were present during both the fast (20 Ktok/s) and slow (12.8 Ktok/s) runs, so they cannot be the cause of the regression. The assistant has spent multiple messages undoing its own cleanup, only to arrive back at the starting point — but now with the knowledge that the real cause lies elsewhere.

Input Knowledge Required

To understand this message fully, one needs:

  1. CUDA Python packaging architecture: Knowledge that NVIDIA ships separate cu11, cu12, cu13 variants of libraries, and that some are binary packages while others are metapackages. The cu13 packages in this case were the binary providers.
  2. PyTorch dynamic linking: Understanding that PyTorch loads CUDA libraries (cudnn, nccl, cusparselt, nvshmem) at import time via ctypes or dlopen, and that missing .so files cause ImportError at the from torch._C import * line.
  3. LXC container management: The pct exec 200 command targets a Proxmox LXC container, and the double SSH hop (from the assistant's environment → Proxmox host → container) adds complexity to debugging.
  4. uv package manager: The use of uv pip install rather than plain pip install reflects the project's choice of the fast Rust-based package manager.
  5. Version pinning: The exact version numbers (2.28.9 for nccl, 3.4.5 for nvshmem) are not arbitrary — they match what was previously installed, suggesting the assistant is deliberately restoring the exact prior state.

Output Knowledge Created

This message produces several concrete outcomes:

  1. A working Python environment: Torch can now import successfully on all 8 GPUs.
  2. Confirmation of package necessity: The cu13 packages are not optional extras but required dependencies for this torch build.
  3. A refuted hypothesis: The cu13 contamination theory is disproven, forcing the assistant to seek the real cause of the throughput regression elsewhere.
  4. A documented dependency graph: The chain of missing libraries (cudnn → cusparselt → nccl → nvshmem) maps out torch's initialization order, which is valuable systems knowledge.

The Broader Lesson

This single message — a 68-millisecond package installation — sits at the inflection point of a debugging session. It represents the moment when a plausible but incorrect hypothesis is exhausted, and the search for the real cause must begin anew. The assistant's willingness to systematically undo its own changes, methodically following the chain of error messages, demonstrates a crucial debugging discipline: when your intervention breaks things, you must be willing to revert step by step until the system works again, even if that means reinstalling the very packages you just removed.

The irony is that the assistant's reasoning was sound in isolation — removing unnecessary packages is good hygiene. But the assumption about which packages were "unnecessary" was wrong, and the cost was several rounds of debugging time. In complex systems with deep dependency chains, the difference between a "contaminant" and a "necessary dependency" is not always visible from package names alone.