The Metadata Mirage: When a CUDA Package Ships Nothing But a Name

In the high-stakes world of large language model training, where every kilotoken per second is fought for and GPU memory is measured in megabytes, the smallest environmental detail can derail an entire training run. Message 9758 of this opencode session captures one such moment — a seemingly routine package cleanup that unexpectedly broke the entire PyTorch installation, and the subsequent detective work that revealed a subtle truth about how NVIDIA distributes its CUDA libraries through Python packages.

Context: The Performance Regression Hunt

The story begins with a frustrating performance regression. The DFlash drafter training pipeline had been running at approximately 20 Ktok/s, but after a dataset expansion and a series of CUDA toolkit version swaps, throughput had dropped to 12.8 Ktok/s — a 36% degradation. The assistant had been methodically tracing the cause, examining everything from sequence length distributions to GPU memory utilization.

The breakthrough came when the assistant checked the installed pip packages and discovered contaminants: nvidia-cudnn-cu13, nvidia-cusparselt-cu13, nvidia-nccl-cu13, nvidia-nvshmem-cu13, and even sglang-kernel — all left over from a previous CUDA 13.0 upgrade attempt. The assistant reasoned that these cu13 libraries, still present in the virtual environment, could be intercepting CUDA calls and causing conflicts with the cu128 build of PyTorch that was supposed to be running. The fix seemed straightforward: remove the cu13 packages.

In message 9744, the assistant executed uv pip uninstall on all ten offending packages, including nvidia-cudnn-cu13. The uninstall completed successfully. The assistant then cleared the torch compile cache and verified the environment was clean. But when they tried to import torch in message 9747, they were greeted with a fatal error:

ImportError: libcudnn.so.9: cannot open shared object file: No such file or directory

Torch could not find its CUDA deep neural network library. The environment was broken.

The Subject Message: Discovery and Resolution

Message 9758 is the assistant's response to this crisis. It contains two parts: an extended reasoning block where the assistant diagnoses the problem, and a single bash command that implements the fix.

The reasoning begins with a stark realization:

The nvidia-cudnn-cu12 package 9.19.0.56 is a metadata-only package — no actual .so files. The actual libcudnn.so was provided by nvidia-cudnn-cu13 which I uninstalled.

This is the crux of the issue. The assistant had checked in message 9748 whether nvidia-cudnn-cu12 was installed — it was, showing version 9.19.0.56. The assumption was that this package provided the libcudnn.so.9 shared library that PyTorch needed. But it didn't. The package existed as a name in the pip registry, a version number, and a set of metadata files — but no actual compiled library binaries.

The assistant then enumerates the possible solutions:

  1. Install an older version of nvidia-cudnn-cu12 that ships the .so files
  2. Reinstall nvidia-cudnn-cu13
  3. Install cudnn from system packages
  4. Check what the system's ldconfig shows for cudnn Each option represents a different strategy. Option 1 would be the "correct" fix — matching the CUDA version to the PyTorch build. Option 2 is the pragmatic workaround — the cu13 cudnn library provides the same libcudnn.so.9 SONAME that cu128 torch expects, regardless of the CUDA version suffix in the package name. Option 3 would bypass pip entirely, relying on system-level package management. Option 4 is diagnostic — understanding the current state before acting. The assistant's reasoning reveals a key insight about NVIDIA's packaging strategy:
Nvidia has split the packages, with the .so files now in a separate package.

This explains the mystery. NVIDIA's Python packaging for CUDA libraries underwent a restructuring. In older versions, nvidia-cudnn-cu12 shipped the actual shared libraries. In version 9.19.x, the package became a metadata-only metapackage that declares dependencies on other packages (like nvidia-cublas-cu12) but expects the actual .so files to come from a different source — perhaps a companion package, a system installation, or a different CUDA version's package.

The assistant chooses option 2 — reinstalling nvidia-cudnn-cu13. The reasoning is pragmatic: "torch just needs libcudnn.so.9 regardless of CUDA minor version." This is technically correct — the SONAME (Shared Object Name) libcudnn.so.9 is versioned by the cuDNN library version (9.x), not by the CUDA toolkit version (12 vs 13). A cuDNN 9.19 binary compiled for CUDA 13 should work with a PyTorch built for CUDA 12.8, as long as the CUDA runtime compatibility is maintained.

The bash command executes the reinstall:

uv pip install nvidia-cudnn-cu13==9.19.0.56

The output confirms success: the package was resolved and installed in 5ms. The environment was restored.

The Deeper Lesson: NVIDIA's Packaging Ecosystem

This message illuminates a poorly documented aspect of the modern ML Python ecosystem. NVIDIA distributes its CUDA libraries through a complex system of pip packages with names like nvidia-cudnn-cu12, nvidia-cublas-cu12, nvidia-cuda-runtime-cu12, and so on. These packages follow a naming convention where the suffix (cu12, cu13) indicates the CUDA toolkit version they were compiled against.

However, the contents of these packages vary significantly between versions. Some ship the actual shared libraries (.so files) that get installed into the Python site-packages directory. Others are metadata-only packages that serve as dependency anchors — they exist to satisfy dependency requirements in pyproject.toml files but don't contain the actual binaries. The binaries might come from a different package, a system-wide CUDA installation, or be bundled directly into PyTorch itself.

This creates a fragile dependency chain. When a user installs torch (which depends on nvidia-cudnn-cu12), pip resolves and installs the metadata package. But if the actual .so files were previously provided by nvidia-cudnn-cu13 (installed as a dependency of some other package like SGLang), removing the cu13 package breaks torch even though the cu12 package appears to be present and correct.

Assumptions and Mistakes

The assistant made a reasonable but incorrect assumption: that the presence of nvidia-cudnn-cu12 in the pip list meant the cuDNN library was available. This is the kind of assumption that any engineer working with Python packages would make — the package name suggests it provides cuDNN for CUDA 12. But the reality was more nuanced.

The deeper mistake was not verifying the dependency chain before removing packages. When cleaning up the cu13 contaminants, the assistant could have checked whether any remaining packages depended on the cu13 cudnn library, or verified that the cu12 package actually contained the required binaries. In the heat of debugging a performance regression, with the pressure of a stalled training run, this verification step was skipped.

However, the assistant's response to the breakage was exemplary. Rather than panicking or trying random fixes, they:

  1. Identified the exact error (missing libcudnn.so.9)
  2. Checked what packages were installed (nvidia-cudnn-cu12 was present)
  3. Investigated whether the package contained the required files (it didn't)
  4. Reasoned through multiple fix options
  5. Chose the most pragmatic solution (reinstalling cu13)
  6. Verified the fix worked This systematic approach, even under pressure, is a model for debugging complex dependency issues.

Knowledge Created

This message creates several pieces of valuable knowledge:

  1. nvidia-cudnn-cu12 version 9.19.0.56 is a metadata-only package. It does not ship libcudnn.so files. This is a specific, actionable fact that future debugging can reference.
  2. libcudnn.so.9 is CUDA-version-agnostic at the SONAME level. A cuDNN library compiled for CUDA 13 can satisfy the runtime requirements of a PyTorch built for CUDA 12.8. This is not obvious — the CUDA version suffix in package names suggests tight coupling, but the actual shared object naming follows cuDNN's own versioning scheme.
  3. NVIDIA's Python packaging for CUDA libraries underwent a structural change. The split between metadata packages and binary packages is a recent development that can catch users off guard.
  4. A clean environment is not always a working environment. The assistant's goal was to remove CUDA version conflicts by eliminating cu13 packages. But "clean" (no cu13 packages) and "working" (torch can import) turned out to be contradictory goals in this case, because the cu12 package was incomplete without the cu13 binary.

Conclusion

Message 9758 is a microcosm of the challenges facing ML engineers today. It shows how the complex, layered dependency ecosystem of Python ML — with its CUDA toolkit versions, pip metapackages, shared library SONAMEs, and cross-version compatibility — can create failure modes that are invisible until the wrong package is removed. The assistant's methodical debugging, clear reasoning, and pragmatic resolution demonstrate the kind of systems thinking required to navigate this landscape.

The fix itself — reinstalling a CUDA 13 library to satisfy a CUDA 12.8 PyTorch — feels like a hack. But it works, and it reveals a deeper truth about the ecosystem: the version numbers in package names are often more about marketing and organization than about actual binary compatibility. The real constraint is the SONAME of the shared library, and libcudnn.so.9 doesn't care whether it was compiled for CUDA 12 or CUDA 13. In the end, the assistant's willingness to look past the labels and understand the actual runtime dependencies is what saved the training run.