The Metadata-Only Package: Tracing a CUDA Dependency Chain Through a Single pip show Command

Introduction

In the middle of a sprawling debugging session spanning dozens of messages, one seemingly innocuous command stands as a quiet pivot point. Message [msg 9756] is a single uv pip show nvidia-cudnn-cu12 command executed over SSH on a remote LXC container. Its output is terse: a package name, version number, location, one dependency, and one reverse dependency. On its surface, it is barely worth a second glance. But within the context of the session — a multi-hour struggle to restore DFlash training throughput on an 8-GPU machine — this message crystallizes a moment of diagnostic insight. It reveals a fundamental misunderstanding about how NVIDIA distributes its CUDA runtime libraries through pip, and it forces the assistant to confront an uncomfortable truth: the environment it thought it had cleaned was actually broken in a subtle and invisible way.

The Broader Context: A Performance Regression and a Contamination Theory

The story begins with a performance regression. The DFlash training pipeline, which processes speculative decoding drafters across multiple GPUs, had been running at approximately 20 Ktok/s. After a dataset expansion and a series of CUDA toolkit version swaps (from cu128 to cu130 and back), throughput collapsed to 12.8 Ktok/s — a 36% drop. The assistant spent several messages analyzing the math: batch sizes, anchor counts, KV lengths, and per-step latencies. The code had not changed. The GPUs were the same. Something in the software environment had shifted.

The breakthrough came in [msg 9741], when the assistant realized that leftover CUDA 13 pip packages — nvidia-cudnn-cu13, nvidia-cusparselt-cu13, nvidia-nccl-cu13, nvidia-nvshmem-cu13, and sglang-kernel — were still present in the virtual environment even after torch had been reverted to the cu128 build. The hypothesis was that these packages were intercepting CUDA library calls and causing conflicts, degrading performance through some form of runtime interference. The assistant's reasoning was sound: if torch dynamically loads CUDA libraries from the environment's site-packages, and if both cu12 and cu13 variants are present, the linker could resolve symbols from the wrong version, producing silently incorrect behavior or slower fallback paths.

Acting on this theory, the assistant removed all cu13 packages in [msg 9744]. Ten packages were uninstalled in 288 milliseconds. The environment appeared clean. But when the assistant tried to import torch in [msg 9747], it received an ImportError: libcudnn.so.9: cannot open shared object file: No such file or directory. The cu13 cudnn package had been the only provider of the actual cuDNN shared library. Its removal had broken torch entirely.

The Discovery: A Package Without Libraries

The assistant then checked whether nvidia-cudnn-cu12 was installed ([msg 9748]). It was — version 9.19.0.56. But when the assistant searched for the actual .so files ([msg 9749]–[msg 9754]), it found nothing. The nvidia/cudnn/ directory under site-packages was empty. The torch lib/ directory contained no cudnn libraries. The package appeared to be a ghost — registered in pip's metadata, listed as a dependency of torch, but carrying no compiled code.

This is where message [msg 9756] enters. The assistant runs uv pip show nvidia-cudnn-cu12 to inspect the package's metadata in detail. The output confirms the package exists at version 9.19.0.56, requires nvidia-cublas-cu12, and is required by torch. But critically, it reveals nothing about what the package actually contains. The pip show command, by design, reports metadata only — it cannot tell you whether a package ships shared libraries, Python modules, or empty wrappers. The assistant is using it to gather evidence for a hypothesis that is forming in real time: that the nvidia-cudnn-cu12 package, at this version, has been restructured by NVIDIA into a metadata-only metapackage that no longer bundles the cuDNN runtime.

The Assumption That Failed

The critical assumption — visible in the assistant's reasoning in [msg 9748] and [msg 9749] — was that nvidia-cudnn-cu12 would function identically to nvidia-cudnn-cu13. Both packages share the same version number (9.19.0.56). Both are published by NVIDIA on PyPI. Both declare themselves as dependencies of torch. The assistant naturally assumed that if the cu13 variant shipped libcudnn.so.9 in nvidia/cudnn/lib/, the cu12 variant would do the same.

This assumption was wrong. NVIDIA's packaging strategy for cuDNN had changed between the cu12 and cu13 package lines. The cu13 package was a "fat" package containing the actual shared libraries. The cu12 package, at version 9.19.0.56, had been split into a metadata-only metapackage that deferred to a system-level cuDNN installation or to a separate, lower-level package. The pip show output in [msg 9756] provides a clue: the package requires nvidia-cublas-cu12 but nothing else. It does not require nvidia-cublas-cu12 to provide cudnn — cuBLAS is a different library entirely. The absence of any cudnn-related dependency suggests the package expects the library to be provided externally.

Input Knowledge and Output Knowledge

To understand message [msg 9756], the reader needs several pieces of input knowledge:

1. The structure of NVIDIA's pip packages: NVIDIA distributes CUDA runtime libraries as separate pip packages (e.g., nvidia-cudnn-cu12, nvidia-cublas-cu12, nvidia-cusparselt-cu12), each targeting a specific CUDA minor version. These packages can either bundle shared objects (.so files) or act as metapackages that depend on other providers.
2. How torch discovers CUDA libraries: PyTorch's CUDA builds (like the cu128 variant) link against CUDA runtime libraries at build time but resolve them at runtime through the dynamic linker. The libraries can come from the system's ldconfig paths, from LD_LIBRARY_PATH, or from the torch installation's own lib/ directory. In a virtual environment, pip-installed NVIDIA packages place their .so files in site-packages/nvidia/<package>/lib/, and torch's __init__.py may add these paths to the dynamic linker search path.
3. The concept of a metapackage: A pip package can exist as pure metadata — a dist-info directory with no accompanying code. This is common for namespace packages or packages that have been split into smaller sub-packages for dependency management. The message creates new knowledge:
4. Confirmation that nvidia-cudnn-cu12 is installed and recognized by pip as a dependency of torch. This rules out the possibility that the package was accidentally removed or never installed.
5. Evidence of the package's dependency structure: It requires nvidia-cublas-cu12 but nothing else. This is unusual for a cuDNN package, which should depend on the core CUDA runtime. The absence of a dependency on nvidia-cuda-runtime-cu12 or similar is a red flag that the package may not be a complete installation.
6. A narrowing of the search space: The assistant now knows the problem is not that the package is missing, but that it is incomplete — it exists in pip's registry but does not provide the library torch needs.

The Thinking Process: A Diagnostic Chain

The reasoning visible across messages [msg 9747]–[msg 9756] reveals a methodical diagnostic process. When torch fails to import after the cu13 package removal, the assistant does not immediately reinstall the cu13 package. Instead, it checks whether the cu12 variant is present ([msg 9748]). When it finds it is, it searches for the actual library files ([msg 9749]–[msg 9754]). Each search returns empty results for .so files, but the assistant does not jump to conclusions — it systematically checks multiple locations: the nvidia/cudnn/ directory, the torch lib/ directory, and the entire venv path. Only after all searches fail does the assistant run pip show to inspect the package metadata directly.

This is a textbook debugging pattern: when a dependency appears to be installed but does not function, inspect the installation itself rather than the dependent code. The pip show command in [msg 9756] is the final piece of evidence before the assistant pivots to a new strategy — reinstalling the cu13 package to restore the missing libraries.

The Broader Lesson: Environment Hygiene in ML Workflows

The saga that unfolds around this message — from the performance regression through the CUDA package contamination to the discovery of the metadata-only package — illustrates a fundamental challenge in modern ML engineering. The Python packaging ecosystem for CUDA libraries is complex and poorly documented. NVIDIA publishes multiple variants of the same library targeting different CUDA versions, and the version numbering does not always indicate which packages contain actual binaries versus metadata wrappers. A pip list showing a package is installed tells you nothing about whether it is functional. A pip show revealing dependencies tells you nothing about whether those dependencies are sufficient.

The assistant's journey through this debugging session is a reminder that in the world of GPU-accelerated machine learning, the environment is not a passive container — it is an active participant in every computation. A single mismatched library version, a leftover package from a previous CUDA toolkit, or a metadata-only metapackage that fails to deliver its promised runtime can silently degrade performance by 36% or crash the entire training pipeline. The tools we use to inspect environments — pip list, pip show, find, ldconfig — are powerful, but they only reveal what we think to ask. The assistant had to ask the right questions in the right order to uncover the truth.

Conclusion

Message [msg 9756] is a single pip show command in a debugging session spanning hundreds of messages. It is not dramatic. It does not fix the problem. But it represents a critical diagnostic step: the moment when the assistant stops assuming that a package is what it appears to be and starts investigating its actual contents. The output — a few lines of metadata — is enough to confirm that the nvidia-cudnn-cu12 package is a hollow shell, a metadata-only metapackage that depends on external library provisioning. This discovery forces the assistant to abandon the clean-environment strategy and reinstall the cu13 package, which in turn reveals the next missing link in the dependency chain (libcusparseLt.so.0 from nvidia-cusparselt-cu13). The debugging continues, but the trajectory has shifted. The assistant is no longer chasing a performance regression; it is reconstructing a broken dependency graph, one package at a time.