The Phantom Library: Debugging a Missing CUDA Dependency After an Aggressive Cleanup

Message Overview

In message <msg id=9757>, the assistant executes a single diagnostic command on a remote LXC container to locate the CUDA Deep Neural Network library (cuDNN) after a routine package cleanup unexpectedly broke the PyTorch runtime. The command searches three locations—the Python virtual environment's nvidia_cudnn_cu12 package directory, any libcudnn* shared object files anywhere in the venv, and the system's dynamic linker cache—to determine why torch can no longer find libcudnn.so.9. The results reveal a critical finding: the nvidia-cudnn-cu12 package exists only as metadata (a .dist-info directory with no accompanying shared libraries), and no libcudnn* files exist anywhere in the environment or the system's linker path.

The Full Message

The assistant ran the following command:

ssh -o ConnectTimeout=10 root@10.1.2.6 'pct exec 200 -- bash -c \
  "find /root/venv -path \"*cudnn_cu12*\" 2>/dev/null | head -5; \
   find /root/venv -name \"libcudnn*\" 2>/dev/null | head -5; \
   ldconfig -p 2>/dev/null | grep cudnn"'

And received this output:

/root/venv/lib/python3.12/site-packages/nvidia_cudnn_cu12-9.19.0.56.dist-info
/root/venv/lib/python3.12/site-packages/nvidia_cudnn_cu12-9.19.0.56.dist-info/REQUESTED
/root/venv/lib/python3.12/site-packages/nvidia_cudnn_cu12-9.19.0.56.dist-info/WHEEL
/root/venv/lib/python3.12/site-packages/nvidia_cudnn_cu12-9.19.0.56.dist-info/INSTALLER
/root/venv/lib/python3.12/site-packages/nvidia_cudnn_cu12-9.19.0.56.dist-info/METADATA

The second and third commands produced no output at all—no libcudnn* files in the venv, and no cuDNN entries in the system linker cache.

Why This Message Was Written: The Chain of Events

To understand why this single diagnostic command matters, one must trace the chain of events that led to it. The session had been struggling with a perplexing throughput degradation in a distributed training pipeline for a speculative decoding drafter model (DFlash). After expanding the training dataset from 500K to 1.1M prompts, throughput had dropped from approximately 20 Ktok/s to 12.8 Ktok/s—a 36% reduction that could not be explained by changes in sequence length or batch composition alone.

The assistant's investigation ([msg 9741]) uncovered a crucial clue: the Python virtual environment was contaminated with CUDA 13 libraries. During an earlier attempt to deploy the GLM-5-NVFP4 model using SGLang (a separate line of work in segment 0 of the conversation), the assistant had installed nvidia-cudnn-cu13, nvidia-cusparselt-cu13, nvidia-nccl-cu13, and other CUDA 13 packages. When the training work resumed, the torch version was reverted to a CUDA 12.8 build (torch 2.11.0+cu128), but the CUDA 13 packages remained in the environment. The assistant hypothesized that these leftover packages were intercepting CUDA library calls and causing silent performance degradation or runtime conflicts.

Acting on this hypothesis, the assistant executed an aggressive cleanup in <msg id=9744>, uninstalling ten packages including nvidia-cudnn-cu13, sglang-kernel, flashinfer-python, and others. The cleanup succeeded, and verification in <msg id=9745> confirmed no CUDA 13 packages remained. The torch compile cache was also cleared in <msg id=9746> to ensure no stale compiled kernels would interfere.

However, the cleanup had an unintended consequence. When the assistant tested the environment in <msg id=9747> by importing torch, it crashed immediately:

ImportError: libcudnn.so.9: cannot open shared object file: No such file or directory

The nvidia-cudnn-cu13 package had been the sole provider of the cuDNN shared library in the environment. Removing it left torch with no cuDNN runtime, and the supposedly equivalent nvidia-cudnn-cu12 package—which <msg id=9748> confirmed was installed—apparently did not fill the gap.

The Diagnostic Command: Structure and Intent

The command in <msg id=9757> is a carefully structured three-part probe designed to answer a single question: where is the cuDNN library?

Part 1: find /root/venv -path "*cudnn_cu12*" 2>/dev/null | head -5

This searches the entire virtual environment for any file or directory whose path contains the string cudnn_cu12. The -path flag matches against the full path, so this will find anything under the nvidia_cudnn_cu12 package namespace. The head -5 limits output to the first five matches, which is sufficient to determine whether the package contains actual library files or just metadata.

Part 2: find /root/venv -name "libcudnn*" 2>/dev/null | head -5

This searches for any shared object file matching the pattern libcudnn* anywhere in the venv. This is a broader search that would catch cuDNN libraries regardless of which package installed them. The absence of results here would be definitive evidence that no cuDNN shared library exists in the Python environment.

Part 3: ldconfig -p 2>/dev/null | grep cudnn

This queries the system's dynamic linker cache for any registered cuDNN libraries. The ldconfig -p command prints the contents of /etc/ld.so.cache, which lists all shared libraries known to the system's runtime linker. Filtering for cudnn checks whether the library might be installed at the system level rather than inside the Python venv.

The three probes are deliberately ordered from most specific to most general, and from most likely to least likely to succeed. Each probe eliminates a possible location for the missing library, progressively narrowing down where the problem might lie.

Assumptions and Their Validity

The assistant made several assumptions in crafting this diagnostic:

Assumption 1: The nvidia-cudnn-cu12 pip package should contain the cuDNN shared library. This assumption was reasonable—pip packages with names like nvidia-cudnn-cu12 typically ship compiled shared objects. However, the results showed that only a .dist-info directory existed, containing metadata files (WHEEL, METADATA, INSTALLER, REQUESTED) but no .so files. This revealed that nvidia-cudnn-cu12 is a metapackage or stub package that declares dependencies (it requires nvidia-cublas-cu12 per the pip show output in <msg id=9756>) but does not itself ship the cuDNN library.

Assumption 2: The cuDNN library might be installed system-wide. The ldconfig -p probe tested this assumption. The empty result confirmed that cuDNN was not registered in the system's dynamic linker cache, ruling out a system-level installation.

Assumption 3: The cuDNN library might be bundled inside the torch package itself. This was implicitly tested by the second probe (searching for libcudnn* anywhere in the venv). Earlier checks in <msg id=9753> and <msg id=9754> had already examined torch/lib/ and found no cuDNN files, so the assistant likely expected this probe to fail but included it for completeness.

Assumption 4: The previous nvidia-cudnn-cu13 package was the sole provider of cuDNN. This turned out to be correct—removing it left no alternative provider. The assumption that nvidia-cudnn-cu12 would serve as a drop-in replacement was incorrect, as the package proved to be a stub.

Input Knowledge Required

To fully understand this message, the reader needs:

  1. The CUDA software ecosystem: Knowledge that NVIDIA distributes CUDA libraries through multiple channels—system packages (via apt), Python wheels (via pip), and bundled within frameworks like PyTorch. The naming convention cu12 vs cu13 refers to the CUDA toolkit version the library is compiled against.
  2. The Python packaging system: Understanding that .dist-info directories contain package metadata (version, dependencies, installation records) but not runtime code. A package with only a .dist-info directory and no actual modules or shared libraries is effectively a stub.
  3. The Linux dynamic linker: Knowledge that ldconfig -p queries the runtime linker cache, and that shared libraries must be either in standard paths (/usr/lib, /usr/local/lib) or registered via ldconfig to be found by the dynamic linker at runtime.
  4. The PyTorch import chain: Understanding that torch loads CUDA libraries (including cuDNN) at import time via torch._C, and that the LD_LIBRARY_PATH environment variable or the torch/lib/ directory determines where these libraries are found.
  5. The conversation's history: The context that the environment had been through multiple CUDA toolkit version swaps (CUDA 12.8 → CUDA 13.0 → back to 12.8), that SGLang and flashinfer had been installed and then removed, and that the training throughput had mysteriously degraded after a dataset expansion.

Output Knowledge Created

This single command produced several pieces of critical knowledge:

Direct finding: The nvidia-cudnn-cu12 package at version 9.19.0.56 exists in the venv only as metadata. It provides no shared library files. This is the root cause of the ImportError: libcudnn.so.9 failure.

Negative finding: No libcudnn* files exist anywhere in the virtual environment. This rules out the possibility that cuDNN is provided by another package (such as nvidia-cublas-cu12 or a transitive dependency).

Negative finding: No cuDNN library is registered in the system's dynamic linker cache. This rules out a system-level cuDNN installation that might have been missed during the cleanup.

Implicit finding: The cuDNN library must have been provided by the now-removed nvidia-cudnn-cu13 package. Since torch was originally installed and working before the CUDA 13 contamination, the cuDNN library must have been bundled with the torch CUDA 12.8 build itself, or the nvidia-cudnn-cu13 package had been installed alongside torch at some point and was providing the library that torch expected.

Inferred finding: The torch CUDA 12.8 build (torch 2.11.0+cu128) apparently does not bundle cuDNN internally—or if it does, the bundled version is not being found by the dynamic linker. This suggests that torch was relying on the separately installed nvidia-cudnn-cu13 (or previously nvidia-cudnn-cu12 if it ever worked) to provide the runtime library.

The Thinking Process: What the Assistant Was Reasoning

The assistant's reasoning, visible in the preceding messages, shows a methodical debugging approach:

  1. Hypothesis formation ([msg 9741]): The assistant identified leftover CUDA 13 packages as a potential cause of throughput degradation, reasoning that "cu13 CUDA libraries still loaded. These likely intercept CUDA calls and cause conflicts with cu128 torch."
  2. Action and verification (<msg id=9744-9746>): The assistant removed the contaminating packages and cleared the compile cache, expecting a clean environment to restore performance.
  3. Unexpected failure ([msg 9747]): The cleanup broke torch entirely. The assistant's reaction in the reasoning block of &lt;msg id=9748&gt;—"Oops. The nvidia-cudnn-cu13 was the only cudnn installed—I removed it and now torch can't find libcudnn"—shows immediate recognition of the mistake.
  4. Root cause investigation (<msg id=9748-9756>): The assistant spent several messages trying to locate the cuDNN library. It checked whether nvidia-cudnn-cu12 was installed (yes), tried to find libcudnn* files (none found), examined torch/lib/ (empty of cuDNN), attempted to import nvidia.cudnn (module not found), and inspected the pip package metadata.
  5. The diagnostic pivot ([msg 9757]): The assistant designed the three-part command to systematically eliminate possible locations for the missing library, moving from the most specific (the cu12 package directory) to the most general (system linker cache).

Mistakes and Incorrect Assumptions

Several errors are visible in this sequence:

The primary mistake: Removing nvidia-cudnn-cu13 without first verifying that an alternative cuDNN provider existed. The assistant assumed that nvidia-cudnn-cu12 (which was installed) would serve the same role, but it did not verify this assumption before removing the working library.

The secondary mistake: Not checking what the nvidia-cudnn-cu12 package actually contained before relying on it. A quick find on the package directory before the cleanup would have revealed that it was a stub package with no shared libraries.

The implicit assumption about torch's CUDA packaging: The assistant assumed that a torch build labeled +cu128 would bundle its own CUDA libraries, including cuDNN. This is true for some torch distributions (especially the official CUDA 12.4 and 12.1 builds) but may not hold for the specific +cu128 nightly or custom build used here. The evidence suggests this torch build relied on a separately installed cuDNN package.

The assumption about package naming consistency: The assistant assumed that nvidia-cudnn-cu12 would be functionally equivalent to nvidia-cudnn-cu13 for providing the cuDNN runtime. While the library API should be the same, the packaging structure differs—one ships .so files, the other does not.

The Broader Context: A Cautionary Tale of Environment Management

This message, though small, captures a universal challenge in machine learning engineering: the fragility of GPU software environments. The CUDA ecosystem involves multiple overlapping package managers (pip, conda, apt, nvidia-container-toolkit), multiple CUDA toolkit versions that may or may not be compatible, and complex library loading chains where a single missing .so file can bring down an entire training pipeline.

The assistant's approach—systematic hypothesis testing, targeted diagnostics, and careful elimination of possibilities—is textbook debugging methodology. The mistake that necessitated this diagnostic (removing a library without verifying its replacement) is equally textbook: the assumption that a package with a similar name provides the same functionality.

The finding that nvidia-cudnn-cu12 is a stub package has immediate practical consequences: the assistant must now either reinstall nvidia-cudnn-cu13 (restoring the original contaminated state), find an alternative cuDNN source (such as a system-level installation via apt), or determine whether torch bundles cuDNN in a non-standard location that requires LD_LIBRARY_PATH configuration. Each option carries trade-offs between cleanliness, compatibility, and performance—the very trade-offs that led to the original throughput degradation investigation.

Conclusion

Message &lt;msg id=9757&gt; is a textbook example of a targeted diagnostic probe in a complex software environment. Its three-part structure systematically eliminates possible locations for a missing shared library, and its results provide definitive evidence that nvidia-cudnn-cu12 is a stub package that does not ship cuDNN runtime libraries. This finding resolves the immediate mystery of why torch fails to import after the cleanup, while also revealing a deeper truth about the environment's dependency structure: the training pipeline had been running on a cuDNN library provided by a CUDA 13 package, even though torch itself was compiled for CUDA 12.8. The performance degradation that started the investigation may well have been caused by this version mismatch, making the cleanup—despite its immediate failure—a step in the right direction.