The Hidden Dependency: Tracing a Missing libcudnn.so.9 After a CUDA Cleanup

Introduction

In the middle of a marathon debugging session spanning dozens of messages and multiple training runs, a single bash command stands out as a quiet diagnostic pivot point. Message 9750 in this opencode conversation is, on its surface, trivial: the assistant runs a find command on a remote machine to locate files matching libcudnn* or cudnn* inside a Python virtual environment. The output reveals a directory named cudnn and a handful of header and source files buried deep in third-party package directories. But this seemingly mundane search is the culmination of a chain of reasoning that reveals how fragile deep learning environments can be, how a single mistaken assumption about package dependencies can cascade into a broken runtime, and how the assistant's debugging methodology shifts from performance analysis to environmental forensics.

The Context: A Performance Mystery

To understand why message 9750 exists, we must trace back through the preceding conversation. The assistant had been training a DFlash drafter model on a multi-GPU setup (eight RTX PRO 6000 Blackwell GPUs, each with 96 GB of memory). The training throughput had plateaued at 12.8 Ktok/s, well below the expected ~20 Ktok/s observed in earlier runs. The assistant's reasoning (visible in [msg 9732]) was thorough: it examined queue utilization patterns (q_hs=[60] permanently full), analyzed per-drafter throughput (4.27 Ktok/s per drafter vs. 10.1 Ktok/s in a previous 2-drafter configuration), and considered whether the 20 Ktok/s baseline was even achieved with the same 5-target + 3-drafter configuration.

The assistant eventually identified a critical clue: the environment had been contaminated. During an earlier attempt to deploy SGLang, the user had installed CUDA 13.0 packages (nvidia-cudnn-cu13, nvidia-cusparselt-cu13, nvidia-nccl-cu13, etc.) alongside the original CUDA 12.8-based PyTorch installation. When the user reverted PyTorch back to the cu128 version, these cu13 packages remained. The assistant hypothesized that these leftover libraries were intercepting CUDA calls and causing performance degradation through version conflicts.

The Cleanup That Broke Everything

In [msg 9744], the assistant executed what seemed like a reasonable cleanup: it uninstalled all cu13 packages along with SGLang, flashinfer, and other training-unrelated packages. The command ran successfully, removing ten packages in 288 milliseconds. The assistant then cleared the torch compile cache (/tmp/torchinductor_root and /root/.cache/torch_extensions) to ensure no stale compiled kernels remained.

But the cleanup had an unintended consequence. When the assistant tested the environment in [msg 9747], PyTorch failed to import with a critical error:

ImportError: libcudnn.so.9: cannot open shared object file: No such file or directory

The nvidia-cudnn-cu13 package had been the sole provider of the cuDNN shared library (libcudnn.so.9) that PyTorch needed at runtime. The cu12 version of the cuDNN package (nvidia-cudnn-cu12) was still installed, as confirmed in [msg 9748], but PyTorch could not find the shared library it depended on. This reveals a subtle but important detail about how NVIDIA's Python packaging works: the nvidia-cudnn-cu12 package installs Python bindings and frontend code, but the actual shared library may be provided by a different package or installed at the system level. The assistant had assumed that the cu12 package would provide the same library path as the cu13 version, but this assumption proved incorrect.

Message 9750: The Diagnostic Search

Message 9750 is the assistant's response to this broken state. The command is:

ssh -o ConnectTimeout=10 root@10.1.2.6 'pct exec 200 -- bash -c "find /root/venv -name \"libcudnn*\" -o -name \"cudnn*\" 2>/dev/null | grep -v __pycache__ | head -20"'

This is a targeted search of the virtual environment for any file whose name contains libcudnn or cudnn. The grep -v __pycache__ filter excludes Python cache directories to reduce noise. The head -20 limits output to the first 20 results.

The output reveals:

/root/venv/lib/python3.12/site-packages/cudnn
/root/venv/lib/python3.12/site-packages/tilelang/3rdparty/cutlass/tools/profiler/include/cutlass/profiler/cudnn_helpers.h
/root/venv/lib/python3.12/site-packages/tilelang/3rdparty/cutlass/tools/profiler/src/cudnn_helpers.cpp
/root/venv/lib/python3.12/site-packages/tilelang/3rdparty/tvm/python/tvm/contrib/cudnn.py
/root/venv/lib/python3.12/site-packages/tilelang/3rdparty/tvm/python/tvm/relax/backend/cuda/cudnn.py

The first result is a directory named cudnn — this is the Python package directory for the cuDNN Python bindings. The remaining results are source and header files from third-party packages (tilelang, which bundles cutlass and tvm). Critically, no shared library file (libcudnn.so.9 or similar) is found anywhere in the virtual environment. This confirms that the cuDNN shared library was not installed inside the venv — it must have been provided by the nvidia-cudnn-cu13 package that was just removed, or it was expected to be found at the system level.

What This Message Reveals About the Assistant's Thinking

The assistant's reasoning process, visible in the preceding messages, shows a methodical progression from performance analysis to environmental debugging. The initial hypothesis was that leftover cu13 packages were causing a performance regression. When removing those packages broke the environment entirely, the assistant immediately pivoted to diagnosis rather than panicking or reverting.

The decision to search the virtual environment specifically (rather than the entire system) reveals an important assumption: the assistant expected the cuDNN library to be installed as a Python package within the venv, consistent with how modern NVIDIA Python packaging works. The nvidia-cudnn-cu12 package, which was confirmed present, should theoretically provide the library. But the search results show that only the Python bindings directory exists, not the actual shared object file.

This is a moment of insight for the assistant. The output of message 9750 tells a clear story: the cuDNN shared library was never inside the virtual environment at all. It must have been provided either by the now-removed nvidia-cudnn-cu13 package (which may have installed system-level libraries) or by a system-level CUDA installation. This means the assistant's original assumption — that removing cu13 packages would be safe because cu12 packages were present — was based on an incomplete understanding of how the cuDNN library was being resolved at runtime.

Input Knowledge Required

To fully understand this message, the reader needs knowledge of:

  1. NVIDIA's Python packaging scheme: The nvidia-cudnn-cu12 and nvidia-cudnn-cu13 packages are part of NVIDIA's effort to distribute CUDA libraries as Python wheels. However, the shared library files (libcudnn.so.*) may be installed to different locations depending on the package version and how it was built. The cu12 package might provide only Python bindings while expecting the shared library from a system installation or a different package.
  2. PyTorch's dynamic library loading: When PyTorch imports, it uses ctypes or CDLL to load libcudnn.so.9 from standard library paths (LD_LIBRARY_PATH, /usr/lib, system default paths). If the library isn't found, the import fails with the exact error seen in [msg 9747].
  3. The uv package manager: The environment uses uv for Python package management, which creates isolated virtual environments. The find command searches within /root/venv — the root of this virtual environment.
  4. LXC containerization: The commands are executed inside an LXC container (via pct exec 200), which adds another layer of indirection. System-level CUDA libraries might be installed on the host but not visible inside the container, or vice versa.
  5. The tilelang package: The output reveals that tilelang bundles third-party code including cutlass and tvm, which contain cudnn-related header and source files. These are irrelevant to runtime library resolution but appear in the search results as noise.

Output Knowledge Created

This message produces several pieces of actionable knowledge:

  1. Confirmation that no cuDNN shared library exists in the venv: The absence of any libcudnn.so file means the runtime dependency must be satisfied from outside the virtual environment.
  2. Evidence that nvidia-cudnn-cu12 installs only Python bindings: The directory /root/venv/lib/python3.12/site-packages/cudnn contains the Python interface but not the shared library. This is a critical distinction — the package provides the API wrapper, but the actual implementation is elsewhere.
  3. A map of what needs to be restored: The assistant now knows it must either reinstall the cu13 cudnn package (which would reintroduce the contamination it was trying to eliminate) or find an alternative way to provide libcudnn.so.9 to PyTorch.
  4. A lesson about dependency assumptions: The clean environment the assistant was trying to create is actually more fragile than the "contaminated" one. The cu13 packages, while potentially causing performance issues, at least provided a working runtime.

The Broader Significance

Message 9750 represents a turning point in the debugging session. Before this message, the assistant was operating under the assumption that the cu13 packages were purely harmful — performance-degrading contaminants that could be safely removed. After this message, the assistant must confront a more complex reality: the cu13 packages were providing essential runtime dependencies that the cu12 packages did not. The "clean" environment is broken, and the "contaminated" environment was at least functional.

This illustrates a fundamental challenge in machine learning infrastructure: CUDA dependency management is notoriously fragile. Different packages may provide overlapping subsets of the same libraries, and removing one package can silently break another that depends on the same shared object. The assistant's methodical approach — searching for the missing file, confirming its absence, and using that information to guide the next step — is exactly the right response to this kind of failure.

The message also demonstrates the value of explicit verification. Rather than assuming the cu12 package would work after removing the cu13 version, the assistant tested the environment immediately. This caught the breakage before any training time was wasted on a broken configuration. The subsequent diagnostic search in message 9750 then provided the data needed to understand why the breakage occurred, transforming a confusing error into a clear picture of missing dependencies.

Conclusion

Message 9750 is a small but revealing moment in a complex debugging session. A simple find command, executed across an SSH tunnel into an LXC container, reveals the hidden structure of a deep learning environment's dependencies. The absence of libcudnn.so.9 in the virtual environment tells a story about how NVIDIA packages distribute their libraries, about the assumptions the assistant made during cleanup, and about the fragile web of dependencies that modern ML frameworks depend on. This message is not just a diagnostic step — it is the moment when the assistant's mental model of the environment shifts from "clean vs. contaminated" to "functional vs. broken," setting the stage for the next phase of the recovery effort.