The Quiet Verification: How One find Command Resolved a CUDA Dependency Crisis

Introduction

In the sprawling narrative of an opencode coding session spanning dozens of segments and thousands of messages, some of the most consequential moments arrive not as grand architectural decisions or breakthrough performance gains, but as quiet, almost mundane verification steps. Message [msg 9759] is precisely such a moment: a single bash command that searches for cuDNN shared library files in a Python virtual environment. On its surface, it is trivial—a find piped through grep that returns five file paths. Yet this message represents the culmination of a deep debugging spiral, the resolution of a critical dependency conflict, and a turning point in a multi-day effort to stabilize a distributed training pipeline for the DFlash speculative decoding drafter.

To understand why this message was written, one must trace the chain of events that led to it—a chain that began with a mysterious throughput degradation and wound through CUDA version mismatches, metadata-only Python packages, and the fragile interdependencies of the PyTorch ecosystem. This article examines that chain, the reasoning behind each decision, the assumptions that were tested and broken, and the knowledge that this single verification step produced.

The Context: A Training Pipeline in Crisis

The session leading up to message [msg 9759] had been consumed by a frustrating performance regression. The DFlash training pipeline, which had previously achieved approximately 20 Ktok/s on a multi-GPU setup, had dropped to roughly 12.8 Ktok/s after a dataset expansion. The assistant had spent considerable effort analyzing potential causes: changes in sequence length distributions, differences in anchor token processing, and shifts in GPU utilization patterns.

The critical breakthrough came when the assistant checked the installed Python packages and discovered that nvidia-cudnn-cu13, nvidia-cusparselt-cu13, nvidia-nccl-cu13, and nvidia-nvshmem-cu13 were all still present in the virtual environment (see [msg 9741]). These packages had been installed during an earlier experiment with CUDA 13.0 and SGLang, and they had never been removed. Even though the assistant had reverted PyTorch to the cu128 (CUDA 12.8) version, these cu13 libraries remained, potentially intercepting CUDA calls and causing conflicts.

This was a classic environment pollution problem—one that is notoriously difficult to diagnose because the symptoms (slower performance, intermittent errors) are vague and the root cause (version-mismatched shared libraries silently overriding each other) is invisible to most monitoring tools.

The Decision to Clean and Its Unintended Consequence

Acting on this hypothesis, the assistant made a decisive move: remove all cu13 packages and all SGLang-related packages that were unnecessary for training ([msg 9744]). The command uninstalled ten packages in 288 milliseconds, including nvidia-cudnn-cu13, nvidia-cusparselt-cu13, nvidia-nccl-cu13, nvidia-nvshmem-cu13, sglang-kernel, sglang, flashinfer-python, flashinfer-cubin, flash-attn-4, and flash-linear-attention.

This decision was based on a reasonable assumption: that the cu12 equivalents of these packages were already installed and would provide the necessary libraries. The assistant had verified that nvidia-cudnn-cu12 was listed in the package inventory ([msg 9748]). However, this assumption proved incorrect in a subtle and instructive way.

When the assistant attempted to verify that PyTorch still functioned after the cleanup ([msg 9747]), it received a stark error:

ImportError: libcudnn.so.9: cannot open shared object file: No such file or directory

PyTorch could not find the cuDNN library it needed to initialize. The assistant had inadvertently broken the environment it was trying to fix.

The Discovery: Metadata-Only Packages

This failure triggered a deeper investigation into how the nvidia-cudnn-cu12 package actually works. The assistant ran a series of diagnostic commands ([msg 9749] through [msg 9757]) that revealed a surprising fact: the nvidia-cudnn-cu12 package at version 9.19.0.56 is a metadata-only package. It contains no shared library files (.so files) whatsoever. Its installation directory consists only of a dist-info folder with metadata files like METADATA, WHEEL, INSTALLER, and REQUESTED.

This is a significant architectural detail of the NVIDIA Python package ecosystem. In older versions, the nvidia-cudnn-cu12 package shipped the actual libcudnn.so files. At some point (around version 9.19.0), NVIDIA split the packaging: the cu12 variant became a lightweight metapackage that declares dependencies (on nvidia-cublas-cu12) but does not include the runtime libraries itself. The cu13 variant, meanwhile, continued to ship the actual .so files.

This created an asymmetric dependency situation: PyTorch cu128 (which uses CUDA 12.8) expects to find libcudnn.so.9 at runtime, and the only pip package in the environment that actually provides this file is nvidia-cudnn-cu13. The cu12 package declares compatibility but doesn't deliver the binary.

The Fix and the Verification

The assistant's response to this discovery was pragmatic rather than principled. Rather than investigating alternative sources for the cuDNN library (such as a system-level installation via apt, or an older version of the cu12 package that includes binaries), the assistant simply reinstalled nvidia-cudnn-cu13 ([msg 9758]):

uv pip install nvidia-cudnn-cu13==9.19.0.56

This decision reflects a practical trade-off: the goal was to get the training pipeline running again, not to achieve a perfectly clean CUDA version environment. Since libcudnn.so.9 is versioned by its soname (the .9 suffix) rather than by the CUDA toolkit minor version, a cu13-provided cuDNN should be binary-compatible with a cu12-based PyTorch. The risk is minimal—cuDNN's API is stable across CUDA versions—and the benefit is immediate: the library becomes available again.

Message [msg 9759] is the verification step that follows this fix. The assistant runs:

ssh -o ConnectTimeout=10 root@10.1.2.6 'pct exec 200 -- bash -c "find /root/venv -path \"*cudnn*\" -name \"*.so*\" 2>/dev/null | head -5"'

The output confirms that the cuDNN shared libraries are now present:

/root/venv/lib/python3.12/site-packages/cudnn/_compiled_module.cpython-312-x86_64-linux-gnu.so
/root/venv/lib/python3.12/site-packages/nvidia/cudnn/lib/libcudnn_adv.so.9
/root/venv/lib/python3.12/site-packages/nvidia/cudnn/lib/libcudnn.so.9
/root/venv/lib/python3.12/site-packages/nvidia/cudnn/lib/libcudnn_graph.so.9
/root/venv/lib/python3.12/site-packages/nvidia/cudnn/lib/libcudnn_engines_precompiled.so.9

The presence of libcudnn.so.9 in the nvidia/cudnn/lib/ directory confirms that the cu13 package has restored the missing runtime dependency. The _compiled_module.cpython-312-x86_64-linux-gnu.so file in the cudnn/ directory (the Python-level cuDNN bindings) was already present and unaffected by the uninstall.

Input Knowledge Required

To fully understand this message, one needs several layers of domain knowledge:

CUDA versioning conventions: NVIDIA's CUDA packages use suffixes like cu12 and cu13 to indicate the CUDA toolkit version they target. PyTorch builds are similarly tagged (e.g., cu128 for CUDA 12.8). Understanding that these suffixes denote compatibility rather than strict requirement is crucial.

Linux shared library resolution: The error libcudnn.so.9: cannot open shared object file indicates that the dynamic linker (ld.so) cannot find the library in its search path. Python packages that ship .so files typically place them in package-specific directories, and the Python interpreter adds these to the library search path at import time.

NVIDIA's packaging strategy: The distinction between metadata-only packages and binary-shipping packages is not obvious. The nvidia-cudnn-cu12 package at version 9.19.0.56 is a metapackage, while nvidia-cudnn-cu13 at the same version ships binaries. This asymmetry is a trap for anyone assuming symmetric behavior.

The soname mechanism: The .9 suffix in libcudnn.so.9 is a Linux soname (shared object name) version indicator. It means that any library built against cuDNN API version 9 can use this file, regardless of the CUDA toolkit minor version. This is why a cu13-provided cuDNN can serve a cu12-based PyTorch.

Output Knowledge Created

This message produces several concrete pieces of knowledge:

  1. The cuDNN library files are now present in the expected locations within the virtual environment. This is the primary verification.
  2. The cu13 package installs to nvidia/cudnn/lib/ within the site-packages directory, which is the standard layout for NVIDIA's pip-distributed CUDA libraries.
  3. The Python-level cuDNN bindings (in cudnn/_compiled_module.cpython-312-x86_64-linux-gnu.so) survived the uninstall/reinstall cycle, confirming they are part of a separate package (cudnn or nvidia-cudnn-frontend).
  4. Multiple cuDNN component libraries exist: libcudnn_adv.so.9 (advanced operations), libcudnn_graph.so.9 (graph API), libcudnn_engines_precompiled.so.9 (precompiled engine kernels), and libcudnn.so.9 (the main library). All are required for full cuDNN functionality.

Assumptions and Their Validation

The assistant operated under several assumptions during this debugging sequence:

Assumption 1: That removing cu13 packages would leave cu12 equivalents functional. This was incorrect for cuDNN, because the cu12 package is metadata-only.

Assumption 2: That nvidia-cudnn-cu12 at version 9.19.0.56 provides the same runtime libraries as nvidia-cudnn-cu13. This was incorrect—the cu12 variant is a metapackage.

Assumption 3: That reinstalling nvidia-cudnn-cu13 would restore libcudnn.so.9 without breaking anything. This was correct—the verification in message [msg 9759] confirms it.

Assumption 4: That the cu13-provided cuDNN is binary-compatible with cu128 PyTorch. This is likely correct given the soname versioning, but the true test would come only when PyTorch actually imports successfully (which the assistant would need to verify next).

The Broader Significance

Message [msg 9759] is a textbook example of a verification step in systems debugging. It is short, precise, and targeted. The assistant does not run a generic "check everything" command; it runs a command specifically designed to answer one question: "Are the cuDNN shared libraries present now?"

This targeting is important because the debugging process had already narrowed the hypothesis space. The assistant knew:

Conclusion

Message [msg 9759] is a small but critical node in the network of decisions, actions, and verifications that constitute a complex debugging session. It demonstrates the importance of understanding the packaging details of the ML ecosystem, the value of targeted verification after a fix, and the reality that even well-reasoned decisions can have unintended consequences. The assistant's journey through the CUDA version maze—from discovering contamination, to breaking the environment, to understanding metadata-only packages, to applying a pragmatic fix—is a microcosm of the challenges faced by anyone building and maintaining deep learning infrastructure. In the end, a simple find command, returning five file paths, was the quiet confirmation that the path forward was clear.