The Hidden Dependency: Tracing a CUDA Library Conflict in DFlash Training

Introduction

In the midst of a high-stakes debugging session aimed at restoring training throughput for a speculative decoding model (DFlash), an assistant working on an opencode session encountered a seemingly simple but deeply revealing problem: after cleaning up contaminated CUDA packages from the Python environment, PyTorch could no longer find libcudnn.so.9. Message [msg 9753] captures a pivotal moment in this debugging journey—a moment where the assistant pauses to reason about why a package that should provide a critical shared library does not, and where the true source of that library might actually reside.

This message is small in volume but dense with diagnostic reasoning. It represents the transition from a surface-level fix (uninstalling conflicting packages) to a deeper investigation of how PyTorch's CUDA dependencies are actually structured. The assistant's thought process in this message reveals assumptions about Python packaging conventions, the relationship between PyTorch distributions and system CUDA libraries, and the fragility of deep learning environment management.

Context: The Road to the Broken Import

To understand why message [msg 9753] exists, we must trace the events that led to it. The session had been wrestling with a persistent training throughput problem. The DFlash training pipeline, which uses multiple GPU worker processes for speculative decoding, had plateaued at 12.8 Ktok/s—far below the expected 20 Ktok/s seen in earlier runs. The assistant had been systematically investigating possible causes: GPU memory pressure, queue contention between target and drafter processes, dataset composition changes, and—most critically—environment contamination.

The breakthrough came when the assistant discovered that CUDA 13 packages were still present in the Python virtual environment even after reverting PyTorch back to the cu128 (CUDA 12.8) version. The nvidia-cudnn-cu13, nvidia-nccl-cu13, nvidia-cusparselt-cu13, and nvidia-nvshmem-cu13 packages, along with SGLang and flashinfer components, had been installed during an earlier attempt to use CUDA 13.0 and were never removed. The assistant hypothesized that these leftover packages were intercepting CUDA calls and causing conflicts with the cu128-based PyTorch, potentially explaining the throughput degradation.

Acting on this hypothesis, the assistant killed all running processes, confirmed the GPUs were idle, and uninstalled ten packages including all the cu13 libraries. It also cleared the torch compile cache (/tmp/torchinductor_root/) to ensure no stale compiled kernels remained. The environment appeared clean—a uv pip list confirmed no cu13 packages remained.

But then came the crash. A simple Python command to verify the PyTorch installation failed with:

ImportError: libcudnn.so.9: cannot open shared object file: No such file or directory

The assistant had removed nvidia-cudnn-cu13, which was apparently the only package providing the cuDNN shared library that PyTorch needed at runtime. A quick check showed that nvidia-cudnn-cu12 was installed, but it wasn't being found by the dynamic linker. This is the precise moment captured in message [msg 9753].

The Reasoning: Tracing the cuDNN Dependency

Message [msg 9753] opens with the assistant's internal reasoning, which reveals a careful, hypothesis-driven thought process:

The nvidia-cudnn-cu12 package might not ship the .so files — it might be a pure Python package that requires the system CUDA cudnn library.

This is a critical insight. The assistant is questioning a fundamental assumption: that a pip package named nvidia-cudnn-cu12 would contain the actual compiled shared library (libcudnn.so). In the NVIDIA pip packaging ecosystem, there are two distinct categories of packages. Some, like nvidia-cudnn-cu12, are wrapper or binding packages that provide Python-level interfaces to CUDA libraries that must be installed separately (either through the system CUDA toolkit or through another package). Others, like the older nvidia-cudnn-cu13 that was just uninstalled, actually bundle the shared libraries themselves.

The assistant's reasoning continues with an alternative hypothesis:

Actually, looking at the torch package itself, torch cu128 should bundle its own CUDA libraries including cudnn.

This reveals an understanding of how PyTorch distributions work. PyTorch wheels for specific CUDA versions (like cu128 for CUDA 12.8) are typically self-contained—they bundle their own copies of CUDA runtime libraries, cuDNN, NCCL, and other dependencies to avoid relying on system-level installations. The torch package's lib/ directory should contain these bundled libraries. If torch cu128 bundles its own cuDNN, then the nvidia-cudnn-cu12 package might be irrelevant, and the real question becomes why torch's bundled libraries aren't being found.

The assistant then executes a bash command to test this hypothesis:

ssh -o ConnectTimeout=10 root@10.1.2.6 'pct exec 200 -- bash -c "find /root/venv -path \"*/torch/lib*\" -name \"*cudnn*\" 2>/dev/null"'

The command searches recursively within the torch installation's lib directory for any files matching *cudnn*. The result is telling: no output. The torch lib directory contains no cuDNN libraries.

What the Message Reveals

This negative result is profoundly informative. It tells us that:

  1. Torch cu128 does NOT bundle cuDNN. Despite being a CUDA-specific wheel, this particular PyTorch build expects cuDNN to be provided externally. This is unusual—most official PyTorch wheels bundle cuDNN—but it may be a custom build or a specific variant.
  2. The nvidia-cudnn-cu12 package is indeed a wrapper, not a library provider. The assistant's first hypothesis was correct: this package provides Python bindings but expects the actual libcudnn.so to be installed elsewhere.
  3. The only source of libcudnn.so.9 was the now-deleted nvidia-cudnn-cu13 package. This creates a dilemma: the cu13 version was removed to eliminate conflicts, but it was also the sole provider of a critical dependency.
  4. The environment is in a broken but recoverable state. The assistant now knows exactly what the problem is and can formulate a targeted fix—either reinstalling a cu12-compatible cuDNN package or finding an alternative source for the library.

Assumptions and Their Consequences

This message exposes several assumptions that shaped the debugging trajectory:

Assumption 1: Package names reflect their contents. The assistant assumed that nvidia-cudnn-cu12 would function similarly to nvidia-cudnn-cu13—both named with the cuXX suffix, both presumably providing cuDNN for their respective CUDA versions. This turned out to be false. The cu13 package bundled the .so files; the cu12 package did not. This inconsistency in NVIDIA's packaging ecosystem is a known source of confusion.

Assumption 2: Uninstalling conflicting packages is safe if the "correct" version is installed. The assistant checked that nvidia-cudnn-cu12 was present before removing the cu13 packages, but didn't verify that the cu12 package actually provided the required shared library. This oversight caused the immediate crash.

Assumption 3: PyTorch cu128 wheels are self-contained. Many PyTorch wheels do bundle CUDA libraries, but this particular build apparently does not. The assistant's fallback hypothesis—that torch itself would provide cuDNN—was also incorrect.

The Broader Significance

While message [msg 9753] is brief, it represents a crucial turning point in the debugging session. The assistant has now pinpointed the exact nature of the dependency failure and can proceed with a targeted solution. More importantly, the reasoning process demonstrates a methodical approach to diagnosing environment issues: form hypotheses about how packages are structured, test them with precise queries, and use negative results to refine understanding.

The message also highlights a recurring challenge in ML infrastructure: the complexity of CUDA dependency management. With multiple packaging systems (pip, conda, system packages), multiple CUDA versions, and inconsistent conventions across NVIDIA's own packages, even experienced practitioners can be tripped up by subtle environment issues. The assistant's willingness to question its own assumptions—first about the cu13 contamination, then about the cu12 package contents, then about torch's self-containment—is a model of effective debugging.

Conclusion

Message [msg 9753] captures a moment of diagnostic clarity in a complex debugging session. The assistant, having inadvertently broken the Python environment by removing a critical but contaminated dependency, works through hypotheses about where libcudnn.so.9 actually comes from. The bash command's empty result confirms that neither nvidia-cudnn-cu12 nor torch cu128 provides the needed library, setting the stage for the next round of fixes. This message exemplifies the kind of systematic reasoning that separates effective debugging from trial-and-error, and it reveals the hidden complexity beneath what should be a simple package management operation.