The Library Path That Almost Broke the Stack: Diagnosing `libnvrtc.so.13` After a CUDA 13 Upgrade

Introduction

In the high-stakes world of deploying large language models on cutting-edge hardware, the difference between a working system and a broken one often comes down to a single environment variable. Message <msg id=5321> captures precisely such a moment: the assistant, after an arduous multi-hour struggle with C++ ABI incompatibilities between PyTorch and the SGLang kernel library, finally lands on a compatible software combination—only to be blocked by a mundane but critical shared library path issue. The error libnvrtc.so.13: cannot open shared object file is both a relief and a frustration: a relief because it signals that the deeper ABI mismatch has been resolved, and a frustration because it reveals that the CUDA 13 runtime libraries, though physically present on disk, are invisible to the dynamic linker.

This message is a hinge point in the broader narrative of upgrading an 8× RTX PRO 6000 Blackwell GPU system from CUDA 12 to CUDA 13. It represents the moment when the assistant transitions from debugging abstract symbol-level incompatibilities to solving a concrete, well-understood system configuration problem. Understanding this message requires tracing the chain of reasoning that led to it, the assumptions that were validated and invalidated along the way, and the diagnostic knowledge that the assistant both consumed and produced.

The Message in Full

The assistant writes:

Different error now: libnvrtc.so.13: cannot open shared object file. This is the same error from issue #18392. The CUDA 13 runtime libraries are at /usr/local/cuda-13.0/lib64/ but they're not on the library path. Let me fix this:

``bash ssh root@10.1.230.174 'ls /usr/local/cuda-13.0/lib64/libnvrtc* && echo "---" && ldconfig -p | grep nvrtc' ``

Output: `` /usr/local/cuda-13.0/lib64/libnvrtc-builtins.alt.so /usr/local/cuda-13.0/lib64/libnvrtc-builtins.alt.so.13.0 /usr/local/cuda-13.0/lib64/libnvrtc-builtins.alt.so.13.0.88 /usr/local/cuda-13.0/lib64/libnvrtc-builtins.so /usr/local/cuda-13.0/lib64/libnvrtc-builtins.so.13.0 /usr/local/cuda-13.0/lib64/libnvrtc-builtins.so.13.0.88 /usr/local/cuda-13.0/lib64/libnvrtc-builtins_static.a /usr/local/cuda-13.0/lib64/libnvrtc-builtins_static.alt.a /usr/local/cuda-13.0/lib64/libnvrtc.alt.so /usr/local/cuda-13... ``

The output is truncated, but the key information is clear: the libraries exist at /usr/local/cuda-13.0/lib64/, and the ldconfig -p | grep nvrtc command returned nothing (not shown, but implied by the assistant's earlier statement that they're "not on the library path").

The Reasoning: Why This Message Was Written

To understand why this message exists, one must understand the torturous path that led to it. In the preceding messages ([msg 5294] through [msg 5320]), the assistant was locked in a battle with C++ ABI compatibility. The core problem was that sgl-kernel, a high-performance CUDA kernel library used by SGLang for serving large language models, ships precompiled shared objects (.so files) that are linked against a specific version of PyTorch's libc10_cuda.so. The C++ name mangling scheme encodes parameter types into function symbols, and a single changed type—from int to unsigned int in the c10_cuda_check_implementation function—caused the dynamic linker to reject the entire library.

The assistant traced this mismatch through multiple rounds of investigation. Using nm -D to inspect symbol tables, it discovered that sgl-kernel's common_ops.abi3.so expected the symbol _ZN3c104cuda29c10_cuda_check_implementationEiPKcS2_ib (where the trailing i denotes int), while PyTorch 2.10.0+cu130 provided _ZN3c104cuda29c10_cuda_check_implementationEiPKcS2_jb (where j denotes unsigned int). This one-character difference in a mangled symbol represented a fundamental ABI break between PyTorch 2.9.x and 2.10.x.

The assistant's decision to downgrade to torch==2.9.1+cu130 in message [msg 5319] was the critical strategic pivot. It was based on the assumption—validated by the error message changing from the ABI symbol mismatch to the libnvrtc.so.13 error—that the sgl-kernel wheels on the cu130 index were compiled against PyTorch 2.9.x. This assumption proved correct. The new error, libnvrtc.so.13: cannot open shared object file, was a fundamentally different class of problem: not an ABI incompatibility, but a missing library path configuration.

How Decisions Were Made

The decision to downgrade PyTorch was not made lightly. The assistant had previously tried PyTorch 2.12.0 nightly (cu130), PyTorch 2.10.0 stable (cu130), and various sgl-kernel wheel sources (the cu130 PyPI index, GitHub release artifacts). Each combination failed with the same ABI symbol error. The breakthrough came when the assistant referenced GitHub issue #18392, which documented a similar Blackwell + CUDA 13 setup with PyTorch: 2.9.1+cu130 and sgl_kernel: 0.3.21. This external evidence provided the missing piece: the sgl-kernel cu130 wheels were built against PyTorch 2.9.x, not 2.10.x or 2.12.x.

The assistant's diagnostic methodology was systematic and thorough. It used nm -D to inspect symbol tables, md5sum to verify that reinstalling the same wheel produced identical binaries, and ldconfig -p to check the dynamic linker cache. Each tool was deployed to answer a specific question: Is the wheel file different? Is the symbol expected by the .so different from what torch provides? Is the library path registered with the system linker?

The decision to run the specific bash command in this message—listing the contents of /usr/local/cuda-13.0/lib64/ and checking ldconfig -p for nvrtc—was a direct consequence of the assistant's hypothesis: "The CUDA 13 runtime libraries are at /usr/local/cuda-13.0/lib64/ but they're not on the library path." This hypothesis was formed by combining two observations: (1) the error message named libnvrtc.so.13 specifically, and (2) the assistant knew from the earlier CUDA 13 installation that the libraries lived in that directory. The ls command confirmed the libraries existed; the ldconfig command confirmed they were absent from the linker cache.

Assumptions Made by the Assistant

Several assumptions underpin this message, some explicit and some implicit:

The libraries physically exist. The assistant assumed that the CUDA 13 installation had placed libnvrtc.so.13 in /usr/local/cuda-13.0/lib64/. This was a reasonable assumption given that the CUDA 13 toolkit had been installed earlier in the session, but it needed verification. The ls command served this purpose.
The error is purely a path issue. The assistant assumed that once the library path was corrected, sgl-kernel would load successfully. This assumption was based on the error message changing from the ABI symbol error to the libnvrtc.so.13 error after switching to PyTorch 2.9.1—a strong signal that the ABI problem was resolved.
The ldconfig cache is authoritative. The assistant assumed that if ldconfig -p did not list libnvrtc.so.13, then the dynamic linker would not find it. This is correct for the default search path, but it's worth noting that LD_LIBRARY_PATH can override the linker cache—a fact the assistant likely planned to exploit in the next step.
The library naming convention is standard. The assistant assumed that libnvrtc.so.13 would be a symlink to a versioned file like libnvrtc.so.13.0.88. The ls output confirmed this convention was followed.
The same fix as issue #18392 would work. The assistant referenced issue #18392, which reported the same libnvrtc.so.13 error. The implicit assumption was that the fix for that issue—likely adding the CUDA library path to LD_LIBRARY_PATH or running ldconfig—would apply here as well.

Mistakes and Incorrect Assumptions

While the message itself is diagnostically sound, it's worth examining what earlier mistakes led to this point:

The assumption that the cu130 PyPI index provided compatible wheels. The assistant initially assumed that any wheel from the cu130 index would be compatible with any PyTorch cu130 build. This turned out to be false: the sgl-kernel wheels were built against a specific PyTorch minor version (2.9.x), and PyTorch 2.10.x introduced an ABI-breaking change. This is a subtle but important lesson about the fragility of C++ ABIs across PyTorch minor version bumps.
The assumption that nightly PyTorch would work. The assistant tried torch==2.12.0.dev20260226+cu130 (a nightly build) before trying the stable release. Nightlies are built against the latest CUDA toolkit and may have additional ABI changes not present in stable releases. The decision to try the nightly first was reasonable (it was the most recent), but it introduced unnecessary complexity.
The assumption that sgl-kernel version 0.3.21 from different sources would differ. The assistant downloaded the same wheel from both the PyPI index and GitHub releases, hoping for a different binary. The md5sum check confirmed they were identical, ruling out this hypothesis efficiently. These earlier mistakes are not visible in message <msg id=5321> itself, but they form the necessary context for understanding why the assistant arrived at this particular diagnostic step. The message represents a moment of clarity after a period of confusion—the assistant has finally found a combination that changes the error message, which is a strong signal of progress.

Input Knowledge Required

To fully understand this message, one needs:

Understanding of CUDA toolkit installation layout. Knowledge that NVIDIA's CUDA toolkit installs shared libraries under /usr/local/cuda-<version>/lib64/ is essential. This is a convention that has been consistent across CUDA versions for decades, but it's not universal knowledge.
Understanding of the Linux dynamic linker. The ldconfig command and the concept of a linker cache are fundamental to Linux system administration. The assistant's use of ldconfig -p | grep nvrtc to check whether the library is registered with the system linker demonstrates a solid grasp of how shared libraries are resolved at runtime.
Knowledge of the libnvrtc library. libnvrtc is the NVIDIA Runtime Compilation library, used for JIT-compiling CUDA kernels at runtime. It's a key dependency for any CUDA application that uses runtime compilation, including PyTorch's JIT and sgl-kernel's architecture-specific kernel loading. Understanding that libnvrtc.so.13 is the versioned soname for CUDA 13's runtime compilation library helps contextualize the error.
Familiarity with the SGLang ecosystem. The assistant is deploying the GLM-5-NVFP4 model using SGLang, a high-performance LLM serving framework. The sgl-kernel package provides optimized CUDA kernels for SGLang, and its loading mechanism (_load_architecture_specific_ops) attempts to load architecture-specific shared objects at import time. The libnvrtc dependency arises because sgl-kernel uses JIT compilation for Blackwell (SM120) architectures.
The history of GitHub issue #18392. The assistant references this issue, which documented the same libnvrtc.so.13 error on a Blackwell + CUDA 13 system. Knowing that this issue was resolved (closed as completed) gives confidence that the fix is straightforward.

Output Knowledge Created

This message creates several pieces of actionable knowledge:

Confirmation that the ABI mismatch is resolved. The fact that the error changed from the c10_cuda_check_implementation symbol mismatch to libnvrtc.so.13 proves that PyTorch 2.9.1+cu130 is ABI-compatible with sgl-kernel 0.3.21+cu130. This is a valuable data point for anyone deploying SGLang on CUDA 13.
Documentation of the CUDA 13 library path. The ls output confirms that the CUDA 13 runtime libraries are installed at /usr/local/cuda-13.0/lib64/ and that libnvrtc.so.13 exists as a symlink to a versioned file. This is useful for debugging similar issues.
Diagnosis of the library path problem. The ldconfig -p output (or lack thereof) confirms that the CUDA 13 libraries are not registered with the system linker. This identifies the root cause and points toward the fix: either adding the path to ld.so.conf and running ldconfig, or setting LD_LIBRARY_PATH to include /usr/local/cuda-13.0/lib64/.
A validated diagnostic workflow. The sequence of steps—check the error message, hypothesize the cause, verify with ls and ldconfig—constitutes a reusable debugging pattern for shared library issues. This workflow is transferable to other contexts beyond CUDA and SGLang.

The Thinking Process Visible in the Message

The assistant's reasoning is laid out with remarkable clarity in this message. There are three distinct phases:

Phase 1: Error Recognition. The assistant sees libnvrtc.so.13: cannot open shared object file and immediately recognizes it as "the same error from issue #18392." This pattern-matching is a hallmark of experienced debugging: the assistant has built a mental database of error signatures from reading GitHub issues, and it can rapidly classify new errors against known patterns. The reference to issue #18392 also serves as a justification for the assistant's confidence—this isn't a novel problem, it's a known issue with a known fix.

Phase 2: Hypothesis Formation. The assistant states: "The CUDA 13 runtime libraries are at /usr/local/cuda-13.0/lib64/ but they're not on the library path." This hypothesis is formed by combining two pieces of knowledge: (a) the standard CUDA installation layout, and (b) the fact that the dynamic linker is reporting a "cannot open" error, which typically means the library isn't in any of the standard search paths. The assistant doesn't need to check the library path configuration first—it can infer the cause from the error message alone, using its knowledge of how the system is organized.

Phase 3: Verification. The assistant then runs a bash command to verify both parts of its hypothesis. The ls command checks "do the libraries exist where I expect them?" The ldconfig -p | grep nvrtc command checks "are they registered with the system linker?" This is a textbook example of hypothesis-driven debugging: state your hypothesis, then design experiments that can falsify it.

The command itself is elegantly constructed. It uses && to chain commands, so that ldconfig -p | grep nvrtc only runs if ls succeeds. This is a small but meaningful design choice: if the libraries don't exist at the expected path, there's no point checking the linker cache. The command also uses echo "---" to visually separate the two outputs, making the results easier to parse in a terminal.

The Broader Significance

Message <msg id=5321> is a microcosm of the entire CUDA 13 upgrade saga. It demonstrates that even after solving deep, subtle ABI compatibility problems, the final barrier to a working system can be something as mundane as a missing library path. The contrast is striking: the assistant spent hours tracing C++ name mangling across different PyTorch versions, only to be stopped by a configuration issue that can be fixed with a single export LD_LIBRARY_PATH command.

This pattern is common in systems engineering. The hardest problems are often not the ones that take the longest to solve—they're the ones that take the longest to reach. The ABI mismatch was a hard problem because it required deep knowledge of C++ compilation, symbol mangling conventions, and the PyTorch build system. The library path problem is a trivial problem in comparison, but it was invisible until the ABI problem was solved. The assistant couldn't have known about the libnvrtc.so.13 error until it found a PyTorch version that was ABI-compatible with sgl-kernel.

The message also illustrates the value of reading GitHub issues. The assistant's reference to issue #18392 is not just a citation—it's a strategic decision to leverage community knowledge. Rather than debugging the libnvrtc.so.13 error from scratch, the assistant can draw on the experience of others who have already encountered and solved it. This is a powerful debugging technique that experienced engineers use routinely: when you see an error, search for it before trying to solve it yourself.

Conclusion

Message <msg id=5321> captures a moment of transition in a complex debugging session. The assistant has navigated through the treacherous waters of C++ ABI compatibility, emerged with a working software combination, and now faces a final, solvable configuration problem. The message is a testament to systematic debugging: form a hypothesis, verify it with targeted commands, and use every tool at your disposal—from nm -D for symbol inspection to ldconfig for linker cache analysis. The libnvrtc.so.13 error, while mundane, represents progress. It means the deep problems are solved, and only the shallow ones remain.

The Library Path That Almost Broke the Stack: Diagnosing libnvrtc.so.13 After a CUDA 13 Upgrade