The Missing Symlink: How Two ln -sf Commands Unblocked SGLang on Desktop Blackwell GPUs

In the sprawling, multi-session effort to deploy the GLM-5-NVFP4 model using SGLang on a cluster of 8× RTX PRO 6000 Blackwell GPUs, there is a message that appears, at first glance, utterly mundane. Message 9528 is a single bash command executed over SSH inside a Proxmox LXC container. It creates two symbolic links:

ln -sf libcudart.so.13 /root/venv/lib/python3.12/site-packages/nvidia/cu13/lib/libcudart.so
mkdir -p /root/venv/lib/python3.12/site-packages/nvidia/cu13/lib/stubs
ln -sf /usr/lib/x86_64-linux-gnu/libcuda.so /root/venv/lib/python3.12/site-packages/nvidia/cu13/lib/stubs/libcuda.so

Two symlinks. That is the entire message. Yet these two lines represent the final, decisive stroke in a grueling debugging session spanning dozens of messages, multiple failed compilation attempts, version mismatches across three CUDA toolkit installations, and a deep investigation into the internals of flashinfer's JIT compilation pipeline. Understanding why these two symlinks were necessary requires reconstructing the entire chain of reasoning that led to them — a chain that reveals the fragility of modern GPU software stacks and the detective work required to make bleeding-edge hardware work with bleeding-edge software.

The Context: SM120 and the Missing Cubins

The story begins with a hardware mismatch. The RTX PRO 6000 Blackwell GPU is built on NVIDIA's SM120 architecture — the desktop variant of Blackwell. SGLang, the inference serving engine chosen for this deployment, relies on flashinfer for its attention kernel implementations. Flashinfer ships pre-compiled cubins (CUDA binary files) for common GPU architectures: SM90 (Hopper, used in datacenter H100 GPUs) and SM100 (datacenter Blackwell, used in B100/B200). But SM120 — desktop Blackwell — has no pre-compiled cubins in the flashinfer wheel distribution.

This was discovered in [msg 9513], when the assistant ran a file search and found that all flashinfer cubins were labeled sm100f, with zero pre-compiled binaries for SM120. The only SM120-related files were Jinja templates — source code for JIT compilation. This means that on SM120, flashinfer must JIT-compile its attention kernels at runtime. And JIT compilation, in turn, requires a perfectly matched CUDA toolchain: the nvcc compiler, the CUDA runtime headers, the CUDA driver library, and the ptxas assembler must all speak the same version language.

This is where the trouble began.

The Cascade of Version Mismatches

The Python environment had been set up using uv pip install of nvidia-cu13 and related packages — a pip-distributed subset of the CUDA toolkit. This is a common pattern in ML deployments: rather than installing the full multi-gigabyte CUDA toolkit from NVIDIA's repository, practitioners install individual pip packages (nvidia-cuda-nvcc, nvidia-cuda-runtime, nvidia-cuda-nvrtc, etc.) that provide just enough of the toolchain to compile and run CUDA code.

The problem was that these pip packages were at inconsistent versions. The nvidia-cuda-nvcc package had been upgraded to 13.2.78 (providing nvcc 13.2), but the nvidia-cuda-runtime package remained at 13.0.96 (providing headers declaring CUDART_VERSION as 13000). The nvidia/cu13/include/ directory — which flashinfer's JIT build system picked up via -isystem include paths — contained CUDA 13.0 headers, while the compiler was CUDA 13.2.

This mismatch manifested in two distinct errors depending on the exact configuration. With nvcc 13.0, the bundled CCCL headers in flashinfer generated PTX version 9.2, which ptxas 13.0 could not assemble (it only supports up to PTX 9.0). With nvcc 13.2, the CCCL headers detected a mismatch between the compiler version and the CUDART_VERSION macro in the CUDA 13.0 headers, triggering a fatal error in cooperative_groups.h.

The assistant resolved this in [msg 9520] by upgrading the runtime, nvrtc, and cupti packages to match nvcc at version 13.2. After this fix, the CUDART_VERSION macro read 13020, confirming header-compiler alignment.

The Linker Error: A Tale of Two Directories

With the version mismatch resolved, the assistant relaunched SGLang in [msg 9522]. The JIT compilation began. Ten compilation steps completed successfully — the CUDA source files compiled to object files without error. But the eleventh step, linking, failed:

/bin/ld: cannot find -lcudart: No such file or directory

The linker was looking for libcudart.so in the library search path provided by flashinfer's build system: -L/root/venv/lib/python3.12/site-packages/nvidia/cu13/lib64. But the pip-installed CUDA toolkit places its libraries in lib/, not lib64/. The lib64 directory simply did not exist.

This was diagnosed in [msg 9526]. The assistant's reasoning reveals the insight: "The library is in nvidia/cu13/lib not lib64." The fix was a symlink from lib64 → lib, applied in the same message.

But this revealed a second linker problem. Even with lib64 pointing to the right place, the linker also needed libcuda.so — the CUDA driver library. The pip-installed CUDA packages do not include the driver library (that comes from the NVIDIA kernel driver installation). The lib/ directory contained libcudart.so.13, libcudadevrt.a, and libcudart_static.a, but no libcuda.so. The system had libcuda.so at /usr/lib/x86_64-linux-gnu/libcuda.so (from the NVIDIA driver version 595.71.05), but flashinfer's linker flags pointed into the cu13 package directory, not the system path.

Message 9528: The Two Symlinks

Message 9528 addresses both problems simultaneously. The first symlink solves a naming issue:

ln -sf libcudart.so.13 /root/venv/lib/python3.12/site-packages/nvidia/cu13/lib/libcudart.so

The file libcudart.so.13 exists in the lib/ directory — this is the versioned soname used by the dynamic linker at runtime. But the linker (invoked during JIT compilation) searches for -lcudart, which resolves to libcudart.so — the unversioned symlink that conventionally points to the versioned library. The pip package ships only the versioned file, omitting this symlink. Creating it tells the linker "here is your libcudart.so."

The second symlink addresses the missing driver stub:

mkdir -p /root/venv/lib/python3.12/site-packages/nvidia/cu13/lib/stubs
ln -sf /usr/lib/x86_64-linux-gnu/libcuda.so /root/venv/lib/python3.12/site-packages/nvidia/cu13/lib/stubs/libcuda.so

The flashinfer JIT build system expects to find libcuda.so in a stubs/ subdirectory of the CUDA library path — this is a convention from the full CUDA toolkit installation, where stubs/libcuda.so provides a linker stub for the driver API. The pip packages do not create this directory. By creating it and pointing it to the system's actual libcuda.so, the assistant provides the linker with what it expects.

The Reasoning Behind the Fix

What makes this message noteworthy is not the commands themselves — they are trivial — but the chain of inference that produced them. The assistant had never seen this particular error before. There was no Stack Overflow answer, no GitHub issue, no documentation explaining that nvidia-cuda-runtime pip packages omit the libcudart.so symlink and the stubs/ directory. The fix was derived from first principles: understanding how the Unix linker resolves library names, knowing the conventions of CUDA toolkit directory layout, and inferring from the error message what the build system expected versus what the filesystem actually contained.

The reasoning process visible in the surrounding messages shows a methodical narrowing of possibilities. The assistant first confirmed the exact error (/bin/ld: cannot find -lcudart), then inspected the filesystem to see what files actually existed in the library directory ([msg 9527]), then cross-referenced the linker's search path against the available files. The discovery that libcudart.so.13 existed but libcudart.so did not, and that stubs/ was entirely absent, directly dictated the two symlink commands.

Assumptions and Their Validity

The fix rests on several assumptions. First, that the linker's convention of searching for lib<name>.so when given -l<name> holds — this is a fundamental Unix linking behavior and is universally true. Second, that the versioned soname libcudart.so.13 is the correct target for the unversioned symlink — this is the standard convention, but a broken or mismatched soname could cause runtime loading failures even if linking succeeds. Third, that the system's /usr/lib/x86_64-linux-gnu/libcuda.so is compatible with the CUDA 13.2 toolkit — this is the most fragile assumption, as the driver library must match the CUDA runtime version. In this case, the NVIDIA driver 595.71.05 was installed separately and was known to be compatible with CUDA 13.x, so the assumption was safe.

One potential mistake in the reasoning is the assumption that the stubs/ directory convention is strictly necessary for flashinfer's JIT build. The error message that triggered this fix was the -lcudart failure; the libcuda.so stub was added preemptively based on knowledge of how CUDA toolkit builds work, not because a specific error demanded it. This is a prophylactic fix — solving a problem that hasn't yet occurred but is expected to. If the assistant had been wrong about the stubs/ directory being needed, the fix would have been harmless (an unused directory and symlink), so the risk was minimal.

Input Knowledge Required

To understand this message, the reader needs knowledge of several domains. Unix linking conventions: the difference between -lcudart (which searches for libcudart.so), the soname mechanism (libcudart.so.13), and how the linker resolves library paths. CUDA toolkit directory structure: the convention of lib/ vs lib64/, the stubs/ subdirectory for driver stubs, and the distinction between the CUDA runtime library (libcudart) and the CUDA driver library (libcuda). The pip nvidia-cu13 package layout: that these packages ship only the versioned library files and omit the unversioned symlinks and stubs that a full CUDA toolkit installation would provide. And the flashinfer JIT build system: that it constructs linker search paths pointing into the pip-installed CUDA directory rather than system paths.

Output Knowledge Created

This message creates practical knowledge about deploying SGLang with flashinfer on SM120 GPUs using pip-installed CUDA toolkits. The two symlink commands are a reusable recipe: any deployment using nvidia-cuda-runtime pip packages for JIT compilation with flashinfer will encounter the same missing libcudart.so symlink and missing stubs/ directory. More broadly, the message documents a class of environment debugging that arises from the gap between pip-distributed subsets of the CUDA toolkit and the full toolkit installation that build systems expect. The fix is not specific to flashinfer or SGLang — any CUDA JIT compilation that links against pip-installed CUDA libraries will need these same symlinks.

The Broader Pattern

Message 9528 is a microcosm of a larger pattern in modern ML infrastructure: the tension between lightweight package management (pip, uv) and the heavyweight, convention-laden build systems inherited from traditional HPC and GPU computing. The pip packages provide the content of the CUDA toolkit — the headers, the compiler, the runtime libraries — but they do not provide the scaffolding: the symlinks, the directory conventions, the stub files that build systems implicitly depend on. Every CUDA JIT compilation on a pip-based environment requires rediscovering and recreating this scaffolding. The two symlinks in message 9528 are not a workaround for a bug; they are a reimplementation of a convention that the full CUDA toolkit installer would have created automatically.

In this sense, the message is a testament to the state of the art in 2025: bleeding-edge hardware (SM120 Blackwell), bleeding-edge inference engines (SGLang nightly), bleeding-edge kernel libraries (flashinfer with JIT), and a packaging ecosystem (pip CUDA stubs) that has not quite caught up to the integration demands placed upon it. The assistant's contribution is not the symlink commands themselves — any systems engineer could have written them — but the diagnostic chain that identified which symlinks were missing, why they were missing, and what they needed to point to. That chain of reasoning, compressed into two lines of bash, is the real content of message 9528.