The Missing Stub: How a Single Library File Blocked FlashInfer JIT Compilation on Blackwell SM120

In the high-stakes world of deploying large language models on cutting-edge hardware, the smallest missing file can bring an entire pipeline to a halt. Message 9527 in this opencode session captures one such moment—a brief but critical diagnostic step in the saga of getting SGLang's flashinfer backend to run on an NVIDIA Blackwell RTX PRO 6000 GPU with SM120 architecture. The message is deceptively simple: a single line of reasoning followed by a bash command that probes for library files. Yet this moment represents the culmination of an intricate debugging journey spanning CUDA toolkit version mismatches, PTX assembly incompatibilities, and linker configuration issues.

The Context: A Cascade of Compilation Failures

To understand why message 9527 was written, we must first appreciate the debugging hell that preceded it. The assistant was attempting to launch SGLang with flashinfer as the attention backend on an SM120 GPU—a desktop Blackwell architecture that flashinfer's pre-compiled kernels do not support. Flashinfer ships pre-compiled cubins only for SM90 (Hopper) and SM100 (datacenter Blackwell), leaving SM120 to rely entirely on Just-In-Time (JIT) compilation. This JIT path requires a perfectly matched CUDA toolchain: the nvcc compiler, the CUDA runtime headers, the PTX assembler (ptxas), and the linker must all speak the same version language.

The environment had been assembled from pip-installed NVIDIA CUDA packages, and a dangerous asymmetry had crept in. The nvidia-cuda-nvcc package was at version 13.2.78, but the nvidia-cuda-runtime package—which provides the critical header files—was stuck at version 13.0.96. When flashinfer's bundled CCCL (CUDA Core Compute Libraries) headers attempted to compile attention kernels for SM120, they generated PTX version 9.2 assembly, which the CUDA 13.0 ptxas could not consume. The error message was unambiguous: "Unsupported .version 9.2; current version is '9.0'."

The assistant had already resolved this by upgrading the runtime packages to CUDA 13.2 ([msg 9520]), which brought the header version to CUDART_VERSION 13020. The subsequent compilation attempt ([msg 9522]) succeeded through all 10 compilation steps but then failed at the linking stage with a new error: cannot find -lcudart. The linker was searching for libcudart.so in the lib64 subdirectory of the pip-installed CUDA toolkit, but the library resided in lib instead. A symlink from lib64 to lib fixed that problem ([msg 9526]).

The Message: A Second Linker Gap Discovered

The full text of message 9527 reads:

Also need libcuda.so stub for the linker:

[bash] ssh -o ConnectTimeout=10 root@10.1.2.6 "pct exec 200 -- bash -c 'ls /root/venv/lib/python3.12/site-packages/nvidia/cu13/lib/libcuda* 2>/dev/null; ls /root/venv/lib/python3.12/site-packages/nvidia/cu13/lib/stubs/ 2>/dev/null; ls /usr/lib/x86_64-linux-gnu/libcuda.so* 2>/dev/null'" 2>&1

Output: `` /root/venv/lib/python3.12/site-packages/nvidia/cu13/lib/libcudadevrt.a /root/venv/lib/python3.12/site-packages/nvidia/cu13/lib/libcudart.so.13 /root/venv/lib/python3.12/site-packages/nvidia/cu13/lib/libcudart_static.a /usr/lib/x86_64-linux-gnu/libcuda.so /usr/lib/x86_64-linux-gnu/libcuda.so.1 /usr/lib/x86_64-linux-gnu/libcuda.so.595.71.05 ``

The message opens with the assistant's reasoning: "Also need libcuda.so stub for the linker." This single sentence reveals a deep understanding of the CUDA linking process. The flashinfer JIT compilation doesn't just need libcudart.so (the CUDA runtime library); it also needs libcuda.so—the CUDA driver API library. During compilation, the linker resolves symbols against a stub version of libcuda.so that provides the API entry points without requiring an actual GPU driver context. This stub is typically distributed with the CUDA toolkit in a stubs/ subdirectory.

The assistant's bash command probes three locations to understand the landscape:

The pip CUDA toolkit's lib/ directory (/root/venv/lib/python3.12/site-packages/nvidia/cu13/lib/libcuda*): This reveals only libcudadevrt.a, libcudart.so.13, and libcudart_static.a. No libcuda.so stub exists here.
The pip CUDA toolkit's stubs/ directory (/root/venv/lib/python3.12/site-packages/nvidia/cu13/lib/stubs/): Empty or nonexistent—the pip package does not ship the driver stub.
The system's NVIDIA driver library path (/usr/lib/x86_64-linux-gnu/libcuda.so*): This reveals the real driver libraries installed by the NVIDIA driver package: libcuda.so, libcuda.so.1, and libcuda.so.595.71.05. The output confirms that the pip-installed CUDA toolkit is incomplete—it lacks the libcuda.so stub that the linker needs. The system does have libcuda.so from the NVIDIA driver installation, but this is the runtime driver library, not the compilation stub. Using it for linking could work in practice (the driver library exports the same symbols), but it's not the intended compilation artifact.

Assumptions and Knowledge

This message operates on several key assumptions. First, the assistant assumes that the flashinfer JIT compilation failure is a linking problem rather than a compilation problem—an assumption validated by the earlier successful compilation steps. Second, the assistant assumes that libcuda.so is required by the linker, which is correct for any CUDA program that calls driver API functions (as flashinfer's attention kernels do). Third, the assistant assumes that the pip-installed CUDA toolkit should include a libcuda.so stub, which turns out to be incorrect—the pip packages ship only the runtime components, not the driver stub.

The input knowledge required to understand this message is substantial. One must understand the CUDA compilation model: that .cu files are compiled by nvcc into object files, which are then linked by the system linker (ld) into shared objects (.so files). The linker resolves external symbols like cudaMemcpy, cudaMalloc, and other driver API calls against libcuda.so. Without this stub library, the linker cannot resolve these symbols and fails with an undefined reference error. One must also understand the distinction between the CUDA toolkit (which provides compilers, headers, and stub libraries for building CUDA programs) and the NVIDIA driver (which provides the runtime libcuda.so for running them).

The output knowledge created by this message is the precise state of the library environment: the pip CUDA toolkit at /root/venv/lib/python3.12/site-packages/nvidia/cu13/lib/ contains libcudart.so.13 (versioned) and libcudart_static.a but no libcuda.so stub; the system driver path /usr/lib/x86_64-linux-gnu/ contains libcuda.so (unversioned symlink), libcuda.so.1 (versioned symlink), and libcuda.so.595.71.05 (the actual driver library). This knowledge is immediately actionable: the assistant can create a symlink from the pip toolkit's library directory to the system's libcuda.so, or add the system library path to the linker's search path via -L flags or LD_LIBRARY_PATH.

The Broader Significance

Message 9527 is a textbook example of the "last mile" problem in ML infrastructure debugging. The assistant had successfully navigated through PTX version incompatibilities, header version mismatches, and a missing libcudart.so symlink—only to be stopped by one final missing file. The pattern is familiar to anyone who has deployed complex GPU software: the first error is rarely the real problem; it's merely the first symptom of a deeper configuration issue that surfaces only after the previous errors are resolved.

This message also highlights the fragility of pip-installed CUDA toolkits. While the nvidia-cuda-* pip packages are convenient for installation, they are not complete CUDA toolkit replacements. They omit critical components like the libcuda.so stub, the CUDA header files for driver API development, and tools like ptxas (which must be installed separately via nvidia-cuda-nvcc). For JIT compilation workflows like flashinfer's, where the full toolchain is invoked at runtime, these omissions become blocking issues.

The assistant's approach—probing specific file paths with targeted ls commands—reflects a systematic debugging methodology. Rather than guessing or trying random fixes, the assistant checks each hypothesized missing piece against the actual filesystem state. This pattern of "check, diagnose, fix, retry" is visible throughout the broader conversation and is the reason this particular debugging session ultimately succeeded.

The Thinking Process

The reasoning visible in message 9527 is minimal but dense. The assistant writes "Also need libcuda.so stub for the linker" as a continuation of the previous debugging thread. The word "also" connects this discovery to the earlier libcudart.so fix—the assistant is systematically enumerating the linker's dependencies. The use of the word "stub" is technically precise: the assistant knows that what's needed is not the full runtime driver library but the compilation stub that provides symbol resolution without driver initialization.

The bash command is structured to answer three questions in parallel: Does the pip toolkit have libcuda.so? Does it have a stubs/ subdirectory? Does the system provide libcuda.so? The output answers all three: no, no, and yes. This information immediately constrains the solution space. The assistant cannot use a pip-internal stub (it doesn't exist), so the fix must involve either pointing the linker to the system path or creating a compatibility symlink.

In the broader narrative of this opencode session, message 9527 represents the penultimate step before SGLang finally launches on SM120. The very next message ([msg 9528]) shows the assistant acting on this discovery: it creates a libcudart.so symlink (for the unversioned library name the linker expects) and, critically, creates a stubs/ directory under the pip CUDA toolkit path with a symlink to the system's libcuda.so. The command output simply reads "Symlinks created"—a terse confirmation that the final barrier has been removed. With these two symlinks in place, the flashinfer JIT compilation pipeline can resolve all its linker dependencies, and SGLang proceeds to launch successfully on the SM120 GPU.

But this message—this quiet moment of discovery—is where the last barrier is identified and understood. It is a reminder that in complex systems engineering, the difference between success and failure is often a single file in the right directory.