The Symlinking Dead End: A User's Intervention in the CUDA Dependency Maze

"what is this, why do you expect completely random symlinking to work? How about we install correct versions of software cleanly that can reasonably be expected to work together? Nightly etc, that work on this quite recent blackwell sm_120"

This message, sent by the user at index 11457 in a long-running opencode session, is a moment of decisive intervention. It arrives after a cascade of increasingly desperate technical workarounds by the AI assistant, and it fundamentally reframes the problem from one of patching symptoms to one of architectural compatibility. To understand why this message was written—and why it matters—we must trace the chain of events that led to it.

The Descent into Symlinking

The story begins with a straightforward goal: deploy the Kimi K2.6 model with SGLang on a machine equipped with 8× RTX PRO 6000 Blackwell GPUs (compute capability sm_120). The assistant had already resolved numerous infrastructure issues—CUDA toolkit installation, FlashInfer SM120 compatibility, pipeline parallelism support—and was attempting to benchmark the deployed model's throughput.

At [msg 11442], the assistant installed CUDA 13.0 toolkit (cuda-nvcc-13-0) via apt, alongside the system's existing CUDA 12.8 installation. The reasoning was sound: Blackwell GPUs are recent hardware, and a newer CUDA toolkit would provide better support. The assistant updated systemd service files to point CUDA_HOME at /usr/local/cuda-13.0 and cleared stale JIT caches ([msg 11443]). The service started successfully after 630 seconds ([msg 11444]).

But when the assistant attempted to benchmark the model ([msg 11445]), the service crashed. The journal logs revealed a FlashInfer JIT compilation failure: fatal error: curand.h: No such file or directory ([msg 11447]). FlashInfer, a library for efficient attention kernels, was trying to JIT-compile CUDA code at runtime, and CUDA 13.0's include directory lacked the curand.h header—a header for random number generation that exists in CUDA 12.8 but was missing from the minimal cuda-nvcc-13-0 package installation.

The assistant's response was to search for a cuda-curand-dev-13-0 package ([msg 11448]). It didn't exist. The apt repository had libcurand-dev-12-8 but no corresponding 13.0 package. At this point, the assistant made a fateful decision: install the CUDA 12.8 curand development package and symlink the headers into CUDA 13.0's include directory ([msg 11450]).

ln -sf /usr/local/cuda-12.8/include/curand.h /usr/local/cuda-13.0/include/curand.h
ln -sf /usr/local/cuda-12.8/include/curand_kernel.h /usr/local/cuda-13.0/include/curand_kernel.h
# ... 13 more symlinks for every curand header

This worked—briefly. The service started, and the assistant cleared caches and restarted ([msg 11451]). But when benchmarking was attempted again ([msg 11452]), the service crashed with a new error: fatal error: curand_mrg32k3a.h: No such file or directory ([msg 11454]). The symlinked curand_kernel.h from CUDA 12.8 internally included curand_mrg32k3a.h, which hadn't been symlinked.

The assistant's response was to symlink all curand headers ([msg 11455]):

cd /usr/local/cuda-13.0/include
for f in /usr/local/cuda-12.8/include/curand*; do
    bn=$(basename $f)
    [ ! -e $bn ] && ln -sf $f $bn
done

This time, the user aborted the startup wait ([msg 11456]). They had seen enough.

Why This Message Was Written: The Reasoning and Motivation

The user's message is sharp and direct. It opens with "what is this"—a rhetorical question expressing disbelief at the approach being taken. The phrase "completely random symlinking" captures the essence of the problem: the assistant was treating each missing header as an isolated incident, symlinking one file at a time, without any guarantee that the chain of dependencies would terminate cleanly. CUDA header files are not independent; they include each other in complex ways, and mixing headers from CUDA 12.8 with a CUDA 13.0 toolchain is undefined behavior. The next missing header could be anything—a cuda.h internal, a math_functions.h specialization, or something deeper in the compiler toolchain.

The user's motivation is clear: they want a system that is principled rather than patched. The phrase "install correct versions of software cleanly that can reasonably be expected to work together" reveals a design philosophy. Software dependencies form a compatibility graph; the nodes in that graph (CUDA toolkit, FlashInfer, SGLang, PyTorch) have explicit version compatibility ranges. Mixing versions across those ranges—using CUDA 13.0 headers with FlashInfer compiled for CUDA 12.8—is not a configuration; it's a gamble.

The user specifically mentions "Nightly etc, that work on this quite recent blackwell sm_120." This is a concrete proposal: instead of trying to force CUDA 13.0 to work with a FlashInfer version that doesn't support it, use nightly builds of SGLang and FlashInfer that are already compatible with Blackwell and the available CUDA toolchain. The sm_120 compute capability is key—Blackwell is new enough that only recent nightly builds are likely to include the necessary kernel specializations.

Assumptions Made and Mistakes Revealed

The assistant made several incorrect assumptions during this episode:

First assumption: CUDA 13.0 is a drop-in replacement for CUDA 12.8. The assistant assumed that installing cuda-nvcc-13-0 would provide a complete development environment. In reality, the apt package cuda-nvcc-13-0 is minimal—it includes the compiler but not the full CUDA toolkit with all headers and libraries. The full CUDA 13.0 toolkit requires installing many additional packages (cuda-toolkit-13-0), which may not have been available or compatible with the system's driver version (595.71.05).

Second assumption: Symlinking headers is a valid compatibility strategy. This is the assumption the user directly challenges. Header files are not independent artifacts; they are part of a coherent SDK. A header from CUDA 12.8 may reference types, macros, or functions that don't exist in CUDA 13.0's runtime libraries, or may rely on compiler behavior that differs between versions. The fact that curand_kernel.h from 12.8 included curand_mrg32k3a.h (which wasn't in the initial symlink set) proves this fragility.

Third assumption: The problem is a missing header, not a version mismatch. The assistant kept treating each compilation error as a discrete missing-file problem, rather than recognizing that the fundamental issue was an incompatible software stack. The FlashInfer version installed (0.6.8.post1) was built for a specific CUDA version, and using it with a different CUDA version was never going to work reliably.

Fourth assumption: The service started successfully, so the fix worked. After the first round of symlinks, the service started and passed its health check ([msg 11451]). The assistant assumed this meant the problem was solved. But FlashInfer's JIT compilation happens lazily—it compiles kernels on first use, not at import time. The service started because no sampling kernels had been invoked yet. The crash only occurred when the benchmark script actually tried to generate tokens.

Input Knowledge Required

To fully understand this message, the reader needs knowledge of:

Output Knowledge Created

This message creates several important outputs for the conversation:

  1. A decision point: The assistant must abandon the symlinking approach and adopt a clean-install strategy. This redirects the entire trajectory of the session.
  2. A prioritization of correctness over expedience: The user establishes that a working system must be built from compatible components, not patched together.
  3. A concrete direction: "Nightly etc" gives the assistant a specific search space—find nightly builds of SGLang and FlashInfer that support Blackwell natively.
  4. A framing of the problem: The issue is reframed from "missing headers" to "version incompatibility between CUDA 13.0 and FlashInfer." This is a more accurate and actionable diagnosis.

The Thinking Process Visible in the Message

The user's message is concise, but its structure reveals a clear thought process:

  1. Observation: The assistant is symlinking headers across CUDA versions.
  2. Evaluation: This is "completely random" and unlikely to work.
  3. Diagnosis: The root cause is that software versions are incompatible.
  4. Prescription: Install versions that are designed to work together.
  5. Specific guidance: Use nightly builds that target Blackwell (sm_120). The rhetorical opening ("what is this") signals frustration, but the message quickly moves to constructive guidance. The user doesn't just criticize—they provide an alternative approach. The mention of "nightly" and "sm_120" shows domain expertise: they know that Blackwell support is recent and likely only available in development/nightly branches of the relevant projects.

Conclusion

Message 11457 is a turning point in the session. It halts a descending spiral of increasingly fragile workarounds and reorients the work toward a principled solution. The assistant had been operating under the assumption that any fix that got the service running was acceptable, even if it meant mixing incompatible software versions. The user's intervention re-establishes that the goal is not just a running service, but a correct and maintainable one—a system where each component is compatible with the others by design, not by symlink.