The Verification That Saved a Deployment: CUDA Toolkit Cleanup on Blackwell GPUs

In the high-stakes world of deploying large language models on cutting-edge hardware, the smallest infrastructure misstep can cascade into hours of debugging. This article examines a single message from an opencode coding session — a bash command executed at a critical inflection point — that represents the pivot from a fragile, hacky approach to a clean, professional infrastructure fix. The message, sent by an AI assistant to a remote machine at 10.1.2.200, is a verification step: it checks whether a freshly installed CUDA 13.0 toolkit is complete and functional after the assistant was rightly called out for attempting to symlink headers across incompatible toolkit versions.

Context: A Cascade of Toolkit Failures

To understand why this message exists, we must trace the chain of failures that led to it. The assistant was deploying the Kimi K2.6 model with SGLang on a machine equipped with 8× NVIDIA RTX PRO 6000 Blackwell GPUs (architecture sm_120). This is a very recent GPU architecture, and software support for it is still being actively developed. The machine had two CUDA toolkits installed: a full CUDA 12.8 toolkit and a partial CUDA 13.0 toolkit that included only nvcc and cudart — enough for basic compilation but missing critical libraries like curand, cublas, and cusparse.

The problem emerged when FlashInfer (version 0.6.8.post1), a dependency of SGLang used for efficient attention kernels, attempted to JIT-compile sampling kernels for the SM120 architecture. The compilation failed with a fatal error: curand.h: No such file or directory. The CUDA 13.0 installation was incomplete — it had the compiler but not the development libraries. FlashInfer's JIT compilation pipeline needed curand.h and related headers to build its sampling kernels, and those headers simply didn't exist in the partial CUDA 13.0 installation.

The assistant's first response was to patch the problem by symlinking headers from the CUDA 12.8 installation into CUDA 13.0's include directory. This approach had a certain pragmatic appeal — it was fast, it required no package downloads, and it superficially addressed the immediate error. But as the assistant discovered, curand.h itself includes other headers (curand_kernel.h, curand_mrg32k3a.h, etc.), and symlinking only the top-level file led to a cascade of missing header errors. The assistant symlinked more and more files, creating a fragile web of cross-version dependencies.

The User's Intervention

The user's response in [msg 11457] was sharp and direct: "what is this, why do you expect completely random symlinking to work? How about we install correct versions of software cleanly that can reasonably be expected to work together?"

This is a pivotal moment. The user is not just rejecting a specific technical approach — they are enforcing a principle: infrastructure should be built on clean, compatible, verifiable foundations, not on ad-hoc patches. The user's framing — "correct versions of software cleanly that can reasonably be expected to work together" — is a statement about engineering discipline. In production ML deployments, where the dependency graph spans CUDA toolkits, PyTorch builds, JIT compilation caches, and custom kernel libraries, the cost of fragility is enormous. A symlink that works today can break tomorrow when a library is upgraded, a cache is cleared, or a new kernel is compiled.

The assistant's reasoning in [msg 11458] shows it internalized this critique. It acknowledges: "The user is right - I was doing a terrible job trying to hack together incompatible CUDA toolkit versions with symlinks. That's fragile and unprofessional." This self-correction is important — it demonstrates the assistant recognizing that its earlier approach, while superficially solving the immediate error, was creating long-term maintenance risk.

The Message: Verification as Closure

The subject message ([msg 11461]) is a bash command that runs on the remote machine. It performs seven checks:

  1. nvcc version: Confirms the CUDA 13.0 compiler is installed and reports V13.0.88.
  2. curand headers: Checks that curand.h exists in the CUDA 13.0 include directory.
  3. curand library: Verifies the shared library libcurand.so and its versioned variants are present.
  4. cublas library: Checks for libcublas.so (version 13.1.1.3), confirming BLAS support.
  5. cusparse library: Verifies sparse matrix support.
  6. JIT cache cleanup: Removes flashinfer, tvm-ffi, and torchinductor caches to force a clean recompilation against the now-complete toolkit.
  7. Service configuration: Checks whether CUDA_HOME is set in the systemd service file. The output confirms success on every check. The toolkit is complete. The JIT caches are cleared. The service is ready to restart.

Why This Message Matters

On the surface, this is a mundane verification script. But in the narrative of the session, it represents a turning point. The assistant had been stuck in a loop of increasingly desperate workarounds — symlinking headers, clearing caches, restarting services, watching them fail, symlinking more headers. Each iteration addressed one error but revealed another, because the root cause (an incomplete CUDA toolkit) was never addressed. The user's intervention broke this cycle by insisting on the correct approach: install the full cuda-toolkit-13-0 package via apt.

The verification message is the moment where the assistant confirms that the root cause has been eliminated. It's not just checking "did the package install?" — it's checking the specific failure points that caused the cascade: curand.h (which triggered the original FlashInfer compilation failure), libcublas and libcusparse (which would cause downstream failures in attention and MoE kernels), and the JIT caches (which held stale compiled artifacts from the broken toolkit state).

Assumptions and Input Knowledge

To understand this message, the reader needs several pieces of context:

Output Knowledge and What Was Learned

This message produces several forms of knowledge:

  1. Confirmation of toolkit completeness: The output proves that cuda-toolkit-13-0 installed all required components. This is not trivial — the earlier partial installation showed that apt can install a "CUDA toolkit" that is missing critical pieces. The verification distinguishes between "the package installed" and "the toolkit is usable."
  2. Library version compatibility: The specific library versions are now known: libcurand.so.10.4.0.35, libcublas.so.13.1.1.3. These matter for reproducibility — if a future deployment on similar hardware encounters issues, these versions serve as a known-good reference.
  3. JIT cache state: The caches are empty, meaning the next service start will trigger a full recompilation against the complete toolkit. This is the correct state — any cached kernels from the broken toolkit would have been compiled with missing headers and might have subtle bugs.
  4. Service configuration gap: The grep CUDA_HOME command returns no output (the output section ends with the cusparse check, and no grep result is shown). This implicitly reveals that the service file does not set CUDA_HOME, which means the process inherits the system default — likely the symlinked /usr/local/cuda pointing to the now-complete CUDA 13.0.

The Thinking Process: What the Reasoning Reveals

The assistant's reasoning in the preceding message ([msg 11458]) reveals a thoughtful diagnostic process. It inventories the current state: torch 2.11.0+cu130, SGLang 0.4.3.post2, FlashInfer 0.6.8.post1. It identifies the core mismatch: "The CUDA 13.0 toolkit is incomplete - it has nvcc and cudart but missing libraries like curand, cusparse, cublas etc." It then enumerates three clean options: install the complete CUDA 13.0 toolkit, find a newer FlashInfer with pre-compiled SM120 kernels, or upgrade to a SGLang nightly build.

The decision to install the full toolkit is the correct one because it addresses the root cause rather than working around it. The assistant checks what packages are available (apt-cache search 'cuda.*13-0'), finds cuda-toolkit-13-0 as a meta-package, and installs it. This is the cleanest path — it ensures all CUDA libraries are present and version-matched to the compiler.

Mistakes and Lessons

The primary mistake in this episode was the initial symlinking approach. It's worth examining why the assistant chose this path. The reasoning likely went: "The error is curand.h: No such file or directory. The header exists in CUDA 12.8. A symlink will make it visible to the compiler. Problem solved." This is a natural debugging reflex — address the immediate error message. But it fails to account for:

Conclusion

This message — a simple verification script — represents the moment when a flailing debugging session was pulled back onto solid ground. The assistant had been trapped in a cycle of increasingly creative workarounds, each one revealing a deeper layer of the underlying toolkit incompatibility. The user's insistence on clean installation broke that cycle, and this verification message confirmed that the correct fix had been applied.

In the broader narrative of the coding session, this moment is the foundation for everything that follows. With a complete, clean CUDA 13.0 toolkit, FlashInfer can compile its SM120 kernels, SGLang can start, and the K2.6 model can be deployed and benchmarked. The verification message is the pivot point — the moment when the infrastructure stops fighting back and starts working.