The Clean Slate: Installing the Full CUDA 13.0 Toolkit After a Cascade of Hacky Fixes

The Message

[assistant] [bash] ssh -o ConnectTimeout=5 root@10.1.2.200 "
# Install the full CUDA 13.0 toolkit cleanly
apt-get install -y cuda-toolkit-13-0 2>&1 | tail -15
" 2>&1
Adding debian:emSign_ECC_Root_CA_-_C3.pem
Adding debian:emSign_ECC_Root_CA_-_G3.pem
Adding debian:emSign_Root_CA_-_C1.pem
Adding debian:emSign_Root_CA_-_G1.pem
Adding debian:vTrus_ECC_Root_CA.pem
Adding debian:vTrus_Root_CA.pem
done.
Setting up openjdk-21-jre:amd64 (21.0.10+7-1~24.04) ...
Setting up default-jre-headless (2:1.21-75+exp1) ...
Setting up default-jre (2:1.21-75+exp1) ...
Setting up cuda-nsight-13-0 (13.0.85-1) ...
Setting up cuda-visual-tools-13-0 (13.0.3-1) ...
Setting ...

At first glance, this message looks like a routine package installation—a single apt-get install command executed over SSH. But in the context of the conversation, it represents a critical inflection point: the moment the assistant abandoned a series of increasingly desperate and fragile workarounds and instead performed the correct, clean installation that should have been done from the start. This message is the pivot from hacking to engineering.

The Context: A Cascade of Failures

To understand why this message was written, we must trace the chain of events that led to it. The assistant was in the middle of deploying and benchmarking the Kimi K2.6 model with SGLang on a remote machine (CT200) equipped with 8× RTX PRO 6000 Blackwell GPUs (compute capability SM120). The SGLang service had been running, but after a crash (likely related to an earlier EAGLE-3 deployment that ran out of memory), the FlashInfer JIT compilation cache was invalidated. When the service tried to restart, FlashInfer needed to recompile its sampling kernels for SM120—and it failed.

The root cause was a mismatch between the PyTorch installation and the system's CUDA toolkit. The virtual environment had torch 2.11.0+cu130, which was compiled against CUDA 13.0. But the system's CUDA 13.0 toolkit was incomplete—it had nvcc and cudart but was missing critical libraries like curand, cusparse, and cublas. The system's full CUDA toolkit was version 12.8. FlashInfer's JIT compilation, which requires curand.h and other headers, was pointing at the incomplete CUDA 13.0 installation and failing with fatal error: curand.h: No such file or directory.

The assistant's first response was to try installing the missing package directly: apt-get install -y cuda-curand-dev-13-0. This failed—the package didn't exist in the repository. So the assistant pivoted to a workaround: installing libcurand-dev-12-8 (the CUDA 12.8 version) and symlinking all 17 curand*.h headers from the 12.8 include directory into the 13.0 include directory. This got past the first error but immediately hit the next: curand_mrg32k3a.h was missing. The assistant then symlinked all curand headers from 12.8 to 13.0 and restarted the service.

The service appeared to load—after 660 seconds of waiting, the health check passed. But when the assistant tried to run benchmarks, the service had already crashed again. The symlink approach was a house of cards: it might fix one missing header, but the underlying toolkit was fundamentally incomplete. Each restart triggered a new JIT compilation that hit a different missing dependency.

The User's Intervention

At this point, the user intervened with a sharp rebuke ([msg 11457]):

"what is this, why do you expect completely random symlinking to work? How about we install correct versions of software cleanly that can reasonably be expected to work together? Nightly etc, that work on this quite recent blackwell sm_120"

The user's frustration is understandable. The assistant had been performing ad-hoc surgery on a complex software stack, symlinking headers across CUDA toolkit versions as if the two toolkits were interchangeable. This approach was not only fragile but unprofessional—it created a system state that was impossible to reason about, debug, or reproduce.

The Reasoning Behind Message 11460

The assistant's thinking process, visible in the agent reasoning blocks of the preceding messages, reveals a genuine course correction. In [msg 11458], the assistant wrote:

"The user is right - I was doing a terrible job trying to hack together incompatible CUDA toolkit versions with symlinks. That's fragile and unprofessional."

The assistant then performed an inventory of the current state, discovering:

Assumptions and Mistakes

Several assumptions and mistakes are visible in this sequence:

The assistant's mistaken assumption was that the CUDA 13.0 toolkit was "mostly complete" and only needed a few missing headers patched in. This assumption was wrong—the toolkit was fundamentally incomplete, installed only with minimal build components. The symlinking approach treated the symptom (missing curand.h) rather than the disease (incomplete toolkit installation).

The assistant's correct assumption in message 11460 was that NVIDIA's package repository would have a complete cuda-toolkit-13-0 meta-package that pulls in all necessary development libraries. This assumption was validated by the apt-cache search in the previous message, which showed cuda-toolkit-13-0 exists.

A subtle mistake was the assistant's earlier failure to check for the meta-package. When the initial cuda-curand-dev-13-0 package wasn't found, the assistant immediately fell back to the symlink approach instead of checking whether a broader meta-package like cuda-toolkit-13-0 existed. This is a classic debugging pitfall: fixing the first error encountered rather than stepping back to understand the root cause.

The user's implicit assumption was that the assistant would know to install the full toolkit rather than patch individual components. This is a reasonable expectation for someone familiar with CUDA toolchains—the toolkit is designed to be installed as a coherent whole, not assembled piecemeal.

Input Knowledge Required

To understand this message, the reader needs to know:

  1. CUDA toolkit structure: The CUDA toolkit is a collection of compilers, libraries, headers, and tools. A "partial" installation (e.g., cuda-minimal-build-13-0) includes only nvcc and cudart, while the full cuda-toolkit-13-0 includes everything needed for JIT compilation of CUDA kernels.
  2. FlashInfer's JIT compilation model: FlashInfer compiles CUDA kernels at runtime for the specific GPU architecture detected. For Blackwell (SM120), it needs a complete CUDA toolkit with headers like curand.h for sampling kernels.
  3. The relationship between PyTorch's CUDA version and the system toolkit: PyTorch ships with its own CUDA runtime libraries, but JIT compilation (used by FlashInfer, Triton, etc.) requires the system toolkit's nvcc and headers.
  4. NVIDIA's Debian packaging conventions: CUDA packages follow the pattern cuda-{component}-{version} (e.g., cuda-curand-dev-13-0), with meta-packages like cuda-toolkit-13-0 that pull in all components.

Output Knowledge Created

This message produces several important outcomes:

  1. A complete, coherent CUDA 13.0 toolkit is installed on the CT200 machine, matching the PyTorch installation's CUDA version. This eliminates the version mismatch that caused the JIT compilation failures.
  2. The symlink hack is rendered unnecessary. With the full toolkit installed, FlashInfer can find all required headers in their proper locations within the CUDA 13.0 include directory.
  3. A reproducible system state is established. Instead of a machine with an incomplete CUDA 13 toolkit plus symlinks to 12.8 headers—a configuration that would be impossible to document or replicate—the system now has a standard, complete CUDA 13.0 installation.
  4. The path is cleared for FlashInfer JIT compilation to succeed on the next service restart, which will unblock the benchmarking work that was the original goal.

The Deeper Lesson

This message is a small but telling example of a pattern that recurs throughout software engineering: the tension between the "quick fix" and the "correct fix." The assistant spent multiple rounds trying to patch the incomplete CUDA toolkit with symlinks, each fix revealing a new missing header. The user's intervention broke this cycle by demanding the correct approach.

The assistant's thinking process reveals the internal debate: "I have three options to fix this: install the complete CUDA 13.0 toolkit with all development packages, try a newer FlashInfer version that might have SM120 kernels pre-compiled, or upgrade to a recent SGLang nightly build." The assistant correctly identified the options but initially chose the wrong path—opting for the minimal intervention (symlinks) rather than the clean one (full toolkit install). The user's pushback redirected the assistant to the right choice.

In the broader narrative of the coding session, this message is the turning point that enables everything that follows. Without a working CUDA toolkit, the SGLang service cannot start, the benchmarks cannot run, and the entire deployment pipeline is blocked. By installing the full toolkit cleanly, the assistant unblocks the entire downstream workflow—parallelism benchmarking, DFlash speculative decoding deployment, and the comprehensive performance analysis that occupies the rest of the segment.