The Pivotal Discovery: Driver Supports CUDA 13.1 While Toolkit Lags at 12.8

Introduction

In the course of a deeply technical optimization session for speculative decoding on an 8× RTX PRO 6000 Blackwell (SM120) system, a single bash command—message <msg id=5258>—served as the turning point that redefined the entire trajectory of the investigation. After systematically testing and eliminating six different allreduce optimization approaches (FlashInfer fusion, custom allreduce kernels, NCCL Tree algorithm, Torch symmetric memory, Expert Parallelism, and NCCL channel tuning), the assistant had hit dead end after dead end. The baseline throughput had been improved from 82 to 89.5 tok/s through the discovery of reducing --cuda-graph-max-bs, but EAGLE-3 speculative decoding remained stuck at 54.1 tok/s—well below the baseline. Every promising optimization path was blocked by the same fundamental issue: the SM120 (Blackwell) architecture was too new for the available software stack. The user then posed a strategic question in <msg id=5255>: "Should we update cuda to 13 with more proper support for sm120?" The assistant's response in <msg id=5256> acknowledged the potential and began preliminary checks. Then came the decisive message—a single bash invocation that would reveal a critical asymmetry between the driver and toolkit versions, and in doing so, chart a new course forward.

The Message: A Single Bash Command

The subject message consists of a single shell command executed on the remote server via SSH:

ssh root@10.1.230.174 'nvidia-smi | head -5; echo "---"; cat /usr/local/cuda/version.json 2>/dev/null | head -5 || echo "no version.json"'

The command does two things in sequence: it runs nvidia-smi to display the NVIDIA driver information (limited to the first 5 lines), and then it attempts to read the CUDA toolkit's version manifest from /usr/local/cuda/version.json. The || echo "no version.json" fallback ensures the command doesn't fail if the file doesn't exist. The output reveals a striking discrepancy:

Fri Feb 27 14:38:57 2026       
+-----------------------------------------------------------------------------------------+
| NVIDIA-SMI 565.57.01              Driver Version: 590.48.01      CUDA Version: 13.1     |
|-----------------------------------------+------------------------+----------------------+
| GPU  Name                 Persistence-M | Bus-Id          Disp.A | Volatile Uncorr. ECC |
---
{
   "cuda" : {
      "name" : "CUDA SDK",
      "version" : "12.8.1"
   },

The driver reports CUDA Version: 13.1, while the installed CUDA SDK toolkit is 12.8.1. This is the core discovery: the NVIDIA driver already supports CUDA 13.1 natively, but the system only has CUDA 12.8.1 toolkit installed. This means upgrading the CUDA toolkit to version 13 is feasible without any driver change—the driver is already ahead of the toolkit.

Why This Message Was Written

The assistant wrote this message as a direct response to the user's strategic question about upgrading CUDA. Before the assistant could recommend or plan an upgrade, it needed to answer several foundational questions:

  1. Does the current NVIDIA driver support CUDA 13? If the driver only supported up to CUDA 12, then upgrading the toolkit would be impossible without also upgrading the driver—a much riskier operation that could destabilize the running system.
  2. What version of the CUDA toolkit is actually installed? The previous message <msg id=5256> had revealed that nvcc was not in PATH (suggesting the CUDA toolkit might not be properly configured), and that directories /usr/local/cuda, /usr/local/cuda-12, and /usr/local/cuda-12.8 existed. But the exact version of the installed toolkit needed confirmation.
  3. Is there a version mismatch that could be exploited? If the driver supported a higher CUDA version than the toolkit, then upgrading only the toolkit would be a low-risk, high-reward operation. The message was written at a moment of strategic inflection. The assistant had just spent an entire session (segment 35) systematically testing and eliminating optimization approaches, updating the optimization plan document (eagle-fast-verify.md) with experimental results, and summarizing the situation for the user. When the user proposed the CUDA 13 upgrade, the assistant recognized this as potentially the most promising remaining avenue—but only if the driver cooperated. The message was the first step in validating that hypothesis.

The Reasoning and Decision-Making Process

The assistant's reasoning, visible in the preceding message <msg id=5256>, shows a clear chain of thought. The assistant immediately recognized the strategic importance of the user's suggestion: "CUDA 13 is the native toolkit for SM120 (Blackwell), while we're currently on CUDA 12.8 which only has early/partial SM120 support. This could unblock several dead ends—flashinfer fusion JIT, torch symmetric memory, and possibly MSCCL++." This analysis correctly identified that the root cause of all the dead ends was the SM120 architecture's newness and the lack of support in the CUDA 12.8 toolchain.

The assistant then proceeded methodically. First, in <msg id=5256>, it checked for nvcc (not found) and listed the installed CUDA directories, confirming the presence of CUDA 12.8. Next, in <msg id=5257>, it checked the PyTorch CUDA version (12.8) and GPU compute capability (12.0, confirming SM120). Then came the subject message <msg id=5258>, which checked the driver's CUDA version via nvidia-smi and the toolkit version via version.json.

This three-step investigation shows careful, systematic thinking. The assistant didn't jump to conclusions or immediately propose an upgrade plan. Instead, it gathered data from multiple sources—the driver, the toolkit filesystem, and the PyTorch runtime—to build a complete picture of the current CUDA stack before making any recommendations.

Assumptions Made

Several assumptions underpin this message:

  1. The driver version is authoritative for CUDA compatibility. The assistant assumed that nvidia-smi's reported "CUDA Version" accurately reflects the maximum CUDA runtime version the driver can support. This is a reasonable assumption—the NVIDIA driver's CUDA compatibility is a well-defined property.
  2. The toolkit version file is reliable. The assistant assumed that /usr/local/cuda/version.json accurately reflects the installed CUDA toolkit version. This is generally true for official CUDA toolkit installations.
  3. Upgrading only the toolkit is feasible. The implicit assumption is that if the driver supports CUDA 13.1, then installing a CUDA 13.x toolkit will work without driver changes. This is correct in principle—the CUDA toolkit is a user-space library that communicates with the driver through a stable API.
  4. The SM120 dead ends are caused by CUDA version, not hardware limitations. The assistant assumed that flashinfer fusion, torch symmetric memory, and other optimizations failed because of CUDA toolkit version, not because of fundamental hardware incompatibilities. This assumption would need validation after the upgrade.

Mistakes and Incorrect Assumptions

One potential issue with the message is that it only checks the version of the CUDA toolkit installed at /usr/local/cuda/, which is the symlink pointing to the default CUDA installation. However, the system also has /usr/local/cuda-12 and /usr/local/cuda-12.8 directories, suggesting multiple CUDA versions are present. The message doesn't verify which version the symlink points to—it reads version.json from /usr/local/cuda/, which could be the symlink target. In this case, the output confirms it points to the 12.8.1 installation, but the message doesn't explicitly resolve the symlink to confirm.

Additionally, the message doesn't check whether the CUDA 13.1 driver compatibility is sufficient for all the optimizations the assistant hopes to unblock. FlashInfer's JIT compiler, for example, might need specific CUDA toolkit features beyond what the driver provides. The message only establishes feasibility, not certainty.

Input Knowledge Required

To fully understand this message, the reader needs:

  1. Understanding of the CUDA software stack hierarchy: The distinction between the NVIDIA driver (which handles GPU hardware communication) and the CUDA toolkit (which provides development libraries, compiler, and runtime). The driver's "CUDA Version" indicates the maximum CUDA runtime version it supports, not the installed toolkit version.
  2. Knowledge of the Blackwell (SM120) architecture: SM120 is NVIDIA's newest GPU architecture (RTX PRO 6000 Blackwell). CUDA 13 is the first native toolkit for this architecture, while CUDA 12.8 has only partial/early support.
  3. Context from the preceding optimization session: The reader must understand that six different allreduce optimization approaches had failed, all due to SM120 incompatibility. The user's question about upgrading CUDA was a strategic pivot born from these failures.
  4. Familiarity with the system topology: The 8× RTX PRO 6000 GPUs are connected via PCIe Gen5 without NVLink, which creates unique communication bottlenecks that the optimizations were trying to address.
  5. Knowledge of the optimization targets: FlashInfer allreduce fusion, Torch symmetric memory, and MSCCL++ are all low-latency communication libraries that require specific CUDA architecture support.

Output Knowledge Created

This message creates several pieces of critical knowledge:

  1. The driver supports CUDA 13.1: The NVIDIA driver version 590.48.01 already has native CUDA 13.1 support. This means upgrading the toolkit is possible without a driver change—a low-risk operation.
  2. The toolkit is at 12.8.1: The installed CUDA SDK is version 12.8.1, lagging behind the driver's capability. This confirms a version mismatch that can be exploited.
  3. Upgrade feasibility is confirmed: The path to CUDA 13 is clear. The assistant can now proceed to plan the upgrade, knowing it won't require a driver change.
  4. The upgrade path is low-risk: Since the driver already supports CUDA 13.1, upgrading the toolkit from 12.8.1 to 13.x is a software-only change that doesn't risk GPU communication failures.
  5. A new direction for optimization: The dead ends from the previous session may be unblocked by the CUDA 13 upgrade. FlashInfer fusion JIT, torch symmetric memory, and other SM120-native optimizations could become available. This knowledge fundamentally changes the optimization strategy. Instead of continuing to work around CUDA 12.8's limitations, the team can now pursue a CUDA 13 upgrade that might unlock all the previously blocked optimization paths in one sweep.

The Thinking Process

The thinking process visible in this message and its immediate context shows a methodical, hypothesis-driven approach. The assistant didn't just blindly execute the user's suggestion. Instead, it:

  1. Validated the hypothesis: Before planning an upgrade, the assistant checked whether the driver supported CUDA 13. This is the first and most critical feasibility check.
  2. Triangulated from multiple sources: The assistant checked three independent sources of version information across three messages: the CUDA toolkit directories (<msg id=5256>), the PyTorch runtime (<msg id=5257>), and the NVIDIA driver (<msg id=5258>). Each source confirmed a different aspect of the CUDA stack.
  3. Recognized the strategic opportunity: The assistant immediately connected the CUDA 13 upgrade to the specific dead ends it had encountered, showing deep understanding of the dependency chain between CUDA version and optimization library support.
  4. Documented the finding: The output of the command is clean and unambiguous, making it easy to reference in future planning. The message is deceptively simple—a single bash command—but it represents the culmination of a long optimization journey. After hours of testing dead ends, this message finally found an open door. The driver was ready for CUDA 13; only the toolkit needed updating.

Conclusion

Message <msg id=5258> is a textbook example of a small, focused diagnostic command that produces outsized strategic value. In two lines of output, it revealed that the NVIDIA driver was already running CUDA 13.1 while the toolkit languished at 12.8.1—a version mismatch that turned a risky upgrade proposition into a straightforward software update. This discovery transformed the optimization landscape, offering a path to unblock every dead end that had stymied the previous session. The message demonstrates the power of precise, targeted investigation: sometimes the most valuable thing you can do is simply ask the right question of the right system component.