The Catch-22 of SM120: Debugging FlashInfer JIT Compilation on Desktop Blackwell

In the sprawling ecosystem of GPU-accelerated machine learning, few experiences are as humbling as encountering a CUDA version mismatch on a brand-new architecture. The message at <msg id=9513> captures a pivotal moment in an opencode coding session where an AI assistant, deep in the process of deploying a large language model inference server, confronts a fundamental incompatibility between NVIDIA's desktop Blackwell GPU (SM120) and the flashinfer attention kernel library. What unfolds is a masterclass in diagnostic reasoning—a careful untangling of PTX versions, pre-compiled cubins, include paths, and the uncomfortable truth that bleeding-edge hardware often outpaces its software ecosystem.

The Context: Batch Inference at Scale

To understand why this message matters, we must step back. The broader session (Segment 54 of the conversation) is about expanding training data for a speculative decoding model called DFlash. The assistant had set up SGLang—a high-throughput inference engine—on a machine with 8× RTX PRO 6000 Blackwell GPUs, intending to use them for batch inference to generate 193K diverse prompts. The model being served is Qwen3.6-27B, a 27-billion-parameter language model, and the goal is to run it efficiently using flashinfer, a library of highly optimized CUDA kernels for attention mechanisms.

The assistant had already navigated a maze of environment issues: installing SGLang 0.5.12, matching CUDA 13.2 nvcc with pip-installed CUDA headers, creating symlinks for libcudart and libcuda stubs, overlaying CCCL headers from flashinfer's bundled libcudacxx to resolve nv/target include errors, and switching to --attention-backend flashinfer after discovering that FA3/FA4 (Flash Attention 3 and 4) were unsupported on SM120. Each step was a battle. By the time we reach <msg id=9513>, the assistant has already cycled through multiple versions of nvcc (13.0, then 13.2, then back to 13.0) trying to get flashinfer's JIT compilation to work.

The Message: A Diagnostic Deep-Dive

The message opens with the assistant's reasoning block—a window into its analytical process. It begins by stating a critical finding:

The flashinfer cubins are all sm100f (for SM100 datacenter Blackwell). There are NO pre-compiled cubins for SM120 attention kernels. The SM120 files found are GEMM kernel templates (jinja files for JIT), not pre-compiled cubins.

This is the core discovery. Flashinfer ships with pre-compiled binary cubins (CUDA binary files) for SM90 (Hopper, the previous generation) and SM100 (datacenter Blackwell), but not for SM120 (desktop Blackwell). The SM120 files that do exist are Jinja templates—Python-based code generation templates used for just-in-time (JIT) compilation. This means flashinfer must JIT-compile attention kernels at runtime on SM120 hardware, and JIT compilation requires a perfectly matched CUDA compiler (nvcc) and CUDA toolkit headers.

The assistant then walks through a series of workarounds it considered:

  1. Disabling CUDA graphs (--disable-cuda-graph): This would sidestep the JIT path during graph capture, but the assistant correctly notes that flashinfer still needs JIT for attention kernels during inference regardless of CUDA graph usage. The graph capture merely triggers additional JIT compilation.
  2. Switching to the torch_sdpa backend: PyTorch's native scaled dot-product attention implementation would work universally without custom kernel compilation, but would be significantly slower—a critical concern when generating 523 million output tokens.
  3. Trying vLLM: Another inference engine with its own attention implementation that might handle SM120 better, though its availability and compatibility were uncertain.
  4. Installing the full CUDA 13.0 toolkit from NVIDIA's repository: This seemed like the cleanest solution—ensuring nvcc 13.0 matches the headers, PTX versions are compatible, and flashinfer can JIT compile properly. But the assistant catches itself mid-thought, recognizing a deeper problem:
But wait, there's a fundamental issue: the CCCL headers bundled with flashinfer are generating PTX 9.2, but ptxas 13.0 only supports up to PTX 9.0, so even a proper CUDA 13.0 installation won't resolve this mismatch.

This is the moment of genuine insight. PTX (Parallel Thread Execution) is NVIDIA's intermediate assembly language. Each CUDA version supports a specific PTX version: CUDA 13.0 supports PTX 9.0, while PTX 9.2 requires CUDA 13.2 or later. The flashinfer package bundles CCCL (CUDA C++ Core Libraries) headers that generate PTX 9.2, but the installed nvcc 13.0's ptxas assembler only understands up to PTX 9.0. The mismatch is baked into the package itself—flashinfer's bundled headers are from a newer CUDA toolkit than what the pip-installed CUDA runtime provides.

The Catch-22

The assistant articulates the dilemma with admirable clarity:

We're stuck in a catch-22 — upgrading to nvcc 13.2 would support PTX 9.2, but then the CCCL header version check fails because the system CUDA toolkit headers conflict with the bundled ones.

This is the crux of the problem. There are two possible paths, and both seem blocked:

The Reasoning Process: A Study in Iterative Debugging

What makes this message particularly instructive is the assistant's transparent reasoning process. It doesn't just state a conclusion—it walks through the logic, considers alternatives, rejects them with specific technical justifications, and arrives at a pragmatic decision.

The assistant examines the actual compile commands generated by flashinfer's build system:

Looking at the actual compile command, I can see flashinfer's CCCL paths are included first with -I, but then the nvidia/cu13 headers are also added with -isystem, which means the toolkit headers are still being picked up and causing the version check to fail.

This attention to the mechanics of the build system—how include paths are ordered, how -I vs -isystem affects header resolution—demonstrates a deep understanding of C/C++ compilation toolchains. The -isystem flag adds directories to the system include path, which has lower priority than explicit -I paths but still higher than default system paths. The fact that flashinfer's CCCL headers come first via -I but the nvidia/cu13 headers still interfere suggests that some headers are being pulled in transitively through other mechanisms.

The assistant also correctly identifies the specific header causing the problem:

The version mismatch is the core issue — flashinfer's bundled CCCL headers expect a specific CUDA version, but neither 13.0 nor 13.2 aligns properly.

This is a nuanced point. The CCCL headers bundled with flashinfer are from a specific CUDA version (likely 13.2, given the PTX 9.2 generation). Using nvcc 13.0 fails because the assembler is too old. Using nvcc 13.2 fails because the system headers (from the nvidia/cu13 pip package, version 13.0) are too old. The solution would be to use nvcc 13.2 and CUDA 13.2 headers simultaneously, but the pip-installed nvidia/cu13 package only provides CUDA 13.0 headers. The nvidia-cuda-crt package at version 13.2.78 exists but doesn't create a separate include directory—it may overwrite the cu13 package headers or install elsewhere.

The Pragmatic Decision

After this thorough analysis, the assistant makes a pragmatic choice:

The core issue: flashinfer has no pre-compiled SM120 attention cubins and JIT is broken due to CUDA version mismatches. Let me try the pragmatic route — install a matching CUDA 13.2 toolchain since flashinfer's bundled CCCL generates PTX 9.2.

The assistant decides to install nvcc 13.2.78 again, hoping that this time—with the cache cleared and perhaps with additional include path manipulation—the compilation will succeed. The bash command issued is:

ssh -o ConnectTimeout=10 root@10.1.2.6 "pct exec 200 -- bash -c 'kill -9 36372 2>/dev/null; rm -rf /root/.cache/flashinfer; export PATH=/root/.local/bin:\$PATH && uv pip install --python /root/venv/bin/python3 \"nvidia-cuda-nvcc==13.2.78\" 2>&1 | tail -3'" 2>&1

This command kills the previous SGLang server process (PID 36372), clears the flashinfer JIT cache (which contained partially compiled artifacts from the failed 13.0 attempt), and installs nvcc 13.2.78. The result is:

Installed 1 package in 5ms
 - nvidia-cuda-nvcc==13.0.88
 + nvidia-cuda-nvcc==13.2.78

Assumptions and Their Validity

The assistant makes several assumptions in this message, some explicit and some implicit:

  1. That flashinfer's JIT compilation is the only path to running attention kernels on SM120. This appears correct—the investigation confirmed no pre-compiled SM120 cubins exist in the flashinfer package.
  2. That the PTX version mismatch is the root cause of the 13.0 failure. This is well-supported by the error message: "ptxas ... fatal: Unsupported .version 9.2; current version is '9.0'."
  3. That installing nvcc 13.2 alone will resolve the issue. This assumption is more questionable. The assistant acknowledges that the header version check failed previously with nvcc 13.2, but seems to hope that clearing the cache and retrying will somehow bypass this. The reasoning suggests the assistant plans to manipulate include paths to use flashinfer's bundled CCCL headers instead of the system headers, but this plan isn't fully executed in the message—it's deferred to subsequent steps.
  4. That the nvidia/cu13 headers are version 13.0. This is confirmed in <msg id=9517> where CUDART_VERSION is found to be 13000 (i.e., CUDA 13.0).

Input Knowledge Required

To fully understand this message, one needs:

Output Knowledge Created

This message produces several valuable pieces of knowledge:

  1. A definitive diagnosis: Flashinfer has no pre-compiled SM120 attention cubins, and JIT compilation is blocked by CUDA version mismatches. This is a concrete finding that could inform decisions about alternative inference backends or waiting for flashinfer updates.
  2. A taxonomy of workarounds: The assistant enumerates and evaluates four alternatives (disable CUDA graphs, torch_sdpa, vLLM, full CUDA toolkit installation), providing technical justification for why each might or might not work.
  3. A clear articulation of the catch-22: The dual failure modes (PTX version too new for old nvcc, header version too old for new nvcc) are explained with specific version numbers and error messages.
  4. A pragmatic next step: Installing nvcc 13.2.78 and retrying, with the understanding that include path manipulation may be necessary.

What Follows

The subsequent messages (starting at <msg id=9514>) show the assistant continuing to debug the include path issue. It discovers that nvidia/cu13/include contains CUDA 13.0 headers that conflict with nvcc 13.2, and that nvidia-cuda-crt at version 13.2.78 doesn't create a separate include directory. The assistant eventually finds a solution by overlaying CCCL headers from flashinfer's bundled libcudacxx and using --attention-backend flashinfer with carefully constructed include paths—a testament to the persistence required when deploying on cutting-edge hardware.

Conclusion

The message at <msg id=9513> is a remarkable example of structured diagnostic reasoning in the face of a genuinely difficult toolchain problem. The assistant doesn't guess or flail—it systematically examines the evidence, formulates hypotheses, tests them against known constraints, and arrives at a coherent model of the failure. The catch-22 it identifies is real: the software ecosystem simply hasn't caught up to the hardware. Flashinfer's bundled CCCL headers are from CUDA 13.2, but the pip-installed CUDA runtime is 13.0, and neither version alone provides a consistent toolchain for SM120 JIT compilation.

What makes this message particularly valuable for readers is the transparency of the reasoning process. We see the assistant considering alternatives, rejecting them with specific technical objections, catching itself mid-thought when it realizes a proposed solution has a hidden flaw, and ultimately making a pragmatic decision based on incomplete information. This is not a textbook solution—it's a real debugging session where the answer isn't known in advance and each step requires careful analysis.

The broader lesson is one that anyone who works with GPU computing will recognize: bleeding-edge hardware is always ahead of the software stack, and getting a new architecture to work often requires navigating a minefield of version mismatches, missing pre-compiled binaries, and incomplete documentation. The assistant's methodical approach—gather evidence, formulate hypotheses, test systematically, and iterate—is the only reliable way through.