The Build That Unlocked Blackwell: Compiling sgl-kernel from Source for SM120 FP4 Support

In the sprawling, multi-week journey to deploy production-grade inference on an 8× RTX PRO 6000 Blackwell GPU cluster, few individual messages carry as much weight as the one at index 5908. It is deceptively brief — just two bash commands and their output — but it represents the culmination of a grueling, multi-step effort to compile sgl-kernel from source with Blackwell (SM120) support. Without this single successful build, the entire deployment of the Qwen3.5-397B-A17B-NVFP4 model would have been impossible.

The Context: Why This Build Was Necessary

The assistant was executing a user directive to "update all to nightly" — a sweeping upgrade of the entire ML inference stack to bleeding-edge versions. The goal was to deploy the Qwen3.5-397B-A17B-NVFP4 model, a massive 397-billion-parameter mixture-of-experts (MoE) model using NVIDIA's FP4 (4-bit floating point) quantization format. The target hardware was an 8-GPU machine equipped with RTX PRO 6000 Blackwell GPUs, which use NVIDIA's SM120 architecture (compute capability 12.0).

The problem was stark: the pre-built sgl-kernel wheel that ships with SGLang did not include kernels compiled for SM120, nor did it enable FP4 support. The Blackwell architecture is new enough that upstream support was still being developed. The assistant had already upgraded PyTorch to the nightly 2.12.0.dev20260307+cu130, flashinfer to 0.6.5, and pulled the latest SGLang main branch. But the kernel library — the low-level CUDA code that actually runs on the GPUs — needed to be built from source with specific flags to target Blackwell and enable FP4 matrix operations.

This message at index 5908 is the moment that effort succeeded.

What the Message Contains

The message reports two actions. First, the build command completed successfully, producing a wheel file named sgl_kernel-0.3.21-cp310-abi3-linux_x86_64.whl. Second, the assistant installs that wheel into the Python virtual environment using uv pip install with --force-reinstall and --no-deps flags.

The installation output reveals something important: it replaced a previous installation of sgl-kernel==0.3.21 that came from a file at /tmp/sgl_kernel-0.3.21+cu130-cp312-abi3-manylinux2014_x86_64.whl — likely a pre-built wheel that was installed earlier but lacked SM120 support. The new wheel, built from the assistant's patched source, now carries the architecture-specific compilation needed for Blackwell.

The Hidden Complexity Behind Two Lines

To understand why this message is significant, one must trace the path that led to it. The preceding messages in the conversation (indices 5888–5907) reveal a cascade of build failures and iterative fixes:

The CMake policy problem (msg 5888–5895): The upstream sgl-kernel/CMakeLists.txt used cmake_policy(SET CMP0169 OLD) and cmake_policy(SET CMP0177 NEW) unconditionally. However, CMP0177 was only introduced in CMake 3.30, and the system had CMake 3.28. The assistant applied catid's patches to wrap these in if(POLICY ...) guards, preventing a fatal error. This was a forward-compatibility fix — anticipating that different build environments might have different CMake versions.

The cccl include path (msg 5893–5894): CUDA 13 changed the location of the cccl headers (the CUDA C++ Core Libraries). The patches added an explicit include_directories entry pointing to the cccl path under CUDA 13's installation directory. Without this, the compiler would fail to find critical headers like <cuda/std/optional> used by cutlass and other CUDA template libraries.

The FA3 fallback (msg 5893–5894): FlashAttention-3 (FA3) is not yet compatible with SM120. The patches modified both the CMake configuration (to respect -DSGL_KERNEL_ENABLE_FA3=OFF) and the Python import code (to gracefully handle the missing FA3 module rather than raising a hard ImportError). This was a pragmatic decision: better to have a working kernel library without FA3 than to have the entire build fail.

The build isolation trap (msg 5902–5904): The assistant discovered that uv build was using the system Python interpreter (/usr/bin/python3) instead of the project's virtual environment (/root/ml-env). This meant that scikit-build-core and nanobind — build dependencies installed only in the venv — were invisible to the build process. The fix required passing --python /root/ml-env/bin/python3 to uv build.

The missing CUDACXX (msg 5905–5906): Even with CUDA_HOME set, CMake couldn't find the CUDA compiler. The assistant had to explicitly set CUDACXX=/usr/local/cuda-13.0/bin/nvcc and add /usr/local/cuda-13.0/bin to the PATH.

The dlpack cmake version issue (msg 5907): A transitive dependency (dlpack) used an outdated cmake_minimum_required that triggered a policy error with CMake 4.2.1. The fix was adding -DCMAKE_POLICY_VERSION_MINIMUM=3.5 to the CMake arguments, effectively telling CMake to be lenient about version requirements in fetched dependencies.

Each of these failures was diagnosed from error output, and each required specific domain knowledge to resolve. The assistant understood CMake policy mechanics, the CUDA toolchain's environment variable conventions, uv's build isolation behavior, and the internal structure of sgl-kernel's dependencies.

The Build Configuration: Deliberate Choices

The successful build used a carefully chosen set of environment variables and flags:

• TORCH_CUDA_ARCH_LIST="12.0a": This tells PyTorch's build system to generate code for the SM120 architecture (Blackwell). The a suffix indicates that the compiler should use the full architecture-specific tuning, not just a compatible subset. This is the critical flag that makes the kernels run on RTX PRO 6000 Blackwell GPUs.
• MAX_JOBS=20 and CMAKE_BUILD_PARALLEL_LEVEL=20: These limit parallel compilation to 20 jobs, preventing memory exhaustion. The machine had 128 CPU threads, and earlier flash-attn builds had failed spectacularly when using all of them (see segment 0's struggles with MAX_JOBS).
• -DSGL_KERNEL_ENABLE_FP4=ON: Explicitly enables FP4 kernel generation. Without this, the Qwen3.5 model's NVFP4 quantization format would have no compute kernels to run on.
• -DENABLE_BELOW_SM90=OFF: Disables compilation for older GPU architectures (SM90 and below), saving significant build time and binary size. Since the target hardware is exclusively Blackwell, this is a safe optimization.
• -DSGL_KERNEL_ENABLE_FA3=OFF: Disables FlashAttention-3, which is incompatible with SM120. This avoids compilation errors and produces a library that gracefully degrades to FA2 for attention operations. These choices reflect a deep understanding of the tradeoffs involved. The assistant could have enabled FA3 and hoped for the best, but that would have caused a build failure. It could have left ENABLE_BELOW_SM90=ON, but that would have wasted hours compiling kernels that would never be used. Every flag was a deliberate decision based on the known constraints of the Blackwell architecture.

Assumptions Made

The assistant operated under several assumptions, most of which proved correct:

1. That catid's patches were sufficient. The patches came from a community contributor (catid) who had apparently done similar work. The assistant trusted that these patches addressed the actual differences between upstream expectations and CUDA 13/SM120 reality.
2. That FP4 kernels compiled for SM120 would produce correct numerical results. This was a significant assumption — just because the code compiles doesn't mean it computes correctly. The assistant would later validate this through extensive backend testing (testing flashinfer_cutlass, flashinfer_cudnn, flashinfer_trtllm, and flashinfer_cutedsl backends), discovering that some backends produced NaN or garbage output on SM120.
3. That --no-deps was safe for the reinstallation. The assistant assumed that the existing dependency graph was correct and only the kernel library itself needed replacement. This was reasonable given that the build was specifically targeted at adding SM120 architecture code, not changing the library's API.
4. That the build environment was hermetic enough. The assistant assumed that setting environment variables explicitly (CUDA_HOME, CUDACXX, PATH, TORCH_CUDA_ARCH_LIST) would be sufficient to override any system-level defaults. This assumption was tested and validated through the iterative build attempts.

Mistakes and Incorrect Assumptions

Not every assumption held. The earlier build attempts revealed several incorrect assumptions:

• That uv build would automatically use the venv Python. This was wrong — uv build without explicit --python pointed to /usr/bin/python3, which lacked scikit-build-core. The fix required both --python and setting VIRTUAL_ENV and PATH.
• That CUDA_HOME alone would let CMake find nvcc. CMake actually looks for CUDACXX first, then falls back to CUDA_HOME. Setting only CUDA_HOME was insufficient.
• That fetched dependencies would have modern CMake policies. The dlpack dependency's cmake_minimum_required(VERSION 2.8) triggered CMake 4.x policy errors. The CMAKE_POLICY_VERSION_MINIMUM=3.5 workaround was not obvious and required understanding CMake's policy versioning mechanism. These mistakes are not failures — they are the normal process of building software on a bleeding-edge stack where documentation is sparse and the tools are evolving.

Input Knowledge Required

To understand and execute this build, the assistant needed:

1. CUDA architecture naming conventions: Knowing that Blackwell is SM120 (compute capability 12.0) and that the a suffix in 12.0a enables architecture-specific tuning.
2. CMake policy system: Understanding cmake_policy, CMP0169, CMP0177, and CMAKE_POLICY_VERSION_MINIMUM to resolve version incompatibilities.
3. uv build system internals: Knowing how uv build discovers Python interpreters, how --python overrides the default, and how --no-build-isolation affects dependency resolution.
4. sgl-kernel architecture: Understanding that the kernel library is structured with CMake, uses scikit-build-core for Python packaging, and has optional features (FP4, FA3, SM90A) controlled by CMake options.
5. The Blackwell compatibility landscape: Knowing which features (FA3) are unsupported on SM120 and which (FP4) are essential for the target model.

Output Knowledge Created

This message produced several valuable artifacts:

1. A custom-built sgl-kernel wheel with SM120 FP4 support, enabling the Qwen3.5-397B-A17B-NVFP4 model to run on Blackwell GPUs.
2. A validated build recipe (the exact set of environment variables and flags) that can be reused for future builds or for rebuilding after upstream changes.
3. Confirmation that the patching approach works — the CMake policy guards, cccl include paths, and FA3 fallback were all validated in a real build.
4. A baseline for further debugging — if the installed kernel library had produced incorrect results, the build recipe would have been the starting point for investigation.

The Thinking Process

The assistant's reasoning, visible across the preceding messages, follows a clear pattern: observe the error, understand its root cause, apply a targeted fix, retry. When uv build used the wrong Python, the assistant checked which Python was being used (which python3), verified that the venv had the required packages (~/ml-env/bin/python3 -c "import scikit_build_core"), and then adjusted the invocation. When CMake couldn't find nvcc, the assistant checked that nvcc existed (ls /usr/local/cuda-13.0/bin/nvcc), then set CUDACXX. Each failure was met with diagnosis, not guesswork.

The message at index 5908 is the reward for that methodical approach. It is a moment of quiet triumph in a conversation filled with error messages, build failures, and debugging sessions. The assistant does not celebrate — it simply reports the result and moves on to installation, then to testing, then to deployment. But for anyone reading the conversation, this is the inflection point where the Blackwell deployment became possible.

Conclusion

The build of sgl-kernel from source with SM120 FP4 support, reported in message 5908, is a masterclass in systems-level ML engineering. It demonstrates that deploying cutting-edge models on cutting-edge hardware requires not just knowing which flags to set, but understanding the entire toolchain — from CMake policies to CUDA compiler discovery to Python build isolation. The message itself is brief, but the knowledge and effort it represents are immense. It is the foundation upon which the entire Qwen3.5-397B-A17B-NVFP4 deployment was built.