Building sgl-kernel for Blackwell: The Critical Compilation Step

In the long arc of deploying a 397-billion-parameter language model across eight NVIDIA RTX PRO 6000 Blackwell GPUs, few steps are as technically dense and consequential as the one captured in message 5899. This single message — a bash command issued over SSH to a remote build server — represents the culmination of hours of dependency wrangling, source patching, and architectural reasoning. It is the moment where theory meets practice: all the patches, all the environment upgrades, all the carefully researched configuration choices must now survive contact with the compiler.

The command is deceptively simple on its face. The assistant runs uv build --wheel on the sgl-kernel package, passing a handful of environment variables and build flags. But every variable in that command encodes a hard-won lesson from the preceding session, and every flag represents a deliberate trade-off between compatibility, performance, and correctness. To understand this message is to understand the entire Blackwell deployment challenge.

The Motivation: Why Build from Source?

The immediate trigger for this message is the user's directive to "update all to nightly" — a command that set off a chain of upgrades across the entire machine learning stack. The assistant had already upgraded PyTorch to the nightly 2.12.0.dev20260307+cu130, flashinfer to 0.6.5, and pulled the latest SGLang main branch. But one critical piece remained: the sgl-kernel package, which provides the low-level CUDA kernels for FP4 matrix multiplication, attention, and other operations that the Qwen3.5-397B-A17B-NVFP4 model depends on.

The problem is that pre-built sgl-kernel wheels do not include SM120 (Blackwell) architecture support. The Blackwell GPU, with its compute capability 12.0, is still new enough that mainstream package distributions have not caught up. The only way to get FP4 kernels running on these RTX PRO 6000 GPUs is to compile from source, targeting TORCH_CUDA_ARCH_LIST="12.0a" — the a suffix indicating that tensor core MMA (matrix multiply-accumulate) instructions should be enabled.

But building from source is not merely a matter of running pip install with a source tarball. The sgl-kernel package is a complex C++/CUDA project that pulls in multiple external dependencies — CUTLASS, FlashAttention, deep_gemm, and others — via CMake's FetchContent. It must be compiled against the correct CUDA toolkit (13.0 in this case), the correct PyTorch installation, and with the right set of feature flags. Getting any of these wrong means either a compilation failure or, worse, a silently incorrect binary that produces garbage output at inference time.

The Build Command: A Study in Deliberate Configuration

Let us examine the command in detail. The assistant sets eight environment variables before invoking the build:

CUDA_HOME=/usr/local/cuda-13.0
TORCH_CUDA_ARCH_LIST="12.0a"
MAX_JOBS=20
CMAKE_BUILD_PARALLEL_LEVEL=20
CMAKE_ARGS="-DSGL_KERNEL_COMPILE_THREADS=16 -DENABLE_BELOW_SM90=OFF -DSGL_KERNEL_ENABLE_FP4=ON -DSGL_KERNEL_ENABLE_FA3=OFF"
PATH="/root/.local/bin:$PATH"

Each of these variables reflects a conscious decision rooted in the specific hardware and software environment.

CUDA_HOME=/usr/local/cuda-13.0 points to the freshly installed CUDA 13.0 toolkit. This was not a trivial upgrade — earlier in the session, the assistant had upgraded from CUDA 12.8 to 13.0 specifically to unlock Blackwell-native optimizations like FlashInfer allreduce fusion and Torch symmetric memory. Using CUDA 13.0 is essential because it includes the compiler support for compute_120a architecture targets.

TORCH_CUDA_ARCH_LIST="12.0a" is the most critical flag. It tells the CUDA compiler to generate code for the Blackwell architecture (sm_120) with tensor core acceleration (the a suffix). Notably, the assistant does not include any older architectures like sm_90 (Hopper) or sm_80 (Ampere). This is deliberate: the -DENABLE_BELOW_SM90=OFF flag in CMAKE_ARGS explicitly disables code generation for pre-Hopper GPUs. On an 8-GPU machine with only Blackwell cards, there is no reason to waste compilation time or binary size on architectures that will never be used.

MAX_JOBS=20 and CMAKE_BUILD_PARALLEL_LEVEL=20 limit parallel compilation to 20 concurrent processes. This is a lesson learned from painful experience earlier in the session, where building flash-attn with 128 parallel jobs exhausted system memory and caused the compiler to be killed by the OOM killer. The machine has substantial RAM, but CUDA kernel compilation is memory-intensive — each nvcc invocation can consume several gigabytes. Limiting to 20 jobs is a conservative choice that prioritizes build stability over speed.

-DSGL_KERNEL_ENABLE_FP4=ON enables the FP4 (4-bit floating point) kernels that the Qwen3.5-397B-A17B-NVFP4 model requires. The "NVFP4" in the model name indicates NVIDIA's 4-bit floating point format, and without these kernels, the model cannot run. This is the entire point of the build.

-DSGL_KERNEL_ENABLE_FA3=OFF disables FlashAttention 3 (FA3) kernels. This is another hard-won insight: FA3 does not support SM120 architecture. If left enabled, the build would either fail or produce kernels that crash at runtime. The assistant had previously patched the FA3 import in flash_attn.py to return None gracefully rather than raising an ImportError, so disabling FA3 at the CMake level is the clean approach.

The Patches That Made This Build Possible

The command in message 5899 would have failed outright without the source code modifications applied in the preceding messages. The assistant had written a Python patch script (patch_sgl_kernel.py) that made three critical changes to sgl-kernel/CMakeLists.txt:

  1. CMake policy guards: The upstream CMakeLists.txt unconditionally set policies CMP0169 and CMP0177, but CMP0177 is only recognized by CMake 3.30 and later. The build server had CMake 3.28.3, which would error out on the unknown policy. The patch wrapped these in if(POLICY ...) guards, making them conditional.
  2. cccl include directory: CUDA 13.0 ships with a newer version of the CUDA C++ Core Libraries (cccl) that requires an explicit include path for certain headers. The patch added this include directory to the CMake configuration.
  3. FA3 fallback: The Python-side import of flash_ops was changed from a hard ImportError to a graceful None return, allowing SGLang to detect the absence of FA3 and fall back to alternative implementations. These patches represent the kind of bleeding-edge integration work that is invisible in a mature ecosystem but absolutely essential when deploying on hardware that is ahead of the mainstream package distribution curve.

What the Message Does Not Show: The Failures That Follow

The subject message captures only the first attempt at building sgl-kernel. The output shown is truncated — tail -80 — and reveals only the initial debug logs from uv. The build does not succeed on this attempt. In the messages that follow ([msg 5903], [msg 5905], [msg 5907]), the assistant discovers and fixes two additional issues:

First, uv build --no-build-isolation uses the system Python interpreter (/usr/bin/python3) rather than the project's virtual environment at /root/ml-env. The system Python lacks scikit-build-core, causing an immediate failure. The assistant works around this by adding --python /root/ml-env/bin/python3 to the uv build invocation.

Second, CMake cannot find the CUDA compiler even though CUDA_HOME is set. The assistant discovers that CUDACXX must be explicitly set to the nvcc binary path. This is a quirk of how different build systems discover CUDA — some honor CUDA_HOME, others require the more specific CUDACXX variable.

Third, the build fails on a dependency called dlpack whose CMakeLists.txt uses an older cmake_minimum_required that triggers a policy error with CMake 4.2.1. The assistant adds -DCMAKE_POLICY_VERSION_MINIMUM=3.5 to the CMake arguments, which suppresses the policy compatibility checks for fetched dependencies.

Each of these failures is a learning opportunity, and the assistant's ability to diagnose and fix them in rapid succession demonstrates deep familiarity with the CUDA/C++ build ecosystem.

The Broader Significance

Message 5899 is, in many ways, the turning point of the entire deployment effort. Before this build, the team had a patched source tree and upgraded dependencies but no way to actually run the FP4 model on Blackwell hardware. After this build (and its subsequent fixes), the sgl_kernel wheel is installed with SM120 FP4 kernels, and the assistant can proceed to the critical phase of backend testing — determining which of SGLang's many MoE and FP4 GEMM backends actually produce correct numerical output on the Blackwell GPUs.

The build also establishes a reproducible recipe. The exact set of environment variables, CMake flags, and workarounds documented across messages 5899–5907 can be reused for future Blackwell deployments. This is the kind of institutional knowledge that separates a one-off hack from a maintainable production setup.

Input Knowledge Required

To fully understand this message, one must be familiar with several domains:

Output Knowledge Created

The message produces several forms of output knowledge:

  1. A build command template: The exact invocation serves as a reference for anyone needing to build sgl-kernel for Blackwell GPUs.
  2. A set of environment variable requirements: The discovery that CUDACXX is needed in addition to CUDA_HOME, and that VIRTUAL_ENV must be set for uv build to use the correct Python.
  3. Validation of the patching strategy: The patches applied in preceding messages survive the build process, confirming that the CMake policy guards and cccl include paths are correct.
  4. A baseline for debugging: When the build fails (as it does in subsequent messages), the assistant has a known starting point for diagnosis.

Conclusion

Message 5899 is a masterclass in practical ML infrastructure engineering. It is not glamorous — there are no model architecture diagrams, no novel algorithms, no benchmark charts. But it is the kind of work that makes all of those things possible. Every environment variable, every CMake flag, every patch file represents hours of debugging, reading source code, and understanding failure modes. The Blackwell GPU is a powerful piece of hardware, but it is only as useful as the software stack that supports it, and that software stack must be painstakingly assembled one build flag at a time.