Patching the Unpatchable: Forging Blackwell FP4 Support in sgl-kernel

In the sprawling, multi-day effort to deploy Qwen3.5-397B-A17B-NVFP4 on an 8× RTX PRO 6000 Blackwell (SM120) machine, there arrives a moment that crystallizes the entire endeavor: a single assistant message that SCPs a Python script to a remote server and runs it. The message itself is deceptively short — a bash command followed by a confirmation output — but it represents the culmination of dozens of preceding messages of reconnaissance, analysis, and careful planning. This is message [msg 5894], and it is the hinge upon which the entire deployment turns.

The Message

The assistant writes:

[bash] scp /home/theuser/glm-kimi-sm120-rtx6000bw/patch_sgl_kernel.py root@10.1.230.174:/root/patch_sgl_kernel.py && ssh root@10.1.230.174 '~/ml-env/bin/python3 /root/patch_sgl_kernel.py'
  Applied: cmake_policy guards
  Applied: cccl include directory
  Applied: FA3 enable guard (respects -DSGL_KERNEL_ENABLE_FA3=OFF)
Wrote: /root/sglang-main/sgl-kernel/CMakeLists.txt
  Applied: FA3 import fallback (soft fail)
Wrote: /root/sglang-main/sgl-kernel/python/sgl_kernel/flash_attn.py

All patches applied successfully.

Four patches, two files modified, one goal: make sgl-kernel compile on CUDA 13 with Blackwell SM120 architecture support. The message is the execution of a plan that was carefully constructed across the preceding messages, and it marks the transition from investigation to action.

Why This Message Was Written: The Reasoning and Motivation

The user had given a directive: "update all to nightly." This meant upgrading PyTorch, flashinfer, SGLang, and all supporting libraries to their latest nightly builds to unlock Blackwell-native features — particularly FP4 (4-bit floating point) support, which is essential for running the Qwen3.5-397B-A17B-NVFP4 model efficiently on the RTX PRO 6000 Blackwell GPUs.

By [msg 5888], the assistant had already upgraded PyTorch to 2.12.0.dev20260307+cu130, flashinfer to 0.6.5, and pulled the latest SGLang main branch. The next item on the todo list was: "Apply catid's CMakeLists.txt patches to sgl-kernel." This was the critical path. Without a properly patched sgl-kernel, the FP4 kernels required for the NVFP4 model would not compile, and the entire deployment would stall.

The motivation was straightforward but urgent: the upstream sgl-kernel did not support Blackwell (SM120) GPUs. The CMake configuration was written for SM90 (Hopper) and SM100 (future architectures), and the CUDA 13 toolkit introduced breaking changes to the build system — new CMake policies, a reorganized cccl include directory, and stricter handling of Flash Attention 3 (FA3) dependencies. The assistant had to bridge this gap by applying community-sourced patches (from "catid," a developer who had apparently worked on similar issues) and adapting them to the current state of the upstream code.

How Decisions Were Made

The decision-making process visible in the preceding messages reveals a methodical, reconnaissance-first approach. Before writing the patch script, the assistant spent several messages examining the current state of sgl-kernel/CMakeLists.txt:

• In [msg 5888], it read the first 80 lines of the CMakeLists and searched for key patterns (CMP0169, CMP0177, cccl, ENABLE_FA3, FP4, ENABLE_BELOW_SM90).
• In [msg 5889], it examined the include_directories lines and the FetchContent declarations for flash-attention and mscclpp.
• In [msg 5890], it read the CUDA flags block to understand what architectures were being compiled for.
• In [msg 5891], it examined the version-dependent logic for FA3 and FP4 enablement.
• In [msg 5892], it checked the Python-side FA3 import to understand the error handling. This was not a blind application of patches. The assistant was verifying that the upstream code had changed since catid's gist was written, and that the patches would need adjustment. It explicitly noted: "I see — the upstream CMakeLists.txt has been updated since catid's gist. The include_directories block is now simpler." The decision to write the patch as a Python script (rather than applying raw sed commands or inline patches) was a deliberate choice to avoid shell escaping issues with zsh. The assistant stated this explicitly: "Let me now write a Python patch script and SCP it to the container (to avoid zsh escaping issues)." This is a practical engineering decision — when dealing with complex string manipulations involving CMake syntax, quotes, and special characters, a Python script provides much more reliable control than shell commands piped through SSH.

The Patches: What They Do and Why

The four patches applied by the script address distinct but interrelated issues:

1. cmake_policy guards: The upstream CMakeLists.txt had unconditional cmake_policy(SET CMP0169 OLD) and cmake_policy(SET CMP0177 NEW) on lines 9-10. These CMake policy settings control how CMake handles certain behaviors (CMP0169 relates to FetchContent and CMP0177 to FindPackage). On older versions of CMake (below 3.26, which is the minimum required), these policies may not exist, causing a fatal error. The patch wraps them in conditional guards so they only apply when the CMake version supports them. This is essential for portability across different build environments.

2. cccl include directory: CUDA 13 reorganized the NVIDIA CUDA Compiler Collection (cccl) headers. The cccl library (which includes CUB, Thrust, and libcudacxx) moved to a new location, and the build system needs to know where to find these headers. Without this include path, compilation of CUDA kernels that depend on cccl features (like the FP4 kernels) would fail with missing header errors.

3. FA3 enable guard: The upstream CMakeLists.txt unconditionally enabled FA3 (Flash Attention 3) when CUDA_VERSION >= 12.4 (line 246: set(SGL_KERNEL_ENABLE_FA3 ON)). However, FA3 has dependencies that may not be available or may not compile correctly on SM120. The patch makes this respect the -DSGL_KERNEL_ENABLE_FA3=OFF flag, allowing the builder to explicitly disable FA3 even on CUDA 12.4+. This is critical because FA3 compilation was known to fail on Blackwell — the assistant had discovered this in earlier testing.

4. FA3 import fallback (soft fail): On the Python side, the flash_attn.py module had a hard raise ImportError if the FA3 flash_ops could not be imported. This meant that if FA3 was disabled at compile time, the entire sgl_kernel Python package would fail to load. The patch changed this to a soft fail — a warning or graceful degradation — so that the rest of the kernel library remains usable even without FA3. This is a pragmatic design decision: better to have a working system with reduced functionality than a broken system that refuses to start.

Assumptions Made by the Assistant

Several assumptions underpin this message:

Assumption 1: The patches from catid's gist are correct and sufficient. The assistant is applying patches sourced from a community developer without independently verifying every line. This is a reasonable trust decision — catid appears to be a knowledgeable contributor to the SGLang ecosystem — but it carries risk. If the patches had subtle errors (e.g., wrong include paths for the specific CUDA 13 build on the system), the build would fail, and the assistant would need to debug.

Assumption 2: The upstream CMakeLists.txt has not diverged so much that the patches are inapplicable. The assistant checked this explicitly and noted that the include_directories block had changed, but it proceeded anyway. This was validated by the successful application message.

Assumption 3: The Python script approach will work reliably. The assistant assumed that writing a Python script locally, SCPing it, and running it on the remote server would be more reliable than inline shell commands. This was correct — the output shows clean application.

Assumption 4: Building sgl-kernel from source is the correct path. The alternative would be to wait for official wheels or pre-built binaries for SM120. But those did not exist yet (the nightly wheel index for torch2.12 was a 404 page, as discovered in [msg 5871]). Building from source was the only viable option.

Mistakes or Incorrect Assumptions

The message itself executed cleanly — all patches applied successfully — so there are no errors visible in this specific message. However, looking at the broader context, there is one notable assumption that could be considered optimistic:

The patches might not be sufficient for a successful build. Applying the patches is necessary but not sufficient. The actual build (which happens in subsequent messages) could still fail due to missing dependencies, incompatible CUDA kernel code, or other issues not addressed by these four patches. The assistant is about to embark on a multi-hour compilation of sgl-kernel with TORCH_CUDA_ARCH_LIST=12.0a, and there is no guarantee it will succeed. The message represents a bet — a well-informed bet, but a bet nonetheless.

Additionally, the assistant assumed that the FA3 soft-fall approach would not cause cascading failures elsewhere in the codebase. If other parts of SGLang unconditionally import FA3 functionality, the soft fail might merely defer the crash to a later, harder-to-diagnose point. The assistant would need to test this.

Input Knowledge Required

To understand this message, one needs:

• Knowledge of the SGLang architecture: That sgl-kernel is a separate Python package within the SGLang repository that provides CUDA kernel implementations for attention, MoE, and other operations. That it uses CMake for building and has a Python-side import structure.
• Knowledge of CUDA and GPU architectures: What SM120 (Blackwell) means, why it requires different compilation flags than SM90 (Hopper), and why FP4 kernels are architecture-specific.
• Knowledge of CMake build systems: What cmake_policy does, what include_directories means, how FetchContent works, and why conditional guards are needed for cross-version compatibility.
• Knowledge of the CUDA 13 toolchain changes: That CUDA 13 reorganized cccl headers, breaking builds that hardcode include paths for earlier CUDA versions.
• Knowledge of Flash Attention 3: What FA3 is, why it might fail on Blackwell, and why a soft-fall import pattern is preferable to a hard error.
• Knowledge of the deployment context: That this machine has 8× RTX PRO 6000 Blackwell GPUs, that the target model is Qwen3.5-397B-A17B-NVFP4, and that FP4 support is essential for running it efficiently.

Output Knowledge Created

This message produces:

• A patched CMakeLists.txt at /root/sglang-main/sgl-kernel/CMakeLists.txt with conditional cmake_policy guards, cccl include directory, and FA3 enable guard.
• A patched flash_attn.py at /root/sglang-main/sgl-kernel/python/sgl_kernel/flash_attn.py with soft-fall FA3 import.
• A validated patch application — the script ran without errors and confirmed all four patches were applied.
• A foundation for the next step: Building sgl-kernel from source with TORCH_CUDA_ARCH_LIST=12.0a, which will produce the FP4 kernels needed for the model. More broadly, the message creates knowledge about the process of porting sgl-kernel to Blackwell. The patches themselves, and the reasoning behind them, constitute a form of documentation — a record of what needed to change to make the build work. This is valuable for anyone else attempting to deploy SGLang on SM120 hardware.

The Thinking Process Visible in the Reasoning

The assistant's reasoning is visible in the sequence of reconnaissance messages leading up to this one. The pattern is:

1. Read the current state (cat the file, grep for key patterns)
2. Compare with expected state (what catid's patches assume vs. what's actually there)
3. Identify discrepancies ("the include_directories block is now simpler")
4. Design the patch (write a Python script that handles the specific differences)
5. Execute and verify (SCP, run, check output) This is textbook software engineering methodology: never apply a patch blindly; always understand what it does and whether the target has changed. The assistant's comment "I see — the upstream CMakeLists.txt has been updated since catid's gist" is the key insight that drives the entire approach. Rather than blindly applying catid's exact patch (which might fail or produce incorrect results), the assistant writes a custom script that adapts the patches to the current codebase. The choice of a Python script over shell commands is also revealing. The assistant anticipates that the patch logic might need to be more sophisticated than simple sed substitutions — it might need to check for the existence of certain patterns, conditionally modify different sections, or handle edge cases. A Python script provides this flexibility. It also allows for clean confirmation output ("Applied: cmake_policy guards"), which serves as verification that each step succeeded.

Conclusion

Message [msg 5894] is a masterclass in practical systems engineering. It is not glamorous — there are no benchmark numbers, no architectural diagrams, no clever algorithms. It is a simple act of patching two files. But it is the critical enabler for everything that follows: the FP4 kernels, the production deployment, the 2100+ tok/s throughput. Without these patches, the Blackwell GPUs would be running Hopper code paths, unable to leverage their native FP4 capabilities.

The message also illustrates a fundamental truth about deploying cutting-edge ML infrastructure: the official releases never quite support your hardware. The nightly builds are broken. The documentation is outdated. The community patches are close but not exact. Success comes from the willingness to read the source, understand the build system, and write the missing glue yourself. This message is that glue.