The Pivot to SGLang: Investigating SM120 Kernel Compatibility for Blackwell GPUs

Introduction

In the sprawling, months-long effort to deploy the 1-trillion-parameter Kimi-K2.5 INT4 model across eight NVIDIA RTX PRO 6000 Blackwell GPUs, few moments are as quietly decisive as message <msg id=3101>. This message, appearing deep in segment 23 of the conversation, marks the precise inflection point where the assistant pivots from a failed speculative decoding strategy in vLLM toward the SGLang inference engine—only to immediately encounter a new class of problem: building custom CUDA kernels for NVIDIA's bleeding-edge SM120 architecture (compute capability 12.0, Blackwell).

The message itself is deceptively brief. It contains a single bash command that probes the filesystem of a remote server, listing the contents of an SGLang kernel source directory and checking for a setup.py build file. But this simple reconnaissance operation carries the weight of an entire project's trajectory. To understand why, we must trace the events that led to this moment.

The Road to SGLang

The assistant had just completed one of the most sophisticated ML engineering pipelines in the entire conversation: a full EAGLE-3 speculative decoding training pipeline for Kimi-K2.5. This involved generating 10,000 synthetic reasoning samples (5.3 hours), extracting hidden states at 3,165 tok/s (producing 828 GB of training data), and finetuning a draft model for 5 epochs (2.6 hours). The pipeline was a technical triumph—every stage completed without error.

Yet when the trained drafter was loaded into vLLM, the results were devastating. The acceptance rate—the fraction of draft tokens accepted by the target model—was only ~15%, compared to the 60–80% that EAGLE-3 typically achieves. Throughput dropped to 0.66× of baseline, meaning speculative decoding was actively hurting performance. Crucially, the assistant confirmed this wasn't a training quality issue: even the pre-trained AQ-MedAI baseline drafter achieved the same 15% acceptance rate on vLLM. The root cause was a fundamental integration bug in vLLM's handling of Multi-Head Latent Attention (MLA) hidden state extraction during EAGLE-3 decode for DeepSeek V3 architecture models like Kimi-K2.5.

The user's directive was clear: "Get the models up on SGLang, if acceptance rate seems low try to debug things like data and predictions" ([msg 3093]). SGLang had first-class EAGLE-3 support explicitly tested with Kimi-K2 drafters, and the AQ-MedAI checkpoint had been designed and benchmarked for SGLang (reportedly achieving 1.8× speedup). The pivot was logical, but it opened a new front of uncertainty: would SGLang even run on SM120?

The Subject Message: A Filesystem Reconnaissance

Let us examine the message in full:

Message 3101 (assistant): There's a recent SGLang checkout. The sgl_kernel is the problem — it's compiled for SM100 not SM120. Let me check if we can build it for SM120: ``bash ssh root@[REDACTED] 'ls /root/sglang/sgl-kernel/; echo "---"; cat /root/sglang/sgl-kernel/setup.py 2>/dev/null | head -20; echo "---"; ls /root/sglang/sgl-kernel/src/ 2>/dev/null | head -10' ` Output: ` CMakeLists.txt Dockerfile LICENSE Makefile README.md THIRDPARTYNOTICES.txt analyze_whl_kernel_sizes.py benchmark build.sh cmake csrc include kernel-runner-setup.sh pyproject.toml pyproject_cpu.toml pyproject_musa.toml pyproject_rocm.toml python rename_wheels.sh setup_musa.py setup_rocm.py tests --- --- ``

The assistant begins with a diagnosis: "The sgl_kernel is the problem — it's compiled for SM100 not SM120." This is a critical observation. In the previous message ([msg 3098]), the assistant had attempted to import sgl_kernel and received an ImportError stating that no architecture-specific ops could be loaded for SM120. The installed sgl_kernel had been compiled for SM100 (the previous Hopper-generation compute capability), but the Blackwell GPUs require SM120-specific kernels.

The reconnaissance command is carefully designed to answer three questions in parallel:

  1. What build infrastructure exists? The ls /root/sglang/sgl-kernel/ lists the top-level directory, revealing CMakeLists.txt, Makefile, build.sh, and multiple pyproject.toml variants. This tells the assistant that sgl-kernel uses a CMake-based build system with Python bindings, and supports multiple backends (CUDA, ROCm, MUSA).
  2. Is there a setup.py for standard pip builds? The cat /root/sglang/sgl-kernel/setup.py command returns empty output (indicated by the two --- markers with nothing between them). This is a meaningful finding: there is no setup.py file in the sgl-kernel directory. Instead, the project uses pyproject.toml, which is the modern PEP 517/518 standard for Python packaging. This means building the kernel will require understanding the pyproject.toml-based build system rather than a traditional setup.py.
  3. What source code is available? The ls /root/sglang/sgl-kernel/src/ command also returns empty output (no listing between the second --- and the end of the message). This suggests there is no src/ directory at that path—the CUDA source files are likely in csrc/ (which was listed in the directory output) or include/. This is an important structural insight for anyone attempting to rebuild the kernel.

The Thinking Process Visible in the Message

The assistant's reasoning is compressed but visible. The message opens with a declarative statement—"There's a recent SGLang checkout"—which is itself a conclusion drawn from the previous message's discovery that SGLang was installed as a development checkout from /root/sglang/ rather than as a pip package. The assistant then immediately identifies the bottleneck: "The sgl_kernel is the problem."

The phrase "Let me check if we can build it for SM120" reveals the assistant's working hypothesis: that the path forward is to recompile the sgl-kernel with SM120 support enabled. This is a reasonable assumption—CUDA kernels compiled for one architecture cannot run on another, and the solution is typically to add the target architecture to the compilation flags (e.g., -gencode arch=compute_120,code=sm_120 in NVCC).

The three-part bash command is a textbook example of efficient debugging: rather than running three separate commands and waiting for each result, the assistant chains them together with echo "---" separators, allowing all three probes to execute in a single SSH session. This parallel approach minimizes latency—a practical consideration when each SSH round-trip adds network delay to a remote server.

Assumptions and Their Validity

The message operates on several assumptions, most of which are well-founded:

  1. That recompiling sgl-kernel for SM120 is feasible. This assumes that the sgl-kernel source code is compatible with SM120 architecture and that the CUDA toolkit installed on the server (CUDA 12.8, as verified in [msg 3097]) supports Blackwell compilation. CUDA 12.8 does include SM120 support, so this assumption is correct.
  2. That the SGLang checkout is recent enough to support SM120. The git log showed three recent commits, suggesting active development. However, SM120 support in SGLang was itself a moving target—the assistant would later discover that SGLang deadlocks on SM120 after loading the model in 22 seconds ([msg 3103] onward).
  3. That building sgl-kernel follows standard CUDA extension patterns. The presence of CMakeLists.txt, csrc/, and pyproject.toml supports this assumption. However, the absence of a setup.py and the empty src/ directory indicate a non-standard build system that may require additional investigation. One assumption that proved partially incorrect was that building sgl-kernel for SM120 would be the primary blocker. In reality, even after successfully building sgl-kernel for SM120 (which took 48 minutes in a subsequent message), the SGLang server would load the model but then deadlock with zero CPU/GPU utilization—a far deeper compatibility issue than mere kernel compilation.

Input Knowledge Required

To fully understand this message, the reader needs:

  1. The CUDA compute capability naming scheme. SM120 refers to the compute capability of NVIDIA Blackwell architecture GPUs (RTX PRO 6000 Blackwell). SM100 was the previous Hopper generation. CUDA kernels must be compiled for specific architectures, and a kernel compiled for SM100 will not load on SM120.
  2. The SGLang project structure. SGLang uses a custom kernel library called sgl-kernel that provides optimized operations for inference (attention, MoE, etc.). This kernel library is architecture-specific and must be compiled for each target GPU.
  3. The conversation's recent history. The pivot from vLLM to SGLang was motivated by vLLM's broken EAGLE-3 integration for DeepSeek V3/MLA models. The assistant had just spent days building an EAGLE-3 training pipeline that produced a drafter with only 15% acceptance rate in vLLM.
  4. The hardware context. Eight NVIDIA RTX PRO 6000 Blackwell GPUs connected via PCIe, running on Ubuntu 24.04 with CUDA 12.8 and PyTorch 2.10.0.
  5. Python packaging conventions. The distinction between setup.py (traditional setuptools) and pyproject.toml (modern PEP 517/518) is relevant to understanding why the cat command returned empty.

Output Knowledge Created

This message produces several concrete pieces of knowledge:

  1. The sgl-kernel build system uses CMake + pyproject.toml, not the traditional setup.py pattern. The presence of CMakeLists.txt, Makefile, build.sh, and pyproject.toml indicates a CMake-based build with Python bindings.
  2. The source code lives in csrc/ and include/, not src/. The csrc/ directory likely contains CUDA source files, while include/ contains headers. The python/ directory likely contains Python bindings.
  3. Multiple backend configurations are supported. The presence of pyproject_cpu.toml, pyproject_musa.toml, and pyproject_rocm.toml indicates support for CPU-only, MUSA (Moore Threads), and ROCm (AMD) backends alongside the default CUDA backend.
  4. The build is non-trivial. The existence of build.sh, rename_wheels.sh, analyze_whl_kernel_sizes.py, and kernel-runner-setup.sh suggests a complex build pipeline with multiple steps beyond a simple pip install.
  5. SM120 support is absent from the current build. The installed sgl-kernel was compiled for SM100, confirming that Blackwell support is not yet available in the pre-built wheels and must be compiled from source.

The Broader Significance

Message <msg id=3101> is a quintessential example of the "infrastructure debugging" that dominates real-world ML engineering at scale. The grand challenges—training models, designing architectures, optimizing algorithms—are often overshadowed by the mundane work of making software run on specific hardware configurations. Here, the assistant is not writing neural network code or tuning hyperparameters; it is reading directory listings and checking for build files.

Yet this work is essential. The decision to pivot to SGLang was correct in principle—SGLang's EAGLE-3 integration is demonstrably better than vLLM's for Kimi-K2 models. But the path from "SGLang supports this" to "SGLang runs on our specific GPUs" is paved with exactly this kind of kernel compatibility investigation. The message captures the moment of transition between two major phases of the project: the EAGLE-3 training pipeline (completed) and the SGLang deployment effort (just beginning).

The empty output from the setup.py and src/ probes is itself informative. It tells the assistant that the build system is not the standard one, and that further investigation of pyproject.toml and csrc/ will be necessary. In the very next message ([msg 3102]), the assistant would examine pyproject.toml and discover the --blackwell build flag, leading to a successful 48-minute kernel compilation. But the deeper SM120 compatibility issue—the server deadlock—would remain unsolved for many more messages, requiring extensive debugging of SGLang's CUDA graph capture and attention kernels.

Conclusion

Message <msg id=3101> is a small but pivotal moment in a much larger engineering narrative. It represents the clean break from a failed approach (vLLM EAGLE-3) and the first cautious steps into a new one (SGLang on SM120). The assistant's methodical reconnaissance—listing directories, checking for build files, understanding the project structure—is the foundation upon which all subsequent debugging rests. In the world of bleeding-edge ML infrastructure, where every GPU generation breaks existing software, the ability to quickly assess and adapt to new build systems is as valuable as any algorithmic insight. This message, for all its brevity, captures that skill in action.