The SM120 Detection Gap: A Critical Diagnostic Moment in SGLang Blackwell Tuning

In the middle of an intensive performance-tuning session for the Kimi-K2.5 INT4 model on 8x NVIDIA RTX PRO 6000 Blackwell GPUs, a single diagnostic command—message [msg 3233]—revealed a fundamental mismatch between SGLang's hardware detection logic and the actual GPU architecture being used. The message is deceptively short: a two-line Python script executed over SSH, checking whether is_sm100_supported() returns True for a GPU with compute capability (12, 0). The answer is False. But this False cascades through dozens of lines of SGLang's server configuration code, silently disabling optimized attention backends and leaving performance on the table.

The Context: Chasing a 30% Single-Stream Gap

To understand why this single boolean check mattered so much, we need to trace the reasoning chain that led to it. The session had reached a critical inflection point. After an exhaustive campaign testing speculative decoding—including patching kimi_k25.py for EAGLE-3 delegation, training a custom drafter on 10,000 samples, and testing both the AQ-MedAI and custom drafters—the assistant had concluded that EAGLE-3 provided no benefit on this hardware. The acceptance rates were disastrously low (42% for the best drafter, 25% for the custom one), and SGLang's automatic max_running_requests=48 limit for speculative mode crippled throughput.

The baseline comparison told a stark story. SGLang base with CUDA graphs achieved an impressive 2,370 tok/s peak throughput at C=128, beating vLLM's peak of 1,536 tok/s by 54%. But in single-stream latency—the metric that matters for interactive applications—SGLang lagged at 63.6 tok/s versus vLLM's 82.5 tok/s, a 30% deficit. The user's directive in [msg 3222] was clear: "vLLM still wins single-stream at 82.5 vs 63.6 tok/s (30% faster per-request) — can we apply tuning that made vllm fast here? (nccl LL etc)?"

The assistant's response in [msg 3223] was to tackle two problems in parallel: tune SGLang's single-stream performance and set up a new EAGLE-3 training pipeline using SGLang-based hidden state extraction. A research sub-task (launched in [msg 3224]) identified the exact NCCL environment variables vLLM used: NCCL_PROTO=LL, NCCL_ALGO=Ring, NCCL_P2P_LEVEL=SYS, NCCL_MAX_NCHANNELS=16, NCCL_BUFFSIZE=16777216, NCCL_NTHREADS=512. The assistant killed the loading server and prepared to restart with these settings.

But then the assistant's attention shifted to a deeper issue. In [msg 3226], it asked: "Now let me check if flashinfer MLA backend actually works on SM120 for DeepSeek models — the current default was triton." This was the moment the assistant began questioning not just the NCCL configuration, but the fundamental attention backend selection.

The Investigation: Tracing the Attention Backend Decision

The assistant's next moves were methodical. In [msg 3227], it grepped server_args.py for attention backend choices, finding the constants SAMPLING_BACKEND_CHOICES, DETERMINISTIC_ATTENTION_BACKEND_CHOICES, and the flashmla_auto option. In [msg 3228], it found the critical line self._handle_attention_backend_compatibility() at line 740, and the assignments self.attention_backend = "trtllm_mla" at line 1286 and self.attention_backend = "triton" at line 1344. In [msg 3229], it found the sm100_default_attention_backend="flashinfer" setting.

The crucial discovery came in [msg 3230] and [msg 3231]. The assistant found that SGLang has explicit code paths for SM100+ Blackwell GPUs:

if model_arch in [
    "DeepseekV3ForCausalLM",
    "KimiK25ForConditionalGeneration",
    ...
]:
    # Set attention backend for DeepSeek

And at line 1288: "Use trtllm_mla as attention backend on sm100 for DeepseekV3ForCausalLM". The trtllm_mla backend uses TensorRT-LLM's Multi-Head Latent Attention kernels, which are specifically optimized for the DeepSeek V2/V3/V4 architecture's MLA mechanism. On SM100 GPUs, this backend should provide significantly better performance than the generic triton fallback.

But in the logs, SGLang was using triton. The assistant's hypothesis in [msg 3232] was precise: "For SM100+ (Blackwell SM120), SGLang auto-selects trtllm_mla attention backend for DeepSeek/KimiK25. But in our logs, it said triton — that's because SM120 might not be detected as is_sm100_supported()."

The assistant then attempted to verify this by running a Python command that imported is_sm100_supported from sglang.srt.utils.utils. But this failed with a ModuleNotFoundError—the correct import path is sglang.srt.utils, not sglang.srt.utils.utils.

The Subject Message: A Corrected Diagnostic

Message [msg 3233] is the corrected diagnostic:

[assistant] [bash] ssh root@[REDACTED] '/root/ml-env/bin/python3 -c "
import torch
print(f\"cap: {torch.cuda.get_device_capability()}\")
from sglang.srt.utils import is_sm100_supported
print(f\"SM100 supported: {is_sm100_supported()}\")
"'
cap: (12, 0)
SM100 supported: False

The output is devastatingly clear. The GPU has compute capability (12, 0)—this is SM120, the architecture of the RTX PRO 6000 Blackwell GPU. But is_sm100_supported() returns False. The function that gates access to the optimized trtllm_mla attention backend does not recognize SM120 as a supported architecture.

Why This Matters: The Performance Implications

This detection gap means that SGLang's Blackwell-specific optimizations—the trtllm_mla attention backend, the flashinfer_trtllm MoE runner backend, and potentially other SM100-specific code paths—are all silently disabled on RTX PRO 6000 Blackwell GPUs. Instead, the server falls through to the generic triton attention backend, which may not be optimally tuned for the MLA architecture on these GPUs.

The performance impact is potentially significant. The trtllm_mla backend uses NVIDIA's TensorRT-LLM kernels, which are hand-tuned for specific GPU architectures and can exploit hardware features like fourth-generation Tensor Cores, FP8/FP4 support, and the Transformer Engine available on Blackwell. The triton backend, while functional, relies on JIT-compiled GPU kernels that may not achieve the same level of optimization.

This finding also reframes the single-stream performance gap. The 30% deficit between SGLang (63.6 tok/s) and vLLM (82.5 tok/s) might not be solely due to NCCL configuration differences. It could also be caused by SGLang running a suboptimal attention backend while vLLM uses its own optimized implementations that properly detect and utilize SM120 capabilities.

Assumptions and Knowledge Required

To fully understand this message, one needs several layers of context. First, knowledge of NVIDIA GPU compute capability numbering: SM100 (compute capability 10.0) refers to the Blackwell B100/B200 data center GPUs, while SM120 (12.0) refers to the RTX PRO 6000 Blackwell workstation GPU. These are both Blackwell architecture but have different capability numbers, and software must explicitly handle both.

Second, familiarity with SGLang's architecture: the server_args.py file contains the logic that selects attention backends based on model architecture and hardware platform. The is_sm100_supported() function is a gatekeeper that determines whether Blackwell-specific optimizations are enabled.

Third, understanding of the MLA (Multi-Head Latent Attention) mechanism used by DeepSeek V2/V3/V4 and Kimi-K2.5 models. MLA is a novel attention architecture that compresses the key-value cache, and it requires specialized kernel support. The trtllm_mla backend provides this support through TensorRT-LLM, while the triton backend uses a more generic implementation.

The assistant made an important assumption: that SM120 should be treated as SM100-compatible for the purposes of attention backend selection. This assumption is reasonable—both are Blackwell architecture—but it may not match SGLang's implementation, which could have been written before SM120 GPUs were available or could have intentionally excluded them due to differences in supported features.

The Output Knowledge Created

This message creates critical diagnostic knowledge: SGLang's SM100 detection does not cover SM120 GPUs. This explains why the trtllm_mla attention backend was not being used, and it provides a clear path forward: either patch is_sm100_supported() to include SM120, or manually specify the attention backend via --attention-backend trtllm_mla (if that backend supports SM120 at all).

The message also validates the assistant's investigative methodology. Starting from a performance observation (SGLang is 30% slower than vLLM single-stream), the assistant traced the issue through the codebase, identified the attention backend selection logic, hypothesized a detection gap, corrected an import error, and confirmed the hypothesis with a clean diagnostic command. This is a textbook example of systematic debugging.

The Broader Implications

This detection gap has implications beyond this single session. Any SGLang user deploying on RTX PRO 6000 Blackwell GPUs (or any SM120 GPU) would silently miss out on Blackwell-specific optimizations. The issue would manifest as unexplained performance deficits that are difficult to diagnose without deep codebase knowledge.

The fix is straightforward: modify is_sm100_supported() to return True for compute capability 12.0 as well as 10.0. But the fix requires careful consideration—are the SM100-optimized kernels actually compatible with SM120? The TensorRT-LLM kernels might need SM120-specific versions, or they might work with degraded performance. The answer requires further testing.

In the subsequent messages ([msg 3234] and [msg 3235]), the assistant continues the investigation, searching for the definition of is_sm100_supported and is_blackwell in the SGLang codebase, finding them in sglang/multimodal_gen/runtime/platforms/interface.py and sglang/test/test_utils.py—but notably not in the core sglang/srt/utils/ directory where the import expected them. This search reveals that the function may not exist in the expected location, or may have a different implementation than assumed.

Conclusion

Message [msg 3233] is a masterclass in diagnostic precision. In just two lines of Python, it exposes a fundamental mismatch between SGLang's hardware detection and the actual GPU architecture, providing the key insight needed to understand—and potentially resolve—a 30% single-stream performance gap. The message demonstrates that sometimes the most valuable contributions are not complex code changes or elaborate experiments, but simple, well-targeted questions that cut to the heart of a problem. The False returned by is_sm100_supported() is not just a boolean—it's a roadmap to the next optimization frontier.