The SM120 Discovery: When a Compiled Kernel Becomes a Dead End

Introduction

In the course of optimizing EAGLE-3 speculative decoding on an 8× RTX PRO 6000 Blackwell system, the assistant reached a critical juncture. Message [msg 5132] captures a moment of discovery that would fundamentally alter the trajectory of the optimization effort. It is a short message—barely a paragraph of analysis followed by a single bash command—but it represents the culmination of a deep investigative chain and the revelation of a hard architectural constraint. This article examines that message in detail: the reasoning that led to it, the assumptions it challenged, the knowledge it produced, and the strategic pivot it forced.

The Message

The assistant writes:

The allreduce kernel is compiled into sm100/common_ops.abi3.so (and sm90/). Since we're on SM120 (Blackwell), let me check which one is being used:

``bash ssh root@10.1.230.174 'ls /root/ml-env/lib/python3.12/site-packages/sgl_kernel/ | head -30' ``

Output: `` __init__.py __pycache__ _fa4_interface.py allreduce.py attention.py cutlass_moe.py elementwise.py expert_specialization.py flash_attn.py flash_mla.py flash_ops.abi3.so flashmla_ops.abi3.so fused_moe.py gemm.py grammar.py kvcacheio.py load_utils.py mamba.py marlin.py memory.py moe.py quantization sampling.py scalar_type.py sm100 sm90 sparse_flash_attn.py spatial.py spatial_ops.abi3.so speculative.py ``

The message is deceptively brief. A casual reader might see only a routine directory listing. But in the context of the session, this listing delivers a devastating verdict: there is no sm120 directory. The pre-compiled allreduce kernels ship only for SM90 (Hopper) and SM100 (a compute architecture), not for SM120 (Blackwell). This single observation collapses an entire line of optimization work.

The Reasoning Chain That Led Here

To understand why this message was written, one must trace the reasoning that brought the assistant to this point. The session had been wrestling with a persistent performance problem: EAGLE-3 speculative decoding was producing only ~54 tok/s against a baseline of ~90 tok/s. The bottleneck was the "verify pass"—the step where the draft model's predictions are checked against the target model. Analysis revealed that each verify cycle spent approximately 30 milliseconds, with roughly 97% of that time consumed by the target model forward pass. Within that forward pass, 122 NCCL all-reduce operations accounted for the bulk of the communication cost.

The assistant's optimization strategy, documented in the eagle-fast-verify.md plan, was to replace NCCL all-reduce with SGLang's custom allreduce kernel. The custom kernel uses GPU peer-to-peer (P2P) access and shared memory to perform reductions in a single pass, potentially cutting latency from ~200µs per allreduce to ~30-50µs. For 122 allreduces per verify cycle, this could save 18-21ms—enough to make speculative decoding profitable.

The implementation proceeded through several layers. First, the assistant read the Python dispatch code in custom_all_reduce.py and discovered that the constructor and should_custom_ar method both gate on full_nvlink. On PCIe-connected GPUs, this flag is False, and the code explicitly refuses to use custom allreduce for more than two GPUs. The assistant patched these gates, adding a SGLANG_FORCE_CUSTOM_AR_PCIE environment variable to bypass the check.

But the Python patches were only the first layer. The assistant then examined the C++ kernel header (custom_all_reduce.cuh) and discovered a deeper problem: the kernel dispatch logic has no branch for PCIe with more than two GPUs. When full_nvlink_ is False and world_size_ is not 2, the kernel simply does nothing—no reduction is performed. The code structure is:

if (world_size_ == 2) {
    // 2 GPUs: always 1-stage
} else if (full_nvlink_) {
    // NVLink: pick 1-stage or 2-stage based on size
}
// else: nothing!

This meant the C++ kernel itself needed modification to add a PCIe branch. The 1-stage reduce kernel, which performs a simple barrier-and-read pattern, would be suitable for small tensors on PCIe. But modifying the kernel raised a new question: is the kernel compiled at install time, or is it JIT-compiled at runtime? The answer would determine whether a source-level patch was sufficient or whether the entire sgl-kernel package needed rebuilding.## The Discovery: No SM120 Kernel

Message [msg 5132] is the moment this question is answered. The assistant runs ls on the sgl_kernel package directory and finds two architecture-specific subdirectories: sm100 and sm90. There is no sm120. The pre-compiled allreduce kernel lives inside sm100/common_ops.abi3.so—a binary compiled for the SM100 compute architecture, not for Blackwell's SM120.

This is a critical architectural constraint. NVIDIA GPU compute capabilities are identified by "SM" (Streaming Multiprocessor) version numbers. SM90 corresponds to the Hopper architecture (H100), SM100 corresponds to... well, this is where it gets interesting. The naming suggests SM100 is an intermediate architecture, possibly a variant or a specific compute capability level. Blackwell (RTX PRO 6000) uses SM120. The compiled .so files are architecture-specific—they contain PTX or SASS code compiled for specific SM versions. A kernel compiled for SM100 cannot run on SM120 without either a JIT recompile from PTX (intermediate assembly) or a dedicated SM120 build.

The assistant's immediate reaction—"Since we're on SM120 (Blackwell), let me check which one is being used"—shows a correct intuition. If the allreduce kernel is only compiled for SM90 and SM100, and the system runs SM120, then either: (a) the kernel runs through PTX JIT recompilation (if PTX is bundled), or (b) the kernel simply doesn't work on Blackwell, and the system falls back to NCCL silently.

Assumptions and Their Consequences

Several assumptions underpin the work leading to this message, and the discovery challenges each of them.

Assumption 1: The custom allreduce kernel is architecture-agnostic. The assistant had been working under the implicit assumption that the C++ kernel code, once patched to add a PCIe branch, would compile and run on any GPU. The discovery that kernels are pre-compiled per-architecture reveals a more complex reality. The sm100/common_ops.abi3.so file is a compiled binary object; modifying the source in custom_all_reduce.cuh would require rebuilding the entire sgl-kernel package for the target architecture.

Assumption 2: The Python patches would be sufficient. The assistant invested significant effort in modifying the Python dispatch logic—adding env var checks, bypassing NVLink gates, setting size limits. These changes were necessary but insufficient. Without a corresponding C++ kernel that can execute on SM120, the Python patches route calls to a kernel that may silently fall through to a no-op.

Assumption 3: The kernel dispatch would gracefully degrade. The assistant had previously verified that P2P access works across all 8 GPUs over PCIe. The assumption was that if the custom kernel could be made to run, it would work correctly. The discovery about SM120 compilation raises a more fundamental question: does the SM100-compiled kernel even load and execute on SM120? If it does (via PTX JIT), it may work but with suboptimal performance. If it doesn't, the fallback to NCCL is invisible—the system continues running, just without the optimization.

Assumption 4: The architecture support matrix is known. The assistant assumed that sm100 was the relevant directory for Blackwell. In reality, Blackwell is SM120, and the absence of an sm120 directory is a significant gap. This suggests that the sgl-kernel package was built before Blackwell support was added, or that Blackwell support requires a newer version of the package.

The Input Knowledge Required

To understand this message, a reader needs several layers of context:

The optimization goal: EAGLE-3 speculative decoding is bottlenecked by NCCL all-reduce during the verify pass. The custom allreduce kernel promises to reduce latency from ~200µs to ~30-50µs per operation.
The architecture of the custom allreduce system: It has two components—a Python dispatch layer that decides whether to use custom allreduce based on NVLink status, and a C++ CUDA kernel that performs the actual reduction using P2P shared memory.
The PCIe topology: The 8 RTX PRO 6000 GPUs are connected via PCIe Gen5, not NVLink. P2P access works, but the NVLink detection gates prevent the custom kernel from being used.
The SM architecture numbering: SM90 = Hopper, SM100 = an intermediate architecture, SM120 = Blackwell. Compiled CUDA kernels are architecture-specific.
The sgl-kernel package structure: The package ships pre-compiled .so files organized by SM version in sm90/ and sm100/ subdirectories.

The Output Knowledge Created

This message produces several critical pieces of knowledge:

The allreduce kernel is pre-compiled, not JIT. The presence of sm100/common_ops.abi3.so confirms that the kernel is compiled at package build time, not at runtime. This means source-level patches to custom_all_reduce.cuh are insufficient—the package must be rebuilt.
SM120 (Blackwell) has no dedicated kernel directory. The absence of sm120/ means either the kernel runs through PTX compatibility from the SM100 build, or it doesn't run at all. This requires further investigation.
The optimization path is blocked at the compilation level. Even if the Python patches are correct and the C++ kernel source is modified, the compiled binary won't work on SM120 without a proper build. This shifts the problem from "how do we enable custom allreduce on PCIe?" to "how do we get a Blackwell-compiled sgl-kernel?"
The fallback behavior is silent. The system continues to serve requests even if the custom allreduce kernel fails to load or execute. The NCCL fallback is transparent, which means the optimization could be "enabled" in configuration but have zero effect—a dangerous class of bug where changes appear to work but don't actually change behavior.## The Strategic Pivot The discovery in [msg 5132] did not end the optimization effort—it redirected it. The assistant now understood that the custom allreduce kernel, as shipped, was compiled for architectures that do not include Blackwell. The path forward required either: (a) rebuilding sgl-kernel from source for SM120, which would require a CUDA toolkit that supports Blackwell compilation; or (b) upgrading the entire CUDA stack to version 13, which natively supports SM120 and might unblock other Blackwell-native optimizations like FlashInfer fusion and Torch symmetric memory. The user, recognizing this dead end, proposed exactly that: upgrading to CUDA 13. The assistant confirmed that the driver (590.48.01) already supports CUDA 13.1, but the installed toolkit was only 12.8. Research revealed that PyTorch nightly provides cu130 wheels, sgl-kernel has a dedicated cu130 index, and FlashInfer also supports CUDA 13. This upgrade path offered a way to bypass the SM120 compilation gap entirely—not by patching the existing SM100 kernel, but by moving to a CUDA version where Blackwell is a first-class target.

The Thinking Process Visible in This Message

The assistant's reasoning in [msg 5132] reveals a methodical, layered investigative approach. The thought process visible in the message's structure is:

Observation: "The allreduce kernel is compiled into sm100/common_ops.abi3.so (and sm90/)." This is the key insight from the previous investigation—the assistant had just discovered that the kernel lives in architecture-specific directories.
Hypothesis: "Since we're on SM120 (Blackwell), let me check which one is being used." The assistant correctly identifies that the SM architecture of the GPUs (SM120) may not match the compiled kernel directories (SM90, SM100). This is a hypothesis that needs testing.
Test: A simple ls command on the sgl_kernel package directory, listing the top-level contents. This is the minimal test that answers the question: which architecture directories exist?
Interpretation: The output shows sm100 and sm90 but no sm120. The assistant doesn't need to state the conclusion explicitly—the listing speaks for itself. The absence of an SM120 directory means the pre-compiled kernel may not be compatible with Blackwell GPUs. This thinking process exemplifies a core skill in systems optimization: tracing a problem through layers of abstraction. The assistant started at the application level (EAGLE-3 throughput), identified a bottleneck (NCCL all-reduce), traced it to the dispatch layer (Python gates), then to the kernel implementation (C++ CUDA code), and finally to the compilation system (architecture-specific binaries). Each layer revealed new constraints, and the final layer—the package structure—revealed a constraint that could not be patched away with source modifications.

Conclusion

Message [msg 5132] is a turning point in the optimization narrative. It is the moment when a promising line of work—patching the custom allreduce kernel for PCIe—hits a fundamental architectural barrier. The discovery that sgl-kernel ships pre-compiled binaries for SM90 and SM100 but not for SM120 (Blackwell) means that no amount of Python or C++ source patching can make the custom allreduce work on this hardware without a full package rebuild targeting the correct architecture.

The message is brief, but its implications are far-reaching. It forces a strategic pivot from "patch the existing kernel" to "upgrade the CUDA stack to support Blackwell natively." It reveals that the optimization problem has a compilation dependency that was invisible at the start. And it demonstrates the importance of understanding the full software stack—from application code through dispatch logic through kernel source through build system—when debugging performance issues on modern GPU hardware.

For anyone working with pre-compiled GPU kernels on cutting-edge hardware, the lesson is clear: always verify that the compiled binaries match your target architecture. The presence of sm90/ and sm100/ directories does not guarantee that sm120/ exists, and the silent fallback to NCCL means you could spend hours optimizing a path that was never actually being executed.