The First Smoke Test: Validating SGLang on Blackwell SM120 Hardware

In the sprawling, multi-session journey to deploy and train the Qwen3.6-27B model across eight NVIDIA RTX PRO 6000 Blackwell GPUs, there comes a moment that every engineer recognizes: the first smoke test. Message [msg 9473] captures this pivotal instant — a single bash command dispatched to launch SGLang on GPU 0, the first time the freshly assembled software stack meets the unfamiliar SM 12.0 hardware. The message is brief, almost anticlimactic on the surface: a server launch command, a PID echo, a three-second sleep, and a tail of the log. But beneath this terseness lies the culmination of extensive research, a cascade of dependency decisions, and the quiet tension of a system that has never been tested on this architecture before.

The Strategic Context: Why This Message Exists

To understand why [msg 9473] was written, one must trace the strategic pivot that preceded it. The broader session (Segment 54) had been focused on training the DFlash drafter architecture, but the user made a deliberate decision to halt training and repurpose the eight Blackwell GPUs for high-throughput batch inference instead. The goal was data expansion: generating 193K diverse prompts across multiple datasets to enrich the training corpus. This was a data-centric intervention, a recognition that model quality might be bottlenecked by data diversity rather than architecture.

The assistant's immediate task was therefore to transform a machine configured for distributed training into a high-throughput inference server farm. This required installing SGLang — a relatively new inference engine — on a GPU architecture (SM 12.0, the desktop Blackwell variant) that had barely been supported by the software ecosystem. The assistant had spent the preceding messages ([msg 9452] through [msg 9472]) researching compatibility constraints, wrestling with dependency resolution, and navigating a minefield of version mismatches.

The Research That Led Here

The assistant's reasoning in the messages leading up to [msg 9473] reveals a deep engagement with the hardware-software compatibility landscape. Through extensive research, the assistant had established several critical constraints:

SM 120 is not SM 100. The RTX PRO 6000 Blackwell uses the SM 12.0 compute architecture, which differs significantly from the datacenter Blackwell (SM 100) used in B200 GPUs. The most consequential difference: SM 120 lacks the tcgen05 instructions required for Flash Attention 3 and 4 (FA3/FA4). Any attempt to use these attention backends would fail silently or crash. The assistant correctly identified that --attention-backend flashinfer was the only viable option, forcing SGLang to use Flash Attention 2 (FA2) via the flashinfer library.

Triton attention is also broken. The assistant discovered that Triton-based attention kernels assume 114 KB of shared memory per thread block, but the RTX PRO 6000 only provides 99 KB. This mismatch means the Triton backend would fail at runtime. The flashinfer backend avoids this issue entirely.

Tensor parallelism requires CUDA 12.9+. NVIDIA's NCCL library has a known issue with SM 12.x on CUDA versions earlier than 12.9. However, the assistant correctly reasoned that this constraint only applies to tensor parallelism (where GPUs communicate via NCCL during a single forward pass). Since the deployment strategy was data parallelism — eight independent SGLang processes communicating via HTTP, each handling different requests — NCCL was not in the critical path. CUDA 12.8 (or in this case, 12.9/13.0 as resolved by uv) was sufficient.

The mamba scheduler strategy matters. For the Qwen3.6 hybrid architecture (which combines transformer attention with Mamba state-space model layers), SGLang offers two scheduler strategies: extra_buffer (which pre-allocates extra GPU memory for the Mamba state to maximize throughput) and no_buffer (which is more memory-conservative). The assistant chose extra_buffer for this test, prioritizing throughput — a decision that would later be reversed when memory pressure became an issue.

MTP/EAGLE speculative decoding was unavailable. The assistant checked for mtp.safetensors in the model directory ([msg 9453]) and found none. This meant the Multi-Token Prediction (EAGLE) heads were not present, ruling out speculative decoding. The assistant rationalized this as acceptable for high-concurrency batch inference, where GPU compute is already saturated and MTP overhead could hurt throughput.

The Installation Ordeal

Before the smoke test could happen, SGLang had to be installed — and this proved far from straightforward. The assistant's reasoning traces a painful dependency chain:

The initial attempt to install SGLang v0.5.11 failed because uv could not resolve the flashinfer-python==0.6.8.post1 dependency. Adding --index-strategy unsafe-best-match led to a flash-attn-4 version conflict. Only by adding --prerelease=allow did the installation succeed — but it resolved to SGLang v0.5.12 (a newer version than intended) and, critically, upgraded PyTorch from 2.11.0+cu128 to 2.11.0+cu130 (and eventually to 2.12.0+cu130 after a torchvision reinstallation cascade).

This version drift from cu128 to cu130 was a significant change. The assistant initially welcomed it ("actually better for SM120 support since CUDA 13.0 has better SM12x support" — [msg 9466]), but this assumption would later prove costly. The cu130 upgrade consumed additional GPU memory that would eventually cause out-of-memory errors during training, forcing a painful rollback in subsequent chunks.

The assistant also had to bootstrap the environment from scratch — there was no pip in the venv, no uv installed, and ensurepip was missing. The solution was to install uv via the astral.sh install script ([msg 9461]), a bootstrap that added yet another layer of tooling to the stack.

The Smoke Test Command

With the installation finally verified — PyTorch 2.12+cu130, CUDA available, SGLang importable — the assistant wrote a launch script for all 8 GPUs ([msg 9472]) and then, in [msg 9473], executed the critical first test on a single GPU:

mkdir -p /workspace/sglang_logs && CUDA_VISIBLE_DEVICES=0 /root/venv/bin/python3 -m sglang.launch_server \
  --model-path /dev/shm/Qwen3.6-27B \
  --reasoning-parser qwen3 \
  --attention-backend flashinfer \
  --mem-fraction-static 0.88 \
  --max-running-requests 64 \
  --context-length 8192 \
  --host 0.0.0.0 --port 30000 \
  --trust-remote-code \
  --chunked-prefill-size 4096 \
  --mamba-scheduler-strategy extra_buffer \
  --mamba-ssm-dtype bfloat16

Each flag represents a deliberate choice informed by the research:

• --reasoning-parser qwen3: Enables the Qwen3-specific reasoning parser, which handles the model's chain-of-thought output format. This was documented in the SGLang Qwen3.6 cookbook and is essential for proper output formatting.
• --attention-backend flashinfer: The only viable backend for SM 120, as established by the research. FA3/FA4 and Triton are both non-functional on this hardware.
• --mem-fraction-static 0.88: Slightly conservative relative to the 0.90 the assistant had initially planned. This reserves 88% of GPU memory for the model and KV cache, leaving 12% for overhead. The RTX PRO 6000 has 48 GB of VRAM, so this allocates ~42 GB for SGLang, leaving ~6 GB of headroom.
• --max-running-requests 64: A generous concurrency limit. Each request consumes KV cache memory, and with 8K context windows, 64 concurrent requests would require significant memory. This was an aggressive starting point.
• --context-length 8192: Matches the generation scripts' expected context window. The model supports longer contexts, but 8K was the target for data generation.
• --chunked-prefill-size 4096: Enables chunked prefill, where long input prompts are processed in 4096-token chunks. This reduces peak memory usage during the prefill phase and improves scheduling flexibility.
• --mamba-scheduler-strategy extra_buffer: Prioritizes throughput by pre-allocating extra GPU memory for Mamba state management. This was the throughput-maximizing choice.
• --mamba-ssm-dtype bfloat16: Uses bfloat16 precision for the Mamba state-space model computations, matching the model's native precision.

The Assumptions Embedded in This Test

The smoke test in [msg 9473] rests on several assumptions, some explicit and some implicit:

That the sgl-kernel (now sglang-kernel) package includes SM 120 binaries. This assumption proved incorrect. The installed sgl_kernel package (version 0.4.2.post2) only contained sm90/ and sm100/ directories — no sm120/ support. The server would fail to start with an ImportError about missing common_ops libraries, as revealed in [msg 9483]. The assistant had noted in earlier research that "pre-built wheels exist at sgl-project/whl/releases" but had not verified that the specific wheel installed by uv actually contained SM120 support.

That deep_gemm would not interfere. The assistant assumed that since the model runs in BF16 (not FP8), the deep_gemm library (which provides FP8 GEMM kernels) would not be needed and would either be absent or gracefully disabled. In reality, deep_gemm was installed as a dependency of SGLang and failed at import time because it could not find CUDA_HOME (no CUDA toolkit was installed in the container — only the NVIDIA driver libraries). This caused the server to crash before it could even begin loading the model, as seen in the subsequent message [msg 9474].

That the cu130 upgrade was benign. The assistant assumed that moving from cu128 to cu130 was purely beneficial ("better for SM120 support"). While CUDA 13.0 does have improved SM 12.x support, the upgrade also pulled in newer versions of Triton (3.7.0), NCCL, and other CUDA-12.x libraries. These version bumps consumed additional GPU memory — a cost that would not be apparent during this single-GPU smoke test but would become critical later when all 8 GPUs were loaded for training.

That extra_buffer is the right strategy. The assistant chose the throughput-maximizing mamba scheduler strategy. This assumption would later be revisited: when memory pressure became an issue during full-scale generation, the strategy was switched to no_buffer, which doubled the concurrent request capacity from 37 to 72 per GPU.

What This Message Does Not Show

The message captures only the launch — the PID is returned (GPU0_PID=33917), and the first few log lines show deprecation warnings from PyTorch's _pytree.py. The actual outcome of the test is not visible in this message. It is only in the following message ([msg 9474]) that the failure becomes apparent: the server crashed with an AssertionError from deep_gemm, followed by the sgl_kernel SM120 compatibility issue.

This temporal gap is a consequence of the tool execution model. The assistant launched the server in the background (&), slept for 3 seconds, and checked the log. But the server process may have taken longer than 3 seconds to fail, or the log output may have been buffered. The truncated log output in [msg 9473] — ending with "..." — hints that the full error was not yet visible. The assistant would need to wait longer and re-check, which is exactly what happens in [msg 9474] with a 30-second sleep.

The Knowledge Flow

Input knowledge required to understand this message includes: familiarity with NVIDIA's compute capability numbering (SM 12.0 vs SM 10.0), the architectural differences between desktop and datacenter Blackwell GPUs, SGLang's server configuration flags and their meanings, the Qwen3.6 hybrid architecture (transformer + Mamba), and the concept of chunked prefill for inference optimization.

Output knowledge created by this message is primarily negative: it establishes that the current software stack does not work on SM 120 hardware. The failure cascades through two distinct issues — deep_gemm's CUDA_HOME requirement and sgl_kernel's missing SM120 binaries — each requiring separate debugging and remediation. This negative result is valuable: it prevents the assistant from naively scaling to 8 GPUs and failing catastrophically, and it focuses the debugging effort on the two specific components that need fixing.

The Thinking Process

The assistant's reasoning in this message is largely implicit — the thinking happened in the preceding messages, and this message is the execution of that thought. But the choice of flags reveals the reasoning structure:

The assistant is methodically reducing risk. Rather than launching all 8 GPUs simultaneously (which would multiply any failure by 8), it tests on a single GPU first. Rather than using the most aggressive settings, it uses slightly conservative ones (0.88 mem fraction instead of 0.90). Rather than assuming the installation works, it verifies with a real server launch.

This is the engineering mindset: theory must be validated by experiment. All the research about SM120 constraints, flashinfer compatibility, and mamba scheduler strategies was hypothesis formation. The smoke test in [msg 9473] is the experiment that tests those hypotheses against reality. And as the following messages will show, reality had surprises in store.