The Moment the Draft Model Broke: When SM100-Only Kernels Collided with Blackwell

The Message

In the middle of a high-stakes optimization campaign for DeepSeek-V4-Flash on 8× RTX PRO 6000 Blackwell (sm_120) GPUs, the assistant issued a seemingly routine health check:

timeout 25 ssh -o StrictHostKeyChecking=no root@10.1.230.171 \
  'grep -c "fired up and ready" /root/dsv4_mtp.log; \
   echo "--- errors ---"; \
   grep -inE "OutOfMemoryError|SIGQUIT received|Capture cuda graph failed|Traceback|squeeze|shape mismatch|assert" \
     /root/dsv4_mtp.log | grep -ivE "server_args" | tail -10; \
   echo "--- tail ---"; \
   tail -3 /root/dsv4_mtp.log' </dev/null

The output was devastating in its brevity:

0
--- errors ---
258:[2026-06-18 01:48:52 TP2] Scheduler hit an exception: Traceback (most recent call last):
357:[2026-06-18 01:48:52] SIGQUIT received. signum=None, frame=None. It usually means one child failed.
359:[2026-06-18 01:48:52 TP1] Scheduler hit an exception: Traceback (most recent call last):
458:[2026-06-18 01:48:52 TP0] Scheduler hit an exception: Traceback (most recent call last):
557:[2026-06-18 01:48:52 TP3] Scheduler hit an exception: Traceback (most recent call last):

The server had never started — ready=0 — and all four TP ranks had crashed simultaneously with scheduler exceptions, culminating in a SIGQUIT that signaled a complete child-process failure. This message, message index 12656 in the conversation, marks the precise moment when an ambitious attempt to integrate EAGLE speculative decoding (MTP) into a heavily customized inference stack met its first hard wall.

Why This Message Was Written

The message was written as a diagnostic probe — a standard "is it alive?" check following the launch of a new server configuration. In the immediately preceding message ([msg 12655]), the assistant had started an MTP-enabled server with the command:

python -m sglang.launch_server \
  --model-path /root/models/DeepSeek-V4-Flash-NVFP4 --trust-remote-code \
  --tp 4 --moe-runner-backend triton \
  --mem-fraction-static 0.60 --cuda-graph-max-bs 64 \
  --context-length 131072 \
  --speculative-algorithm EAGLE --speculative-num-steps 1 \
  --speculative-eagle-topk 1 --speculative-num-draft-tokens 2 \
  --host 127.0.0.1 --port 30000

This was the culmination of Phase 2 of a three-phase optimization plan. Phase 1 (NCCL all-reduce optimization) had just been concluded with the verdict that the PCIe topology was already at its floor — both flashinfer all-reduce fusion and MSCCL++ had shown no improvement over the baseline NCCL ring protocol. The assistant had accepted this limitation and moved on to Phase 2: MTP (Multi-Token Prediction, also known as EAGLE speculative decoding), which promised to improve throughput by having a lightweight draft model predict multiple tokens per forward pass.

The health check in message 12656 was therefore the first feedback point after launching this new configuration. The assistant needed to know whether the server had initialized successfully, whether the custom MMA attention kernel and Triton indexer kernel were compatible with the EAGLE verify path, and whether the draft model could coexist with the heavily modified inference stack.

The Context: A Hard-Won Optimization Campaign

To understand the weight of this moment, one must appreciate what had been accomplished in the preceding hours. The assistant had been engaged in a grueling kernel optimization campaign for DeepSeek-V4-Flash on Blackwell sm_120 GPUs — hardware that was cutting-edge enough that many CUDA kernel libraries had not yet been ported to it.

The journey had included:

  1. Designing a custom MMA sparse-MLA decode kernel using Triton tl.dot tensor-core operations, replacing a per-head SIMT kernel that was re-reading the KV cache 64× redundantly. This alone delivered a 2.2–2.9× throughput improvement.
  2. Flipping forced-FP32 operations to bf16 — the indexer bmm and MHC-pre linear were running in FP32 despite being tensor-core-eligible. Converting them to bf16 eliminated wasteful cast overhead.
  3. Discovering and fixing the indexer O(max_context) bottleneck — the DSA indexer was computing scores over the full ~1M-token max context (262,208 c4-positions) every decode step, even when the actual context was ~512 tokens. This single issue accounted for ~69% of GPU time. Capping --context-length 8192 cut the indexer work ~128×, delivering a dramatic breakthrough: C=64 went from 29.7 to 531.7 tok/s (17.9× improvement).
  4. Building a capture-safe Triton indexer kernel with early-exit per page, making the computation O(actual seq) regardless of context length, validated at 128K context with ~96–98% throughput retention. These optimizations were committed as checkpoints eb54448ab and 598928d75, and the assistant had just verified a clean git tree before moving to Phase 2. The performance numbers were extraordinary — the system was now delivering 300–600 tok/s, squarely in the target range. The assistant was riding high on this success when it turned to MTP integration.## What the Message Revealed: A Multi-Rank Crash The output of message 12656 told a grim story. The grep -c &#34;fired up and ready&#34; returned 0 — the server had never completed initialization. The error log showed scheduler exceptions on all four TP ranks (TP0 through TP3), all at the same timestamp 01:48:52, followed by a SIGQUIT. The SIGQUIT message explicitly stated: "It usually means one child failed." This pattern — simultaneous crashes across all ranks with a SIGQUIT — is characteristic of a fatal error during CUDA graph capture or model initialization, where one rank encounters an unrecoverable error and the collective communication layer propagates the failure. The assistant had anticipated this possibility: the grep pattern included &#34;squeeze&#34; and &#34;shape mismatch&#34; because the custom MMA attention kernel (flash_mla_sm120_triton.py) assumed a single query token (q.squeeze(1)), and the EAGLE verify path processes multiple draft tokens simultaneously. The assistant had explicitly noted this risk in its reasoning: "My MMA kernel assumes 1 query token (q.squeeze(1)), but EAGLE verify processes num_draft_tokens=2." However, the actual error turned out to be different — and deeper — than the shape mismatch the assistant had feared.

The Diagnostic Trail: Following the Breadcrumbs

The assistant did not stop at message 12656. The following messages ([msg 12657] through [msg 12665]) form a diagnostic chain that reveals the true nature of the problem. In [msg 12657], the assistant extracted the full traceback from the log, which pointed to a crash in deepseek_v4_nextn.py — the EAGLE draft model's MoE (Mixture of Experts) layer — not the attention kernel. The traceback led through deepseek_v2.py:forward_normal → self.experts() → into a flashinfer TensorRT-LLM kernel.

In [msg 12659], the assistant identified the specific kernel that was crashing: trtllm_fp4_block_scale_moe, a kernel with the suffix sm100f — meaning it was compiled exclusively for NVIDIA's SM100 architecture (Hopper/H100). The Blackwell RTX PRO 6000 GPUs use SM120, a newer architecture with a different instruction set. The kernel was simply not compatible.

This led to a deeper investigation of the MoE backend selection logic. The draft model's MoE was quantized as MXFP4 (not NVFP4 like the main model), and the MXFP4 dispatch logic in mxfp4.py was selecting the flashinfer TensorRT-LLM backend — which only supports SM100 — because the code path for SM120 was either incomplete or gated behind conditions that weren't being met.

Assumptions and Misconceptions

Several assumptions underpinned the MTP launch attempt, and several of them proved incorrect:

Assumption 1: The draft model would use the same MoE backend as the main model. The assistant had explicitly passed --moe-runner-backend triton for the main model, and the speculative hook was supposed to copy this to the draft model. However, the draft model's MoE quantization method (MXFP4) had its own backend selection logic that overrode this setting. The --speculative-moe-runner-backend triton flag was added in [msg 12664] as a fix, but it also failed — the draft MoE was hardcoded to use flashinfer_trtllm regardless.

Assumption 2: The NVFP4 quantization detection would set the quantization flag. The assistant assumed that because the model was NVFP4-quantized, the --quantization flag would be set to &#34;modelopt_fp4&#34;, which would trigger the gating logic in server_args.py:2222 that forces the draft MoE to use triton. In reality, NVFP4 was auto-detected from HuggingFace config metadata, leaving quantization=None, and the gating condition was never met.

Assumption 3: The draft model's MXFP4 MoE would work on SM120. This was the most fundamental assumption — that the draft model, being part of the same checkpoint, would be compatible with the same hardware. In fact, the MXFP4 MoE dispatch logic had no working SM120 path; it fell through to the SM100-only flashinfer kernel, which immediately crashed.

Assumption 4: The custom MMA attention kernel would be the point of failure. The assistant had correctly identified that the MMA kernel's squeeze(1) operation would fail with multiple draft tokens, but this turned out to be a secondary concern — the server never even reached the point of running a decode step because the draft model's MoE initialization crashed during CUDA graph capture.## Input Knowledge Required

To fully understand message 12656, a reader needs familiarity with several domains:

SGLang architecture: The message assumes knowledge of how SGLang serves models — the concept of TP (tensor parallelism) across multiple ranks, the scheduler process that manages request batching, CUDA graph capture for latency optimization, and the distinction between prefill and decode phases.

EAGLE speculative decoding: The MTP (Multi-Token Prediction) algorithm, where a lightweight draft model predicts multiple future tokens and the main model verifies them in parallel. The --speculative-eagle-topk 1 --speculative-num-draft-tokens 2 flags indicate a configuration where the draft model proposes 2 tokens per step.

Blackwell sm_120 architecture: The RTX PRO 6000 GPUs use NVIDIA's Blackwell architecture (compute capability 12.0), which is newer than the Hopper SM100 (compute capability 9.0). Many CUDA kernels compiled for SM100 are not compatible with SM120, requiring either recompilation or alternative implementations.

NVFP4 vs MXFP4 quantization: The main model uses NVFP4 (NVIDIA's proprietary FP4 format with block scaling), while the draft model uses MXFP4 (the OCP Microscaling format). These have different kernel implementations and different levels of hardware support.

The custom kernel stack: The environment variables SGLANG_SM120_MMA_FLASHMLA=1 and SGLANG_SM120_TRITON_INDEXER=1 are custom flags that the assistant added to enable its optimized kernels. Understanding the message requires knowing that these exist and what they do.

Output Knowledge Created

Message 12656, combined with the diagnostic chain that follows it, creates several important pieces of knowledge:

The draft MoE incompatibility is confirmed: The crash definitively proves that the EAGLE draft model's MXFP4 MoE cannot run on SM120 hardware with the current flashinfer stack. This is a hard blocker, not a transient configuration issue.

The MMA kernel shape concern is deprioritized: The assistant had been worried about whether the custom attention kernel would handle multi-query-token verify steps. The MoE crash means this question never gets tested — the server fails before any decode step begins.

The speculative backend flag is insufficient: Even with --speculative-moe-runner-backend triton explicitly set, the draft MoE still routes to flashinfer_trtllm. This reveals that the backend selection is happening at the quantization-method level, not the server-arg level, and that the MXFP4 quantizer has its own hardcoded dispatch logic.

A strategic pivot point is reached: The failure of MTP integration forces a re-evaluation of priorities. The assistant has already achieved ~17× throughput improvement on the main model. MTP would provide additional gains (particularly at C=1), but the engineering cost to fix the draft MoE dispatch on SM120 is high and uncertain. The rational choice is to document the blocker and move to Phase 3 (PD disaggregation), which offers higher value without requiring draft-model compatibility.

The Thinking Process: A Study in Diagnostic Discipline

The reasoning visible in the messages surrounding 12656 reveals a methodical diagnostic approach. The assistant does not panic at the crash — it immediately begins a structured investigation:

  1. Check the error type (msg 12656): Is it a shape mismatch in my MMA kernel, or something else? The grep pattern is carefully chosen to catch both anticipated failure modes.
  2. Extract the full traceback (msg 12657): The crash is in deepseek_v4_nextn.py → MoE, not in the attention kernel. This immediately shifts the investigation focus.
  3. Identify the specific kernel (msg 12659): The crash is in trtllm_fp4_block_scale_moe, an SM100-only kernel. The assistant recognizes the sm100f suffix and understands the architectural incompatibility.
  4. Trace the backend selection logic (msg 12660–12663): The assistant reads the hook code, the MXFP4 dispatch logic, and the server_args configuration to understand why the draft MoE selected flashinfer_trtllm instead of triton.
  5. Attempt a targeted fix (msg 12664): Adding --speculative-moe-runner-backend triton is a reasonable hypothesis — if the draft backend was defaulting to flashinfer because the main backend was auto-resolved, explicitly setting it should fix the issue.
  6. Verify the fix failed (msg 12665): The same crash occurs even with the explicit flag. The assistant searches the logs for backend resolution messages and finds none — the draft MoE is silently routing to flashinfer regardless of the server arg.
  7. Make the strategic decision: Rather than diving into the MXFP4 dispatch rabbit hole, the assistant recognizes that this is a deep, non-trivial issue and pivots to Phase 3.## Mistakes and Incorrect Assumptions Beyond the assumptions already discussed, several specific mistakes are visible in the reasoning: The --speculative-moe-runner-backend flag was not verified to exist in the codebase before use. The assistant assumed that because --moe-runner-backend existed, there would be a corresponding speculative variant. While it did exist, the assistant did not verify that the draft model's MoE dispatch actually consulted this flag. The subsequent investigation revealed that the MXFP4 quantizer had its own backend selection logic that ignored the server arg entirely. The NVFP4 auto-detection assumption was not validated. The assistant knew that NVFP4 was detected through HuggingFace config metadata rather than the --quantization flag, but did not verify that this auto-detection was setting the internal quantization state correctly. The gating condition in server_args.py:2222 checked quantization == &#34;modelopt_fp4&#34;, but NVFP4 auto-detection left quantization=None, causing the condition to fail silently. The draft model's MXFP4 quantization was not anticipated. The assistant had been working exclusively with the main model's NVFP4 weights and had not inspected the draft model's quantization format. The discovery that the draft used MXFP4 (with its SM100-only kernel path) came as a surprise, and the assistant had no pre-existing knowledge of how MXFP4 dispatch worked on SM120. The SIGQUIT was initially misinterpreted. The assistant's grep pattern included &#34;squeeze&#34; and &#34;shape mismatch&#34; as error indicators, suggesting the primary concern was the MMA kernel's single-query-token assumption. The SIGQUIT and scheduler exceptions turned out to be caused by a completely different issue (MoE kernel incompatibility), and the assistant had to redirect the investigation after the initial check.

The Broader Significance

Message 12656 captures a pivotal moment in a larger engineering narrative. The assistant had just achieved a spectacular ~17× throughput improvement through custom kernel engineering — one of those rare moments where deep understanding of hardware architecture translates into order-of-magnitude gains. The natural next step was to layer on speculative decoding for even more performance.

But the MTP integration failed not because of a bug in the custom kernels, but because of a fundamental architectural incompatibility between the draft model's quantization format and the target hardware. The MXFP4 MoE dispatch logic had been written for SM100 (Hopper) and had not been updated for SM120 (Blackwell). This is a recurring challenge in cutting-edge ML infrastructure: the hardware moves faster than the software stack, and new GPU architectures often require significant engineering to support existing kernel libraries.

The assistant's response to this failure is instructive. Rather than spending days debugging the MXFP4 dispatch logic — a rabbit hole with uncertain payoff — the assistant recognized that the main model was already performing well and pivoted to PD disaggregation (Phase 3), which offered immediate value without the draft-model dependency. This is the essence of pragmatic engineering: knowing when to push through a barrier and when to work around it.

In the end, message 12656 is a testament to the reality of systems engineering at the frontier. The breakthroughs are real and measurable — 17× throughput improvements, custom CUDA kernels, production deployments with monitoring stacks. But so are the blockers — kernel incompatibilities, undocumented dispatch logic, and the ever-present gap between what the hardware can do and what the software stack supports. The art lies in navigating both.