The Pivot to Performance: Investigating MTP and GEMM Backends for Blackwell

A Single Message That Changed the Trajectory of an ML Deployment

In the middle of a sprawling, multi-session effort to deploy large language models on an 8× NVIDIA RTX PRO 6000 Blackwell GPU server, a single short message from the AI assistant marks a critical inflection point. The message, just 41 words long plus a structured todo update, is deceptively simple:

Good questions. Let me investigate both in parallel.

Yet this brief acknowledgment, followed by the creation of two high-priority investigation tasks, represents the moment the session pivoted from "does it work?" to "how fast can we make it?" — a transition that would unlock a tenfold throughput improvement and fundamentally reshape the deployment architecture.

The Context: A Hard-Won Deployment

To understand why this message matters, we must first appreciate what preceded it. The assistant had just completed an extraordinarily complex deployment of the Qwen3.5-397B-A17B-NVFP4 model — a 397-billion-parameter Mixture-of-Experts (MoE) model quantized to NVFP4 (NVIDIA's 4-bit floating-point format). The deployment required upgrading PyTorch to a nightly build (2.12.0.dev20260307+cu130), building sgl-kernel from source with custom patches for Blackwell SM120 support, fixing CMake policy version mismatches, resolving CUDA 13 include path issues for cccl headers, and applying a soft fallback for Flash Attention 3. The final working configuration used flashinfer_cutlass for the MoE runner, flashinfer_cudnn for FP4 GEMM, and triton for the attention backend.

The smoke tests passed. The model returned correct answers. Thinking mode worked. The server was healthy. The assistant had declared "All Done" in a triumphant summary ([msg 5927]), complete with a component upgrade table and a list of five build patches applied.

Then the user responded ([msg 5928]):

Seems really low throughput. 1. Is there built-in MTP? Qwen supports but is it in the quant? 2. can we tune gemms for SM120 (different SMs vs B200)? Is the backend the fastest one we can get?

The throughput was ~72 tok/s at single-request concurrency — a figure that, while respectable for a 397B-parameter model, left significant room for improvement. The user's two questions cut directly to the heart of performance optimization: speculative decoding via Multi-Token Prediction (MTP) and kernel backend selection for the specific GPU architecture.

What the Message Actually Does

The subject message ([msg 5929]) is the assistant's response to this challenge. On its surface, it does three things:

  1. Acknowledges the quality of the questions ("Good questions.") — This is not mere politeness. It signals that the assistant recognizes the user has identified the two most promising levers for performance improvement, and that the assistant is shifting from deployment mode to optimization mode.
  2. Commits to parallel investigation ("Let me investigate both in parallel.") — This is a strategic decision. Rather than tackling the questions sequentially (which would risk a long serial chain of investigation), the assistant will use SGLang's task tool to spawn a subagent for the GEMM backend research while simultaneously running bash commands to inspect the model checkpoint for MTP heads. This parallel approach is characteristic of the assistant's methodology throughout the session.
  3. Creates a structured todo list with two items, both marked status: in_progress and priority: high — This formalizes the investigation into tracked work items, ensuring neither question is dropped or forgotten.

The Reasoning and Motivation

The assistant's reasoning here is worth unpacking. The user's throughput concern is framed around two specific technical hypotheses:

Hypothesis 1: MTP (Multi-Token Prediction) — Qwen3.5's architecture supports predicting multiple future tokens simultaneously, a form of speculative decoding where a lightweight "draft" model proposes tokens that the main model then verifies. If the NVFP4 quantization checkpoint includes MTP head weights, the assistant could enable --speculative-algorithm NEXTN in SGLang, potentially doubling or tripling throughput. The key question is whether the quantization process preserved these MTP layers.

Hypothesis 2: GEMM backend tuning — The RTX PRO 6000 Blackwell GPUs have compute capability 12.0 (SM120), which differs from the B200/B100 GPUs (SM100/SM100) that most Blackwell optimization work targets. The user correctly notes that different SM counts and architectural details might mean the currently selected backend (flashinfer_cudnn for FP4, flashinfer_cutlass for MoE) is suboptimal. The question implicitly asks: are there SM120-specific kernel implementations that could provide better performance?

The assistant's "Good questions" acknowledgment reflects genuine recognition that these are precisely the right next steps. The deployment was correct, but correctness is only the foundation — performance is the goal.

Assumptions Made

Several assumptions underpin this message and the investigations that follow:

  1. The NVFP4 checkpoint might or might not contain MTP weights. The assistant does not assume either way. The config.json might have mtp_num_hidden_layers set, or it might not. The weight map might contain mtp.* keys, or the quantization process might have stripped them. The investigation is genuinely open-ended.
  2. SGLang's speculative decoding infrastructure supports this model. The assistant assumes that if MTP weights exist, SGLang can use them. This turns out to be correct — SGLang has a dedicated qwen3_5_mtp.py model class and auto-detects MTP-capable architectures by checking num_nextn_predict_layers in the config.
  3. The GEMM backend landscape has multiple candidates worth evaluating. The assistant assumes that flashinfer_cudnn (currently selected for FP4) and flashinfer_cutlass (for MoE) are not necessarily optimal, and that flashinfer_trtllm and flashinfer_cutedsl are viable alternatives worth testing. This assumption is validated by subsequent research showing that flashinfer's mm_fp4 function supports five backends: cudnn, trtllm, cutlass, cute-dsl, and auto.
  4. The investigation can proceed in parallel without conflicts. The MTP check and the GEMM backend research are independent — one does not depend on the other. This makes parallel investigation safe and efficient.

Input Knowledge Required

To fully understand this message, the reader needs:

  1. Knowledge of speculative decoding — Understanding that MTP (Multi-Token Prediction) is a form of speculative decoding where a model predicts multiple future tokens at once, and that this can significantly improve throughput by reducing the number of sequential autoregressive steps.
  2. Understanding of GEMM backends — Knowing that different kernel implementations (cuDNN, CUTLASS, TRT-LLM, CUTE-DSL) can have dramatically different performance characteristics depending on the GPU architecture, and that "backend tuning" means selecting the optimal implementation for the specific hardware.
  3. Awareness of the Blackwell architecture landscape — Understanding that NVIDIA's Blackwell generation includes multiple variants (B200 at SM100, RTX PRO 6000 at SM120) with different compute capabilities, and that optimizations targeting one may not apply to the other.
  4. Context about the preceding deployment — Knowing that the model is Qwen3.5-397B-A17B-NVFP4 running on 8× RTX PRO 6000 GPUs, that the current throughput is ~72 tok/s, and that the assistant has just finished getting the model serving correctly after significant build effort.
  5. Familiarity with SGLang's architecture — Understanding that SGLang supports multiple speculative decoding algorithms (EAGLE, EAGLE3, NEXTN, NGRAM) and that the --speculative-algorithm flag controls which one is used.

Output Knowledge Created

This message itself creates minimal output — it's a commitment to investigate rather than a finding. However, it sets in motion the creation of substantial knowledge:

  1. Discovery that the NVFP4 checkpoint does contain MTP weights — The config.json has mtp_num_hidden_layers: 1 and the weight map contains 1,553 MTP-related tensors including all 512 experts for the draft layer. This confirms that MTP speculation is viable.
  2. Discovery that SGLang auto-detects MTP — The model config code at line 330-334 of model_config.py automatically redirects Qwen3_5MoeForConditionalGeneration to Qwen3_5ForCausalLMMTP when is_draft_model=True, with num_nextn_predict_layers=1.
  3. Discovery that the MTP layer runs in BF16, not FP4 — The MTP model code explicitly states "The MTP model is unquantized in the nvfp4 checkpoint," which has implications for memory usage and kernel selection.
  4. Mapping of available GEMM backends — The research reveals that flashinfer's mm_fp4 supports cudnn, trtllm, cutlass, cute-dsl, and auto backends, and that flashinfer_trtllm MoE runner explicitly checks for SM120 support.

The Thinking Process Visible in the Message

While the message itself is brief, the thinking process is visible in its structure and timing. The assistant has just received the user's throughput concern and immediately:

  1. Validates the user's framing — "Good questions" signals that the user has correctly identified the two most impactful optimization paths. This is important because it establishes shared understanding and builds confidence in the investigation direction.
  2. Chooses parallel over sequential investigation — The phrase "Let me investigate both in parallel" reflects a conscious decision to minimize latency. In a session where every minute of investigation delays the next deployment iteration, parallel exploration is the right call.
  3. Formalizes the work items — The todowrite block transforms vague investigation goals into tracked, actionable items with clear status and priority. This is a pattern the assistant uses throughout the session to maintain coherence across dozens of steps. The subsequent messages reveal the actual investigation in action: bash commands to inspect config.json and model.safetensors.index.json for MTP keys, a task tool call spawning a subagent to research GEMM backends, and grep commands through SGLang source code to understand the speculative decoding infrastructure. The assistant's thinking is methodical and thorough — it doesn't just check whether MTP exists, but traces through the entire activation path from --speculative-algorithm NEXTN through server_args.py to model_config.py to qwen3_5_mtp.py.

Mistakes and Incorrect Assumptions

The message itself contains no factual errors — it's a commitment to investigate, not a claim of knowledge. However, one implicit assumption deserves scrutiny: the assumption that MTP speculation will actually improve throughput. In practice, speculative decoding's benefits depend heavily on the acceptance rate (how often the draft model's predictions are accepted by the main model) and the overhead of running both models. The subsequent investigation would reveal that MTP/NEXTN speculation showed no throughput gain on synthetic benchmarks ([chunk 39.0]), confirming that baseline performance was already optimal for this particular model and hardware combination.

Similarly, the assumption that a different GEMM backend would provide significant gains proved partially incorrect — the flashinfer_cudnn backend for FP4 and flashinfer_cutlass for MoE were already among the best options for SM120, with flashinfer_trtllm and flashinfer_cutedsl proving incompatible or producing NaN/garbage on this architecture.

Why This Message Matters

This message is the hinge point of the session. Before it, the focus was on correctness — getting the model to load, serve, and return accurate responses. After it, the focus shifts to performance — measuring throughput, testing backends, enabling speculative decoding, and ultimately achieving over 2100 tok/s at high concurrency. The message represents the moment when the assistant and user together decide that "working" is not enough, and that the real goal is "working fast."

The brevity of the message is itself a signal of confidence. The assistant doesn't need to explain why these are good questions, or debate alternative approaches, or ask for clarification. It immediately recognizes the path forward and commits to it. In a conversation spanning thousands of messages across multiple sessions, this single short message marks the transition from deployment to optimization — from "can we run it?" to "how fast can we run it?"