Probing the Depths of Blackwell Inference: Investigating MTP and GEMM Backends for Qwen3.5-397B on SM120

Introduction

In the high-stakes world of large language model deployment on cutting-edge hardware, a single number can trigger an entire investigation. When the assistant deployed Qwen3.5-397B-A17B-NVFP4 on an 8× RTX PRO 6000 Blackwell (SM120) system and measured approximately 72 tokens per second at single-request concurrency ([msg 5926]), the user's reaction was immediate and pointed: "Seems really low throughput." This critique launched a focused inquiry captured in message 5930, where the assistant pivoted from deployment completion to performance optimization, executing a multi-pronged investigation into two specific areas: whether the model checkpoint contained built-in Multi-Token Prediction (MTP) heads, and whether the GEMM backends selected for FP4 and MoE computation were optimal for the SM120 architecture.

Message 5930 is a dense, multi-threaded response that embodies the assistant's systematic investigative methodology. It contains three parallel bash commands interrogating the model's file structure and SGLang's configuration options, plus a spawned subagent task dedicated to deep research on SM120 GEMM backends. This message represents the critical transition from "is it working?" to "is it working well?" — a shift that separates basic deployment from production-grade optimization.

Context and Motivation: Why This Message Was Written

The immediate trigger was the user's throughput concern expressed in [msg 5928]. The user posed two specific questions: whether the NVFP4 quantized checkpoint included MTP (Multi-Token Prediction) heads despite being a quantized variant, and whether the GEMM backends could be tuned specifically for SM120 (the compute capability 12.0 architecture of the RTX PRO 6000 Blackwell) as opposed to the more common SM100 (B200) architecture. These were not idle questions — they reflected deep domain knowledge about how speculative decoding and kernel selection interact with model throughput.

The assistant's preceding message ([msg 5929]) acknowledged these questions and set up tracking todos, but message 5930 is where the actual investigation begins. The assistant recognized that answering these questions required parallel, independent lines of inquiry: one examining the model checkpoint's actual weight files and configuration, and another researching the SGLang codebase's backend selection logic for SM120. The structure of the message — three bash commands executed in parallel alongside a spawned subagent task — reflects a deliberate strategy to minimize latency by dispatching independent investigations simultaneously.

The broader context is crucial. This session (Segment 39) had just completed a grueling upgrade process: upgrading PyTorch to nightly 2.12.0+cu130, building sgl-kernel from source with SM120 FP4 support after applying patches for CMake policy guards, CUDA 13 cccl include paths, and FA3 fallback, and exhaustively testing backends to find a configuration that produced correct output on SM120 ([chunk 39.0]). The assistant had already discovered that flashinfer_trtllm and flashinfer_cutedsl backends crashed or produced NaN/garbage, while flashinfer_cutlass for MoE and flashinfer_cudnn for FP4 GEMM worked correctly. The current configuration — --attention-backend triton --moe-runner-backend flashinfer_cutlass --fp4-gemm-backend flashinfer_cudnn — was the correct configuration, but the user was now asking whether it was the fastest configuration.

The Investigation Unfolds

The message opens with a bash command listing all JSON configuration files in the model directory. This is a reconnaissance step — the assistant needs to understand what configuration artifacts exist before diving into their contents. The listing reveals the standard set of HuggingFace model files: config.json, generation_config.json, hf_quant_config.json, model.safetensors.index.json, tokenizer files, and processor configuration. The presence of hf_quant_config.json confirms this is a HuggingFace-quantized model, and model.safetensors.index.json will be critical for understanding the weight layout.

The second command reads config.json and pipes it through python3 -m json.tool for pretty-printing, showing the first 80 lines. This reveals the model architecture: Qwen3_5MoeForConditionalGeneration, a Mixture-of-Experts model with dtype: "bfloat16", hidden_size: 4096, head_dim: 256, and full_attention_interval: 4. The attn_output_gate: true flag is notable — it indicates an attention output gating mechanism that is characteristic of Qwen3.5's architecture. The eos_token_id: 248044 and image_token_id: 248056 confirm this is a multimodal model, though the deployment is focused on text generation.

The third command queries SGLang's --help output for speculative decoding flags, filtering with a regex for mtp|multi.token|speculative|draft|num-speculative. This reveals the available speculative algorithms: EAGLE, EAGLE3, NEXTN, STANDALONE, and NGRAM. The presence of NEXTN is significant — this is SGLang's built-in Multi-Token Prediction mechanism, distinct from the separate EAGLE family of draft-model-based speculation. The command also reveals flags for --speculative-draft-model-path and --speculative-num... (truncated in output), confirming that SGLang supports both external draft models and built-in MTP.

The Subagent Task: Deep Research on SM120 GEMM Backends

The most substantial component of message 5930 is the spawned subagent task, dispatched via the task tool with the description "Research SM120 GEMM backends." This is a critical architectural decision: rather than exploring the codebase manually through bash commands (which would be slow and sequential), the assistant delegates the research to a subagent that can run its own multi-round conversation, read source files, analyze code, and return a comprehensive summary.

The task prompt asks the subagent to investigate:

  1. What GEMM backends are available for --fp4-gemm-backend and --moe-runner-backend in the latest SGLang
  2. Which backends support SM120 (compute capability 12.0) vs SM100
  3. Performance characteristics and recommendations The task result, summarized at the end of the message, reveals a critical architectural distinction in SGLang's codebase: the framework distinguishes three Blackwell tiers based on compute capability major version. The is_sm100_supported() function checks for major version 10 only (B200, B100, GB200), while SM120 (major version 12) is treated differently. This distinction has profound implications for backend selection — backends optimized for SM100 may not be available or optimal on SM120, and vice versa.

Assumptions and Knowledge Requirements

Several assumptions underpin this message. First, the assistant assumes that MTP support, if present in the model, would manifest as weight tensors with "mtp" in their names and a corresponding mtp_num_hidden_layers config field. This is a reasonable assumption based on the Qwen3.5 architecture documentation and SGLang's model implementation patterns. Second, the assistant assumes that GEMM backend selection is a significant lever for throughput optimization — an assumption validated by the earlier experience where incorrect backends produced NaN outputs, demonstrating that backend choice directly affects both correctness and performance.

The input knowledge required to understand this message is substantial. One must understand the HuggingFace model file structure (config.json, safetensors.index.json, weight maps), the Qwen3.5 MoE architecture (hidden_size, head_dim, attention gating), SGLang's speculative decoding framework (EAGLE vs NEXTN vs NGRAM), the Blackwell GPU compute capability numbering (SM100 = major 10, SM120 = major 12), and the concept of GEMM backends as swappable kernel implementations for matrix multiplication. Without this background, the message's significance is lost.

The output knowledge created is equally significant. The message confirms that the NVFP4 checkpoint does contain MTP weights — 1,553 weight tensors under the mtp.* namespace, including a full set of expert weights for the MTP layer's MoE submodule. The config reveals mtp_num_hidden_layers: 1 and mtp_use_dedicated_embeddings: false, indicating a single MTP prediction layer that reuses the base model's embeddings. The subagent's research establishes that SM120 requires careful backend selection distinct from SM100, setting the stage for further optimization work in subsequent messages.

The Thinking Process Visible in the Message

While the message does not contain explicit reasoning blocks, the thinking process is visible in its structure and sequencing. The assistant recognized that the user's two questions were independent and could be investigated in parallel. The three bash commands are dispatched simultaneously (they appear in the same message, meaning they were issued in the same round), followed by the subagent task. This parallelism demonstrates an understanding of the investigation's dependency graph: none of these inquiries depend on each other, so they can proceed concurrently.

The choice to use a subagent task for the GEMM backend research is particularly telling. The assistant could have explored the SGLang source code manually with grep and cat commands, but that would have required multiple sequential rounds. By delegating to a subagent, the assistant effectively parallelizes the research across two agents — the main agent handles the model inspection while the subagent dives into the codebase. This is a sophisticated use of the tooling available in the opencode environment, leveraging subagent spawning as a form of computational parallelism.

Conclusion

Message 5930 represents a pivotal moment in the deployment of Qwen3.5-397B-A17B-NVFP4 on Blackwell hardware. It transitions the conversation from "does it work?" to "how fast can it go?" — a transition that requires deep investigation into both the model's capabilities (MTP heads) and the runtime's configuration (GEMM backends). The message's structure — parallel bash commands interrogating model files and SGLang configuration, coupled with a spawned subagent for codebase research — exemplifies a systematic, parallelized approach to performance optimization. The discoveries within this message — that MTP weights are present in the quantized checkpoint, that SGLang's NEXTN algorithm supports built-in speculative decoding, and that SM120 requires distinct backend considerations from SM100 — directly inform the optimization work that follows, ultimately leading to the production configuration achieving ~172 tok/s single-request and over 2100 tok/s at high concurrency.