Discovery of Built-in MTP Heads in Qwen3.5-397B-A17B-NVFP4

In the high-stakes world of large language model deployment, every token per second counts. When the user running an 8× RTX PRO 6000 Blackwell server observed that their freshly deployed Qwen3.5-397B-A17B-NVFP4 model was achieving only ~72 tok/s single-request throughput, they immediately asked two pointed questions: "Is there built-in MTP? Qwen supports but is it in the quant?" and "can we tune gemms for SM120?" This article examines the assistant's response in message 5931, which set out to answer the first question by inspecting the model checkpoint directly.

The Context: A Production Deployment Under Scrutiny

The conversation leading up to this message had been an intense, multi-session effort to deploy cutting-edge language models on a bleeding-edge hardware stack. The assistant had just completed a full stack upgrade—PyTorch nightly (2.12.0.dev20260307+cu130), flashinfer 0.6.5, a custom-built sgl-kernel with SM120 FP4 support, and the latest SGLang main branch. Smoke tests confirmed the model was producing correct output: "OK" for simple completions, "323" for arithmetic, and proper chain-of-thought reasoning in thinking mode. But the user was underwhelmed by the throughput, suspecting that the model's built-in Multi-Token Prediction (MTP) capability—a speculative decoding technique where the model predicts multiple future tokens in a single forward pass—might be lying dormant.

The user's question was precise: "Is there built-in MTP? Qwen supports but is it in the quant?" This distinction matters enormously. The Qwen3.5 architecture natively supports MTP heads, but a quantized checkpoint (NVFP4 in this case) might have discarded those weights during the quantization process. The assistant needed to determine whether the MTP weights survived quantization and were present in the deployed model files.

The Message: Probing the Checkpoint

Message 5931 contains three tool calls executed in parallel, each targeting a different aspect of the model's configuration and weight structure. The first is a straightforward grep of the model's config.json for MTP-related fields:

[bash] ssh root@10.1.230.174 'cat /data/models/Qwen3.5-397B-A17B-NVFP4/config.json | python3 -m json.tool 2>&1 | grep -iE "mtp|num_nextn|nextn|multi_token|num_draft|speculative|predict"'

The second command dumps the tail of the formatted config JSON, presumably to see the full weight map listing and confirm the structure:

[bash] ssh root@10.1.230.174 'cat /data/models/Qwen3.5-397B-A17B-NVFP4/config.json | python3 -m json.tool 2>&1' | tail -80

The third is a more sophisticated Python script that loads the model.safetensors.index.json file—the index that maps weight tensor names to shard files—and searches for any keys containing "mtp", "draft", "nextn", or "speculative". It also prints the top-level module prefixes to give a structural overview of the model's components:

[bash] ssh root@10.1.230.174 'python3 -c "
import json
with open(\"/data/models/Qwen3.5-397B-A17B-NVFP4/model.safetensors.index.json\") as f:
    idx = json.load(f)
wm = idx.get(\"weight_map\", {})
mtp_keys = [k for k in wm if \"mtp\" in k.lower() or \"draft\" in k.lower() or \"nextn\" in k.lower() or \"speculative\" in k.lower()]
print(f\"MTP-related keys ({len(mtp_keys)}): {mtp_keys[:20]}\")
all_prefixes = set()
for k in wm:
    parts = k.split(\".\")
    if len(parts) >= 2:
        all_prefixes.add(\".\".join(parts[:2]))
print(f\"Top-level module prefixes: {sorted(all_prefixes)[:30]}\")
"'

The Results: A Definitive Discovery

The output from these commands was unambiguous and exciting. The config.json grep returned:

"mtp_num_hidden_layers": 1,
"mtp_use_dedicated_embeddings": false,
"mtp.layers.0*"

This confirmed that the model configuration explicitly declares a single MTP hidden layer, and that the weight map includes entries under mtp.layers.0. The mtp_use_dedicated_embeddings: false flag indicates that the MTP head shares the input embeddings with the main model rather than having its own separate embedding table—a design choice that reduces parameter count while still enabling multi-token prediction.

The Python script's output was even more striking:

MTP-related keys (1553): ['mtp.fc.weight', 'mtp.layers.0.input_layernorm.weight', 'mtp.layers.0.mlp.experts.0.down_proj.weight', 'mtp.layers.0.mlp.experts.0.gate_proj.weight', 'mtp.layers.0.mlp.experts.0.up_proj.weight', 'mtp.layers.0.mlp.experts.1.down_proj.weight', ...]

1553 MTP-related weight tensors. This is a substantial model component, not a token gesture. The listing reveals that the MTP head includes a full feed-forward network (mtp.fc.weight), layer normalization (input_layernorm.weight), and a complete MoE (Mixture of Experts) sub-layer with multiple experts, each having down_proj, gate_proj, and up_proj weight matrices. The experts are indexed from 0 through at least 10 in the sample, suggesting the MTP head has its own dedicated expert pool—a non-trivial architectural investment.

Why This Matters: The Reasoning and Motivation

The assistant's motivation for running these specific probes was rooted in a pragmatic performance optimization goal. Speculative decoding techniques like MTP can dramatically increase throughput by generating multiple tokens per forward pass, then verifying them in parallel. For a 397-billion-parameter MoE model, even a modest speculation budget of 1–2 extra tokens could yield 30–50% throughput improvements—turning that "low" 72 tok/s into something much more competitive.

But speculative decoding only works if the model has been trained with the appropriate heads. Many quantized checkpoints strip auxiliary components to save space, so the assistant could not assume the MTP weights were present. The user's question "is it in the quant?" reflected exactly this concern. By probing both the config (which declares architectural intent) and the actual weight index (which proves physical presence), the assistant was building a complete, evidence-based answer.

The choice to run three commands in parallel rather than sequentially reflects the assistant's understanding that these are independent probes—none depends on the output of another—and that the user was waiting for actionable information. This parallel execution pattern is characteristic of the assistant's operating model, where multiple tool calls in a single round are dispatched simultaneously and their results collected before the next reasoning step.

Assumptions and Knowledge Requirements

To fully understand this message, one needs significant background knowledge. First, familiarity with the SGLang serving framework and its speculative decoding architecture is essential—specifically the distinction between EAGLE, EAGLE3, NEXTN, and other algorithms. The assistant had already confirmed in message 5930 that --speculative-algorithm NEXTN was a supported option, and that --speculative-draft-model-path existed for external draft models. The discovery here was that this model didn't need an external draft model—it had MTP heads baked in.

Second, understanding the NVFP4 quantization format is important. ModelOpt FP4 is NVIDIA's 4-bit floating-point quantization scheme, and the question of whether auxiliary components like MTP heads survive quantization is non-trivial. The quantization process could theoretically discard or merge these weights, so their presence in the safetensors index is a meaningful finding.

Third, knowledge of the Qwen3.5 architecture helps interpret the results. The model uses a Mixture of Experts design with 512 routed experts and shared experts, and the MTP head mirrors this structure with its own expert pool. The mtp_num_hidden_layers: 1 indicates a single-layer MTP head (predicting one future token), which is the standard configuration for this model family.

The assistant made a reasonable assumption that the config.json and safetensors index would be present and well-formed in the checkpoint directory. This assumption proved correct, but it's worth noting that some quantized checkpoints reorganize or omit these files. The assistant also assumed that the weight map key naming convention would use "mtp" as the prefix, which was confirmed by the grep results.

Output Knowledge Created

This message produced several concrete pieces of knowledge:

  1. MTP is architecturally present: The config.json declares mtp_num_hidden_layers: 1, confirming the model was designed with multi-token prediction capability.
  2. MTP weights survived quantization: 1553 weight tensors under the mtp.* namespace are present in the safetensors index, proving the NVFP4 quantization process preserved the MTP heads.
  3. MTP head structure: The head includes a fully-connected layer (mtp.fc.weight), layer normalization, and a complete MoE sub-layer with multiple experts—a substantial computational component, not a lightweight addition.
  4. No dedicated embeddings: The mtp_use_dedicated_embeddings: false flag means the MTP head shares embeddings with the base model, which has implications for memory usage and potential fine-tuning.
  5. Top-level module structure: The model's module prefixes were enumerated, providing a map of the model's organizational hierarchy for anyone needing to inspect or modify specific components. This knowledge immediately informed the next steps in the conversation. In the following message (msg 5932), the assistant discovered that SGLang has a dedicated qwen3_5_mtp.py model file, confirming that the framework already supports loading and running this exact MTP configuration. The path was clear: enable --speculative-algorithm NEXTN and let the model use its built-in MTP heads to boost throughput.

The Thinking Process Visible in the Message

While the message itself is a straightforward sequence of tool calls, the reasoning behind it reveals a systematic investigative approach. The assistant is operating on a hypothesis: "The model might have MTP heads that we're not using." To test this hypothesis, it needs evidence from two independent sources—the configuration (what the model claims to have) and the weights (what the model actually has). This dual-source verification is a hallmark of rigorous debugging.

The grep patterns chosen are also revealing. The assistant searched for "mtp", "num_nextn", "nextn", "multi_token", "num_draft", "speculative", and "predict"—covering the terminology used across different model families and speculative decoding implementations. This breadth suggests the assistant wasn't certain which naming convention Qwen3.5 would use, and was hedging against multiple possibilities.

The Python script's design shows careful attention to the data structure. It loads the safetensors index, extracts the weight map, filters for MTP-related keys, counts them, and prints a sample. It also computes top-level module prefixes to provide structural context. This is not a quick-and-dirty check but a deliberate, structured exploration designed to produce comprehensive, interpretable output.

Conclusion

Message 5931 represents a pivotal discovery in the optimization journey. What began as a user's hunch—"is MTP in the quant?"—was transformed into concrete evidence through systematic probing of the model checkpoint. The discovery of 1553 MTP weight tensors and a declared mtp_num_hidden_layers: 1 in the configuration opened the door to speculative decoding optimizations that could dramatically improve throughput on the 8× RTX PRO 6000 Blackwell system. The assistant's methodical approach—parallel independent probes, dual-source verification, and careful pattern matching—turned an uncertain question into actionable knowledge, setting the stage for the NEXTN speculative decoding implementation that would follow in subsequent messages.