The Recipe Hunt: Decoding DeepSeek-V4's MTP Configuration Through Test-Driven Discovery

[assistant] [bash] timeout 30 ssh -o StrictHostKeyChecking=no root@10.1.230.171 'sed -n "78,118p" /root/sglang-dsv4/test/registered/amd/test_deepseek_v4_pro_fp4_mtp.py'
        env.update(FP4_ENV_VARS)

        other_args = [
            "--trust-remote-code",
            "--tp",
            "8",
            "--disable-radix-cache",
            "--attention-backend",
            "dsv4",
            # MTP / EAGLE speculative decoding (NextN head from the base model).
            "--speculative-algorithm",
            "EAGLE",
            "--speculative-num-steps",
            "3",
            "--speculative-eagle-topk",
            "1",
            "--speculativ...

Introduction

In the middle of a grueling optimization campaign to wring acceptable throughput out of DeepSeek-V4-Flash on 8× NVIDIA RTX PRO 6000 Blackwell GPUs, the assistant pauses the performance tuning to hunt for a recipe. Message [msg 12465] captures a deceptively simple moment: a single SSH command that reads lines 78 through 118 of a test file. Yet this act of reading is the culmination of a search that spans four preceding messages, and it represents a critical juncture in the deployment workflow. The assistant is not merely browsing code—it is reverse-engineering the correct launch configuration for Multi-Token Prediction (MTP) speculative decoding from the official test suite, because the documentation is ambiguous and the stakes are high. A wrong flag could silently disable the feature, waste hours of benchmarking, or crash the service entirely.

The Context: A Performance Campaign at an Impasse

To understand why this message exists, one must appreciate the broader narrative. The assistant has been deploying DeepSeek-V4-Flash—a massive Mixture-of-Experts model with FP4 quantization—across eight RTX PRO 6000 GPUs (sm_120 architecture). The headline deliverable of prefill-decode disaggregation has been achieved, but throughput languishes at roughly 28 tok/s at concurrency 16, against a user target of ~1000 tok/s. Every configuration lever has been pulled: NCCL LL+Ring tuning, CUDA graphs, tilelang indexer fusion, non-Marlin MoE backends, expert parallelism—none moved the needle significantly.

The previous chunk's analysis revealed the root cause: the _tiled_sparse_decode_kernel, a Triton fallback for sparse MLA attention that launches only 64 blocks on ~170 SMs, serially iterating all 512 top-k tokens. This is the same low-occupancy pathology that plagued the earlier Kimi K2.6 deployment. The assistant has been systematically working through a todo list: pull the latest upstream SGLang commits, generate SM120 FP8 GEMM autotune configs, confirm NCCL=LL is active, and enable MTP/EAGLE speculative decoding—the one lever that showed a 47% single-request throughput improvement in earlier testing.

But enabling MTP is not a simple boolean flag. DeepSeek-V4's architecture includes a built-in NextN (MTP) head—a lightweight prediction layer that generates multiple tokens per forward pass. SGLang's EAGLE speculative decoding framework can leverage this head, but the exact combination of flags is model-specific and poorly documented. The assistant needs the recipe.

The Search: Four Messages of Dead Ends

The path to [msg 12465] is a story of narrowing searches. In [msg 12462], the assistant reads the deepseek_v4_hook.py file—the SGLang module that applies model-specific defaults for DeepSeek-V4. The hook reveals constraints: it requires EAGLE algorithm with eagle_topk == 1. But it does not specify the draft model path, the number of steps, or the number of draft tokens. These remain open questions.

In [msg 12463], the assistant searches the registered test directory for MTP recipes, grepping for patterns like speculative-algorithm, speculative-draft-model-path, and num-steps. The results are noisy—they surface EAGLE3 configurations for other models (GPT-OSS-120B, Mistral-Large-3) but nothing specific to DeepSeek-V4. The assistant hits a dead end.

In [msg 12464], the assistant narrows the search to the single relevant test file: test_deepseek_v4_pro_fp4_mtp.py. A grep for speculative-related flags returns partial results—enough to see that other_args contains the MTP configuration, but not enough to see the full argument list. The output is truncated.

This brings us to [msg 12465]. The assistant now reads the exact lines (78–118) of the test file using sed, extracting the complete other_args block that defines the MTP launch configuration.

What the Message Reveals

The output shows the beginning of the other_args list used in the official AMD ROCm test for DeepSeek-V4-Pro FP4 with MTP:

other_args = [
    "--trust-remote-code",
    "--tp",
    "8",
    "--disable-radix-cache",
    "--attention-backend",
    "dsv4",
    # MTP / EAGLE speculative decoding (NextN head from the base model).
    "--speculative-algorithm",
    "EAGLE",
    "--speculative-num-steps",
    "3",
    "--speculative-eagle-topk",
    "1",
    "--speculativ...

The output is truncated mid-flag at "--speculativ...", so the assistant does not get the complete picture from this single command. The remaining flags—likely --speculative-num-draft-tokens and possibly --speculative-draft-model-path—are cut off by the sed range ending at line 118. This truncation is itself significant: it forces the assistant to either extend the line range or piece together the remaining flags from the earlier grep in [msg 12464], which showed "--speculative-num-draft-tokens", "2".

Input Knowledge Required

To fully understand this message, the reader needs several layers of context. First, knowledge of SGLang's architecture: that it supports speculative decoding via the EAGLE algorithm, that MTP (Multi-Token Prediction) is a specific form of self-speculation where the draft head is embedded in the base model checkpoint, and that the --speculative-draft-model-path flag is typically required to point to a separate draft model—but for DeepSeek-V4's NextN head, the draft weights live inside the main checkpoint, making this flag's necessity ambiguous.

Second, familiarity with the DeepSeek-V4 model architecture: it includes a built-in NextN prediction head (sometimes called MTP or "Multi-Token Prediction") that can generate multiple tokens per forward pass without a separate draft model. This is distinct from EAGLE-2 or EAGLE-3, which use an external draft model.

Third, understanding of the test infrastructure: the file test_deepseek_v4_pro_fp4_mtp.py lives under test/registered/amd/, indicating it is an AMD ROCm-specific test for the MI350-series GPUs. The assistant is reading an AMD test to derive a configuration for NVIDIA Blackwell GPUs—a cross-platform inference that assumes the launch arguments are architecture-agnostic at the Python level.

Fourth, awareness of the ongoing FP8 autotuning: the assistant launched the tuning script in [msg 12461] and is monitoring it in the background while simultaneously preparing the MTP configuration. This is a multi-tasking optimization campaign where the assistant works on parallel workstreams.

Output Knowledge Created

This message produces several concrete pieces of knowledge:

  1. The exact MTP flags for DeepSeek-V4: --speculative-algorithm EAGLE, --speculative-num-steps 3, --speculative-eagle-topk 1. These are confirmed by the official test suite.
  2. The base launch configuration: --trust-remote-code, --tp 8, --disable-radix-cache, --attention-backend dsv4. The TP8 configuration confirms the model uses all 8 GPUs for tensor parallelism, and --disable-radix-cache is notable—it suggests the test avoids the radix cache for reproducibility, which may differ from production settings.
  3. The absence of --speculative-draft-model-path: The visible portion of the other_args list does not include this flag. Combined with the earlier grep result showing that the test references {DRAFT_MODEL_PATH} only for other model families (GPT-OSS, Mistral), this strongly suggests that DeepSeek-V4's MTP does not require a separate draft model path—the NextN head is loaded automatically from the main checkpoint.
  4. The truncation boundary: The output cuts off at "--speculativ...", revealing that the remaining flags are on lines beyond 118. This tells the assistant to extend the read range to capture the full argument list.

Assumptions and Their Risks

The assistant makes several assumptions in this message, some of which carry risk:

Assumption 1: The AMD test configuration is valid for NVIDIA hardware. The test file lives under test/registered/amd/ and is designed for AMD ROCm MI350-series GPUs. The assistant assumes that the SGLang launch arguments are platform-agnostic—that --speculative-algorithm EAGLE means the same thing on AMD and NVIDIA. This is likely true for the Python-level flags, but there could be AMD-specific patches or environment variables (like the FP4_ENV_VARS referenced in the test) that differ on NVIDIA. The assistant already has its own FP4_ENV_VARS from the earlier deployment, but any subtle differences could affect behavior.

Assumption 2: The test configuration represents optimal production settings. The test uses --disable-radix-cache, which disables the prefix caching that improves throughput for shared prefixes. The assistant may need to re-enable this for production. The test also uses TP8, but the assistant's earlier deployment used TP4 with PD disaggregation—a different parallelism strategy. The MTP flags may interact differently with TP4 versus TP8.

Assumption 3: The NextN head is auto-detected. By not specifying --speculative-draft-model-path, the assistant assumes SGLang will automatically find and load the MTP head from the DeepSeek-V4 checkpoint. This is the behavior suggested by the test, but it is not explicitly documented. If the auto-detection fails, the EAGLE algorithm may silently fall back to autoregressive decoding, wasting the optimization effort.

Assumption 4: The sed range of 78–118 captures the complete MTP configuration. It does not—the output is truncated. The assistant must recognize this and extend the range in a follow-up message.

The Thinking Process: Methodical Reverse-Engineering

The reasoning visible in the preceding messages reveals a methodical, almost forensic approach. The assistant does not guess the MTP flags or rely on documentation. Instead, it:

  1. Reads the hook file ([msg 12462]) to understand the model-specific constraints—learning that EAGLE with top-k 1 is required.
  2. Searches broadly ([msg 12463]) for any mention of DeepSeek-V4 MTP in the test suite, getting noisy results from other model families.
  3. Narrows to the specific test ([msg 12464]) by grepping the AMD DSv4 MTP test file, getting partial results.
  4. Reads the exact lines ([msg 12465]) to extract the full other_args block. This progression—from broad search to narrow extraction—mirrors how a skilled engineer would navigate an unfamiliar codebase. The assistant treats the test suite as ground truth: if the test passes with these flags, then these flags are correct. This is a pragmatic choice in the absence of clear documentation. The truncation at line 118 is an artifact of the sed command's line range. The assistant chose 78,118p based on the earlier grep output showing that the other_args block started around line 80. The 40-line window was a reasonable estimate, but it fell short. This is not a mistake—it is an iterative refinement. The assistant will likely follow up with a broader range (e.g., 78,130p) to capture the complete flag list.

Why This Message Matters

In the grand narrative of the optimization campaign, [msg 12465] is a quiet but pivotal moment. The assistant has exhausted configuration tuning, kernel optimization, and parallelism strategies. MTP speculative decoding represents one of the few remaining levers with demonstrated impact (47% single-request improvement). Getting the flags right is essential—a single wrong flag could silently disable the feature and waste hours of benchmarking.

The message also illustrates a broader theme of the session: the tension between upstream documentation and ground-truth test code. The assistant repeatedly turns to the test suite as the authoritative source of configuration recipes, bypassing incomplete or ambiguous documentation. This is a survival strategy in a rapidly evolving open-source ecosystem where features land faster than their documentation.

Moreover, the message captures the assistant working at two levels simultaneously. At the tactical level, it is reading a file. At the strategic level, it is orchestrating a multi-threaded optimization campaign: FP8 tuning runs in the background, NCCL configuration is verified, MTP flags are being assembled, and the next benchmarking round is being planned. The sed command is one thread in a larger weave.

Conclusion

Message [msg 12465] is a testament to the importance of reading the source. In a complex deployment involving bleeding-edge hardware (Blackwell sm_120), a cutting-edge model (DeepSeek-V4-Flash), and a rapidly evolving inference engine (SGLang main), the assistant cannot rely on documentation alone. It must reverse-engineer the correct configuration from the test suite—the only artifact that is guaranteed to reflect working behavior.

The output is incomplete, but that is by design. The assistant has learned enough to proceed: EAGLE algorithm, 3 speculative steps, top-k of 1, no explicit draft model path. The remaining flags can be inferred from the earlier grep or captured in a follow-up read. The recipe hunt is nearly complete, and the next phase—benchmarking MTP throughput on the Blackwell GPUs—can begin.

In the end, this message is about knowledge transfer: from the AMD test suite to the NVIDIA deployment, from the SGLang developers to the assistant, and from the assistant's reasoning to the reader. It is a small but essential step in the long march toward 1000 tok/s.