Tracing the Double-Scaling Risk: A Precision Diagnostic in DeepSeek V4's MoE Routing

Introduction

In the high-stakes world of deploying large language models on cutting-edge hardware, the line between an optimization and a bug can be vanishingly thin. Message <msg id=12862> captures a critical moment in a systematic debugging campaign: the assistant has just finished reading the source code of a patched MoE routing module and is now executing a targeted grep to determine whether a speed optimization has introduced a subtle numerical error that could silently corrupt every token the model generates. This message is a hinge point—the moment where hypothesis meets evidence, and where the assistant's reasoning about a potential "double-scaling" bug in the routed expert weights is put to the test.

The broader context is a production deployment of DeepSeek V4 (the GLM-5-NVFP4 model) on eight NVIDIA RTX PRO 6000 Blackwell GPUs, running a heavily customized fork of SGLang. The team has applied eight distinct speed patches—ranging from custom MMA attention kernels to bfloat16 tensor-core operations—to squeeze maximum throughput from the hardware. But a coherence bug has emerged: the model loses context on longer multi-turn prompts, failing to recall facts planted earlier in the conversation. The user's preceding message (<msg id=12854>) laid out a meticulous risk assessment of every patch, ranking the MoE routed-scaling modification as a Tier 1 suspect alongside the MHC bf16 GEMM change. The core fear is that the scaling factor—a multiplicative constant around 2.5 that adjusts the routed experts' contribution—might be applied twice, once in the patched HashTopK module and again downstream in the MoE forward pass, causing every routed expert output to be mis-scaled.

The Message: Reasoning and Execution

The subject message contains two distinct layers: an internal reasoning section where the assistant synthesizes what it has just learned, followed by a concrete diagnostic action. Here is the message in full:

Agent Reasoning So HashTopK uses a fixed lookup table that maps token IDs directly to expert IDs—when the shared experts fusion is disabled, it bypasses the typical selection logic and just uses this precomputed mapping to assign tokens to experts.

>

The weights are then gathered from the scores at those hashed positions and normalized to sum to 1. If apply_routed_scaling_factor_on_output is enabled, these weights get multiplied by the routed scaling factor. The critical question is whether this scaling happens here or downstream in the main MoE forward pass—if it happens in both places, the routed contribution gets scaled twice, which would be incorrect. The stock code had an assertion preventing this flag, suggesting the scaling was meant to happen elsewhere, but our patch enables it here, so I need to verify whether the downstream code still applies the scaling. Now let me trace routed_scaling_factor end-to-end: where HashTopK is instantiated, what apply_routed_scaling_factor_on_output is set to, and whether the factor is ALSO applied downstream in the MoE forward (double-application risk).

>

[bash] cd /tmp/opencode echo "=== deepseek_v4.py: routed_scaling_factor occurrences ===" grep -n "routed_scaling_factor\|apply_routed_scaling_factor_on_output\|HashTopK\|hash_topk\|final_hidden\|routed_scaling" deepseek_v4.py echo "" echo "=== deepseek_v4.py: TopK / select_experts / get_moe_impl / topk instantiation ===" grep -n "TopK\|select_experts\|topk_output\|self.topk\|=.*TopK(\|scaling" deepseek_v4.py | head -60

>

=== deepseek_v4.py: routed_scaling_factor occurrences ===

>

=== deepseek_v4.py: TopK / select_experts / get_moe_impl / topk instantiation === 307: rope_theta, rope_scaling = get_rope_config(config) 308: if rope_scaling: 309: rope_scaling["rope_type"] = "deepseek_yarn" 318: rope_scaling=rope_scaling, 327: original_seq_len = rope_scaling["original_max_position_embeddings"] 336: factor=rope_scaling["factor"], 337: beta_fast=rope_sc...

The reasoning section reveals the assistant's mental model of the HashTopK module. It has just read the full source of hash_topk.py (in the preceding message <msg id=12861>) and the diff showing the patch that replaced a assert not apply_routed_scaling_factor_on_output, "not implemented" with actual scaling logic. The assistant now understands the architecture: HashTopK is a specialized router that uses a deterministic hash to assign tokens to experts, bypassing the learned gating network. When shared experts fusion is disabled, it falls back to a precomputed token-to-expert mapping (tid2eid), gathers scores at those positions, normalizes them to sum to 1, and—if the flag is set—multiplies by the routed scaling factor.

The critical insight is the stock code's assertion: the original SGLang developers explicitly prevented apply_routed_scaling_factor_on_output from being enabled, with the comment "not implemented." This strongly suggests that the scaling was intended to be handled elsewhere—likely in the main MoE forward pass in deepseek_v4.py or the underlying deepseek_v2.py model code. By removing this assertion and implementing the scaling in HashTopK, the patch may have introduced a double-application: the factor gets multiplied into the weights here, and then the downstream code (which was written assuming HashTopK does NOT apply it) applies it again.

Why This Message Matters

This message is the execution of Step Q2 from the user's diagnostic plan (<msg id=12854>): "On the triton MoE runner, is apply_routed_scaling_factor_on_output=True, and is routed_scaling_factor applied exactly once end-to-end?" The assistant has already answered Q1 (confirmed the Triton indexer path is live via environment variable inspection in <msg id=12857>), and is now working through Q2 and Q3 in parallel.

The grep command is elegantly targeted. It searches for five key terms in deepseek_v4.py:

  1. routed_scaling_factor — to find any direct references to the scaling factor
  2. apply_routed_scaling_factor_on_output — to find where the flag is set
  3. HashTopK — to find where the module is instantiated
  4. hash_topk — to find import or usage sites
  5. final_hidden — a related term from the MoE output path
  6. routed_scaling — a broader catch-all The second grep searches for the MoE routing infrastructure: TopK, select_experts, get_moe_impl, topk_output, self.topk, and scaling. This is designed to locate the exact line where the top-k router is instantiated and where the MoE forward pass lives. The output is revealing—and surprising. The first grep returns empty: deepseek_v4.py contains no direct references to routed_scaling_factor, apply_routed_scaling_factor_on_output, HashTopK, hash_topk, or final_hidden. This is a significant finding. It means the MoE architecture is not defined inline in deepseek_v4.py but is instead inherited or imported from a shared module—most likely deepseek_v2.py, as the assistant discovers in the very next message (<msg id=12863>). The second grep confirms this: the only "scaling" references found are to rope_scaling (the rotary position embedding configuration), which is entirely unrelated to the MoE routing concern.

Input and Output Knowledge

Input knowledge required to understand this message includes:

Assumptions and Potential Mistakes

The assistant makes several assumptions in this message. First, it assumes that if the routed scaling factor is applied downstream, it would be found in deepseek_v4.py itself. This turns out to be incorrect—the factor is handled in the shared deepseek_v2.py module, which deepseek_v4.py inherits from or imports. This is not a mistake per se; it's a reasonable first hypothesis that is quickly corrected by the evidence.

Second, the assistant assumes that the stock code's assertion (assert not apply_routed_scaling_factor_on_output, "not implemented") is a reliable indicator of intent—that the scaling was "meant to happen elsewhere." This is a logical inference, but it's worth noting that the assertion could also reflect incomplete implementation rather than deliberate design. The original developer may have simply not gotten around to implementing the scaling in either location, and the downstream code might also be missing it. The assistant's reasoning is sound, but it's an assumption that requires verification.

Third, the assistant implicitly assumes that the grep patterns are comprehensive enough to catch all relevant references. The choice of search terms is good, but there's always a risk of missing an indirect reference—for example, if the scaling factor is applied through a lambda, a closure, or a dynamically constructed function. In practice, the follow-up search in deepseek_v2.py (in <msg id=12863>) confirms that the assistant's approach was sufficient.

The Thinking Process

The reasoning section reveals a clear, methodical thought process. The assistant has just finished reading the hash_topk.py source and the diff, and it synthesizes the key facts:

  1. Architecture: HashTopK uses a fixed lookup table (tid2eid) to map token IDs to expert IDs, bypassing learned gating when shared experts fusion is disabled.
  2. Weight computation: Scores are gathered from the hashed positions and normalized to sum to 1.
  3. The scaling gate: If apply_routed_scaling_factor_on_output is enabled, the normalized weights are multiplied by the routed scaling factor.
  4. The risk: If this scaling happens in HashTopK AND also downstream in the MoE forward pass, the routed contribution is scaled twice, producing incorrect outputs.
  5. The evidence: The stock code had an assertion preventing this flag from being set, strongly suggesting the scaling was intended to happen downstream.
  6. The action: Trace end-to-end to find all application sites. The assistant then executes the grep, interpreting the results in real time. The empty output from the first grep is a meaningful signal that shifts the search strategy. The second grep's output—showing only rope_scaling references—confirms that deepseek_v4.py delegates MoE routing to a shared module.

Significance in the Broader Arc

This message is part of a larger narrative about the tension between performance optimization and numerical correctness in production ML systems. The team has pushed the hardware to its limits with custom CUDA kernels, bfloat16 tensor-core operations, and aggressive quantization (NVFP4). But each optimization carries a risk of introducing subtle errors that only manifest under specific conditions—long contexts, multi-turn interactions, or particular numerical regimes.

The assistant's approach in this message exemplifies good debugging practice: formulate a clear hypothesis, design a targeted test, execute it, and let the evidence guide the next step. The empty grep result doesn't resolve the double-scaling question—it simply redirects the search to deepseek_v2.py. But that redirection is itself progress, narrowing the search space and bringing the assistant closer to the root cause.

In the broader context of the session, this diagnostic work ultimately exonerates the MoE routed-scaling patch (and all other speed patches) as the cause of the coherence bug. The real culprit turns out to be the DSA sparse attention's top-512 selection mechanism, which loses recall beyond ~4K tokens due to aggressive quantization in the index keys. But that discovery is still many messages away. Here, at <msg id=12862>, the assistant is still in the thick of the investigation, methodically eliminating suspects one by one.

Conclusion

Message <msg id=12862> is a masterclass in targeted diagnostic reasoning. It takes a specific, well-defined hypothesis—that the MoE routed-scaling factor might be double-applied—and tests it with a focused grep command. The reasoning section shows the assistant synthesizing source code, diffs, and architectural knowledge into a clear mental model of the risk. The empty result is not a dead end but a redirect, pushing the investigation toward the shared MoE infrastructure. In the high-stakes environment of production LLM deployment, where a single numerical error can corrupt every generation, this kind of systematic, evidence-driven debugging is not just good practice—it's essential.