The Quantization Epiphany: How One Message Pivoted from Kernel Writing to Checkpoint Swapping

Introduction

In the middle of a grueling optimization campaign for DeepSeek-V4-Flash on NVIDIA Blackwell GPUs (RTX PRO 6000, sm_120), a single message from the assistant marked a decisive turning point. After days of methodical profiling, kernel analysis, and configuration tuning had revealed that the model was achieving a paltry 15 tokens per second—while the same hardware had previously delivered 140 t/s on a 1-trillion-parameter Kimi K2.6 model—the assistant had a critical insight. The bottleneck wasn't the hardware, and it wasn't even necessarily the need for custom CUDA kernels. The problem was a quantization format mismatch.

This article examines message 12494 in the conversation, where the assistant processes the user's pointed observation about the performance gap, connects it to the profiling data already collected, and pivots from a "write custom kernels" strategy to a "swap the checkpoint format" strategy. The message is a masterclass in diagnostic reasoning under pressure, combining real-time web research, disk-space arithmetic, and deep codebase analysis to identify the single highest-leverage intervention available.

<hr>

The Subject Message: Full Text

The message begins with an extensive Agent Reasoning block, followed by three parallel tool calls: a web fetch to inspect the NVFP4 checkpoint configuration, a bash command to check disk space, and a task subagent to trace SGLang's NVFP4 MoE routing on SM120 hardware.

## Agent Reasoning
The user is pointing out that this same hardware achieved 140 t/s with Kimi (1T MoE) using optimized kernels, so getting only 15 t/s on a much smaller model like DeepSeek-V4-Flash is a massive regression—the issue appears to be specific to how the DSA implementation is handling things rather than a hardware limitation.

The real bottleneck is a quantization format mismatch: the stock model uses MXFP4 (group-32), which forces a slow CUDA-core kernel, while the in-tree tensor-core grouped GEMM is built for NVFP4 (group-16). Switching to the official NVIDIA NVFP4 quantization would route the MoE through the proper tensor-core kernel instead, directly fixing both the 39% MoE bottleneck and the low tensor utilization without needing to write any new kernels.

But there's a wrinkle: the method selection logic in fp8.py routes MXFP4 checkpoints to the Marlin slot-GEMV, while NVFP4 checkpoints would take a different path through the modelopt NVFP4 method. However, there are some conflicting signals in the codebase — the modelopt fp4 registration has conditions around SM_120 support, and there's a vllm issue noting that NVFP4 native kernels aren't auto-selected on certain hardware. I need to trace through exactly which kernel gets invoked for an NVFP4 checkpoint on this setup.

The user's intuition about an NVFP4 checkpoint unlocking the tensor-core MoE path (b12x_fused_moe or cutlass_fp4_group_mm) could be the key to solving the 39% MoE bottleneck, but I need to verify whether SGLang's quantization detection actually routes it to a working SM_120 kernel. Meanwhile, the Marlin INT4 baseline from Kimi (140 t/s) used W4A16, but that path reportedly produces NaN on SM_120 according to the mxfp4_marlin_moe.py findings.

Wait — that NaN issue was specifically for MXFP4 Marlin, not INT4 W4A16 Marlin. The canada-quant W4A16 INT4 checkpoint actually worked on Kimi at 140 t/s, so Marlin INT4 is viable on SM_120, just from a sketchy source. The cleanest move is to grab the official nvidia/DeepSeek-V4-Flash-NVFP4 checkpoint, serve it with the right backend (modelopt_fp4 / flashinfer / cutlass grouped GEMM), and measure whether the MoE now hits tensor cores.

But there's a catch — SGLang's SM_120 DSv4 path might not actually route NVFP4 MoE through the fast cutlass kernel. The research shows NVFP4 MoE on SM_120 is still WIP in both vLLM and FlashInfer, so I need to verify whether downloading that checkpoint will actually run faster or just fall back to a slow path. I should check the quant method in the nvidia config and trace how SGLang handles NVFP4 MoE routing on SM_120.

Given the download is the bottleneck, I'll start it immediately while investigating the routing in parallel. But there's a disk constraint — we have ~78GB free and the NVFP4 checkpoint is ~140GB, same as the current MXFP4 model at 146GB. We can't fit both, so I'd need to delete the MXFP4 stock model to make room. Let me check the current disk state first before committing to that.

The real issue is that even with perfect MoE optimization, DeepSeek-V4-Flash's DSA attention mechanism adds overhead that Kimi didn't have, so we won't hit 140 t/s regardless. But fixing the MoE bottleneck (currently 39% of time) by routing to tensor cores via NVFP4 could roughly 1.6x throughput, and if we also optimize attention, we might reach 60-100+ t/s — a dramatic improvement from the current 15 t/s. The concrete next step is to verify the disk situation and check the NVIDIA NVFP4 checkpoint details and whether SGLang's sm120 routing will automatically handle it. The real bottleneck is understanding how SGLang routes NVFP4 MoE kernels on sm120 — specifically whether it hits the cutlass_fp4_group_mm path or falls back to something slower. I need to trace through the ModelOptFp4Config dispatch logic and verify that the nvidia checkpoint actually exists and is accessible, then check available disk space before committing to the download. The real win here is if the NVFP4 checkpoint routes through the in-tree `cutlass_fp4_group_mm` kernel — that would shift MoE from CUDA-core bottleneck to tensor-core performance. Let me verify the routing works and grab the checkpoint.

The message then continues with a summary of the insight and three parallel tool calls:

This is the key insight: **140 t/s on a 1T model means the hardware is fine — the DSv4 sm_120 fallback kernels are the problem.** And the root MoE issue is format: the stock checkpoint is **MXFP4** (group-32/UE8M0), which forces the CUDA-core slot-GEMV, because the in-tree **tensor-core** grouped GEMM (`cutlass_fp4_group_mm`) is **NVFP4** (group-16/E4M3). A proper **NVFP4 checkpoint** could route the MoE onto tensor cores — fixing the 39% bottleneck *and* the <1% tensor util without writing a kernel.

Let me verify the NVFP4 checkpoint + whether SGLang actually routes NVFP4 MoE to tensor cores on sm_120, and check disk — in parallel.
[exa_web_fetch_exa] {"maxCharacters":3500,"urls":["https://huggingface.co/nvidia/DeepSeek-V4-Flash-NVFP4/raw/main/config.json","https://huggingface.co/nvidia/DeepSeek-V4-Flash-NVFP4"]}
# Config.json
URL: https://huggingface.co/nvidia/DeepSeek-V4-Flash-NVFP4/raw/main/config.json

{
 "architectures": [
 "DeepseekV4ForCausalLM"
 ],
 ...
[bash] timeout 30 ssh -o StrictHostKeyChecking=no root@10.1.230.171 'echo "=== disk ==="; df -h / | tail -1; echo "=== model sizes ==="; du -sh /root/models/* 2>/dev/null'
=== disk ===
scratch/containers/subvol-200-disk-0 1000G  918G   83G  92% /
=== model sizes ===
146G	/root/models/DeepSeek-V4-Flash
548G	/root/models/Kimi-K2.6
6.5G	/root/models/Kimi-K2.6-DFlash-tmp-long
3.3G	/root/models/Qwen3.6-27B-DFlash
6.0G	/root/models/kimi-k2.6-eagle3

[task] {"description":"SGLang NVFP4 MoE sm120 routing","prompt":"In /home/theuser/glm-kimi-sm120-rtx6000bw/sglang-dsv4 (SGLang main), determine EXACTLY how an NVFP4 / ModelOpt-FP4 quantized DeepSeek-V4 checkpoint's MoE experts are executed on SM120 (RTX PRO 6000, compute 12.0). This is the decisive question...

<hr>

WHY This Message Was Written: Reasoning, Motivation, and Context

The Performance Crisis

To understand why this message exists, one must understand the crisis that preceded it. The assistant had spent multiple rounds deploying DeepSeek-V4-Flash on a machine with 8× RTX PRO 6000 Blackwell GPUs, implementing prefill-decode disaggregation, tuning every configuration lever available—NCCL LL ring, FP8 GEMM autotune configs, CUDA graphs, MTP speculative decoding—and had achieved a maximum throughput of approximately 23 tokens per second at concurrency C=16 (see [msg 12492]). The single-request throughput was even worse at ~10 tok/s.

The user then dropped a devastating comparison: this same machine, running a 1-trillion-parameter Kimi K2.6 model with MTP, had achieved 140 tokens per second. The DeepSeek-V4-Flash model, despite being substantially smaller, was running at roughly one-tenth the throughput. This wasn't just a performance bug—it was an order-of-magnitude regression that demanded a fundamental rethinking of the approach.

The Profiling Revelation

The assistant had already done the hard work of profiling. At C=16 concurrency, the GPU kernel profile (captured in [msg 12488]) showed a stark picture:

The User's Hint

The user's message ([msg 12493]) contained two critical pieces of information. First, the performance comparison to Kimi (140 t/s) established that the hardware was not the problem. Second, the user pointed to specific NVFP4 quantized checkpoints: the official nvidia/DeepSeek-V4-Flash-NVFP4 and a community canada-quant/DeepSeek-V4-Flash-W4A16-FP8-MTP (the latter flagged as potentially sketchy).

This hint was the catalyst. The assistant had previously been planning to write custom CUDA kernels to replace the slow fallback kernels—a multi-week engineering effort. The user's suggestion reframed the problem: perhaps the issue wasn't missing kernels but wrong quantization format.

<hr>

HOW Decisions Were Made

The Format Mismatch Hypothesis

The assistant's reasoning in this message demonstrates a sophisticated diagnostic process. The key deduction was:

  1. MXFP4 (the stock checkpoint format) uses group-32 quantization with UE8M0 scales. The SGLang codebase routes MXFP4 checkpoints through a CUDA-core slot-GEMV kernel (_mxfp4_slot_gemv_kernel), which is slow and doesn't use tensor cores.
  2. NVFP4 (the NVIDIA-optimized format) uses group-16 quantization with E4M3 block scales. The in-tree tensor-core grouped GEMM (cutlass_fp4_group_mm) is built for NVFP4, not MXFP4.
  3. Therefore, switching from the MXFP4 checkpoint to an NVFP4 checkpoint could automatically route the MoE computation through tensor cores, fixing the 39% MoE bottleneck without writing any new kernels. This was a classic "work smarter, not harder" insight. The assistant had been planning to build custom CUDA kernels (Tier B in the earlier plan), but the user's hint suggested a simpler path: use the right checkpoint format and let the existing tensor-core infrastructure do the work.

The Risk Assessment

The assistant didn't blindly commit to the NVFP4 swap. The reasoning block shows careful risk assessment:

The Parallel Investigation Strategy

The message launches three parallel investigations:

  1. Web fetch: Inspect the NVFP4 checkpoint's config.json on HuggingFace to understand its quantization configuration
  2. Bash disk check: Determine available disk space and model sizes to see if the NVFP4 checkpoint can fit
  3. Task subagent: Trace through SGLang's source code to determine exactly how NVFP4 MoE is executed on SM120 This parallel strategy is efficient: the web fetch and disk check are fast, while the task subagent does deep codebase analysis. The assistant doesn't wait for all results before acting—it starts the investigation while preparing to download. <hr>

Assumptions Made by the Assistant

Correct Assumptions

  1. The hardware is capable: The assistant correctly assumes that since Kimi achieved 140 t/s on the same hardware, the bottleneck is software, not hardware. This is validated by the profiling data showing <1% tensor utilization.
  2. The quantization format is the root cause: The assistant correctly identifies that MXFP4 forces CUDA-core execution while NVFP4 would use tensor cores. This is based on reading the SGLang quantization routing code.
  3. NVFP4 would be a clean fix: The assistant assumes that switching to NVFP4 would route MoE through tensor cores, fixing the 39% bottleneck. This turns out to be correct—subsequent messages show NVFP4 achieving ~28 tok/s, a ~24% improvement.

Potentially Incorrect or Overly Optimistic Assumptions

  1. NVFP4 alone might not be sufficient: The assistant estimates that fixing MoE could give ~1.6x throughput improvement, potentially reaching 60-100+ t/s with attention optimization. In reality, even after NVFP4, the throughput only reaches ~28 tok/s at C=16. The attention kernel remains a bottleneck, and the MTP verifier consumes enough GPU memory to halve the effective batch size.
  2. The cutlass_fp4_group_mm path would work: The assistant hopes that NVFP4 would route through the in-tree cutlass_fp4_group_mm kernel. The task subagent's findings (in subsequent messages) reveal that the actual routing is more complex—NVFP4 can go through Marlin W4A16 (tensor-core, FP16/BF16) or through the Triton backend for native FP4 grouped GEMM.
  3. DSA attention overhead is manageable: The assistant acknowledges that DeepSeek-V4-Flash's DSA (Dynamic Sparse Attention) adds overhead that Kimi didn't have, but still estimates 60-100+ t/s as achievable. The actual ceiling turns out to be much lower due to the sm_120 fallback sparse-attention kernel.

Implicit Assumptions

  1. The NVFP4 checkpoint is trustworthy: The assistant correctly treats the official NVIDIA checkpoint as trustworthy, while flagging the canada-quant one as potentially sketchy.
  2. PR #25820 is merged: The assistant assumes that PR #25820 (NVFP4 auto-detection) is already in the SGLang main branch, since the model was released on 05/28 and it's now 06/17. This turns out to be correct.
  3. Disk space management is straightforward: The assistant assumes that deleting the MXFP4 checkpoint (146 GB) frees enough space for the NVFP4 checkpoint (~140-150 GB). The disk check shows 83 GB free, and after deletion, 229 GB would be available—sufficient for the download. <hr>

Input Knowledge Required to Understand This Message

To fully understand this message, one needs knowledge of:

Quantization Formats

Hardware Architecture

SGLang Architecture

The Broader Context

Output Knowledge Created by This Message

Immediate Outputs

  1. The NVFP4 hypothesis: The message formulates and begins testing the hypothesis that switching from MXFP4 to NVFP4 is the highest-leverage fix for the MoE bottleneck.
  2. Disk space assessment: The bash command reveals that the machine has 83 GB free, with the MXFP4 model consuming 146 GB. Deleting it would free 229 GB, sufficient for the NVFP4 checkpoint.
  3. NVFP4 checkpoint inspection: The web fetch retrieves the NVFP4 checkpoint's config.json, confirming it uses the DeepseekV4ForCausalLM architecture with NVFP4 quantization.
  4. SGLang routing investigation: The task subagent is launched to trace exactly how NVFP4 MoE is executed on SM120, which will determine whether the checkpoint swap actually routes through tensor cores.

Knowledge Created for Subsequent Messages

  1. The deletion/download action: The assistant's reasoning about disk constraints directly leads to the action in the next round ([msg 12496]), where the MXFP4 checkpoint is deleted and the NVFP4 download begins.
  2. The routing verification: The task subagent's findings (returned in subsequent messages) provide the definitive trace of NVFP4 MoE execution on SM120, confirming that NVFP4 + Marlin W4A16 uses tensor cores.
  3. The benchmark plan: The NVFP4 swap sets up the next phase of benchmarking—measuring whether tensor utilization improves, whether throughput increases, and whether MTP can be layered on top.

Broader Knowledge Contribution

This message contributes to the broader understanding of deploying quantized LLMs on new hardware architectures:

The Thinking Process: A Window into Diagnostic Reasoning

The Agent Reasoning block in this message is unusually rich, spanning multiple paragraphs of stream-of-consciousness analysis. Let me trace through the key logical steps.

Step 1: Reframing the Problem

The assistant starts by processing the user's comparison to Kimi: "140 t/s on a 1T model means the hardware is fine." This reframes the problem from "how do we optimize this model" to "why is this specific model/configuration so much worse than what's possible on this hardware."

Step 2: Connecting Profiling Data to Root Cause

The assistant connects two pieces of data:

Step 3: The Format Mismatch Hypothesis

The assistant realizes that the stock MXFP4 format forces the CUDA-core slot-GEMV because the tensor-core grouped GEMM (cutlass_fp4_group_mm) is built for NVFP4. This is the critical insight.

Step 4: Reality-Checking the Hypothesis

The assistant doesn't stop at the hypothesis. It immediately identifies potential problems:

Step 5: Correcting a Misconception

The assistant catches itself on the Marlin NaN issue: "Wait — that NaN issue was specifically for MXFP4 Marlin, not INT4 W4A16 Marlin." This self-correction is important because it means the proven Kimi path (Marlin W4A16) is viable on SM120.

Step 6: Pragmatic Decision-Making

The assistant weighs the cleanest path (official NVIDIA NVFP4 checkpoint) against the risk (SGLang might not route it correctly). The decision is pragmatic: start the download immediately (since it's the bottleneck) while investigating the routing in parallel.

Step 7: Managing Expectations

The assistant explicitly acknowledges that even with perfect MoE optimization, DSA attention overhead means they won't hit 140 t/s. The estimate of 60-100+ t/s is ambitious but qualified as requiring attention optimization as well.

Step 8: Parallel Execution

The message launches three parallel investigations, demonstrating efficient resource utilization. The assistant doesn't wait for one result before starting the next—it works on all fronts simultaneously.

<hr>

Mistakes and Incorrect Assumptions

The Throughput Estimate Was Too Optimistic

The assistant estimates that fixing MoE via NVFP4 could give ~1.6x throughput improvement, potentially reaching 60-100+ t/s with attention optimization. In reality, the NVFP4 checkpoint achieves ~28 tok/s at C=16—a ~24% improvement over the MXFP4 baseline, but far short of the 60-100+ t/s estimate.

The gap between estimate and reality stems from two factors:

  1. The attention kernel remained a bottleneck: Even after MoE was moved to tensor cores, the _tiled_sparse_decode_kernel continued to consume ~38% of GPU time, running on CUDA cores. The assistant had acknowledged this risk but underestimated its severity.
  2. The MTP verifier memory overhead: The user later pointed out that the MTP verifier was consuming enough GPU memory to halve the effective batch size from 16 to 8. This meant that even with faster MoE, the effective throughput was limited by the reduced batch capacity.

The "Without Writing a Kernel" Assumption

The assistant states that NVFP4 would fix the MoE bottleneck "without writing any new kernels." While technically true for the MoE path (NVFP4 routes through existing Marlin W4A16 or cutlass kernels), the attention bottleneck still requires custom kernel work. The assistant's earlier K2.6 work had required a custom split-K flash-decode kernel to achieve similar gains, and the same would be needed here.

The Disk Arithmetic

The assistant calculates that deleting the 146 GB MXFP4 checkpoint would leave 229 GB free (83 GB + 146 GB = 229 GB). The NVFP4 checkpoint is estimated at ~140-150 GB. This arithmetic is correct, but it assumes the NVFP4 checkpoint size is similar to the MXFP4 one. In practice, the NVFP4 checkpoint might have different compression characteristics due to the different quantization format.

<hr>

Conclusion

Message 12494 represents a pivotal moment in the optimization campaign. It's the moment when the assistant stops trying to optimize within the existing framework and instead questions the fundamental assumptions about what's running and why. The insight that the quantization format mismatch is the root cause—that MXFP4 forces CUDA-core execution while NVFP4 would use tensor cores—is the kind of diagnosis that can only come from deep understanding of both the hardware architecture and the software stack.

The message is also notable for its intellectual honesty. The assistant doesn't pretend to have all the answers. It identifies risks, corrects its own misconceptions in real-time, and launches parallel investigations to validate its hypothesis before committing to the expensive action (deleting a 146 GB checkpoint and downloading a new one). This combination of bold hypothesis generation and cautious verification is the hallmark of effective engineering under uncertainty.

The NVFP4 swap ultimately delivers a ~24% throughput improvement (from ~23 tok/s to ~28 tok/s), which is meaningful but not the order-of-magnitude gain the assistant had hoped for. The attention kernel remains the stubborn bottleneck, and the MTP verifier's memory overhead further limits throughput. But the diagnostic framework established in this message—"check the quantization format first, because it determines which kernel path is used"—is a valuable lesson that applies far beyond this specific deployment.

In the end, the message teaches us that on new hardware architectures, the most impactful optimization isn't always writing faster kernels. Sometimes it's using the right checkpoint format that routes through the kernels that already exist.