The Quantization Pivot: Switching from MXFP4 to NVFP4 on Blackwell

Introduction

In the high-stakes world of large language model inference on cutting-edge hardware, few moments are as decisive as the one captured in this message. After an exhaustive optimization campaign spanning multiple days—profiling, kernel analysis, configuration tuning, and architectural investigation—the assistant executes a single, seemingly mundane bash command that represents a make-or-break bet on the future direction of the deployment. The command kills the running inference server, deletes a 146 GB checkpoint, and begins downloading a replacement. This is the quantization pivot: the switch from MXFP4 to NVFP4.

The message ([msg 12497]) arrives at a critical inflection point in segment 67 of the opencode session. The assistant has been optimizing DeepSeek-V4-Flash inference on 8× NVIDIA RTX PRO 6000 Blackwell GPUs (sm_120 architecture), and despite deploying with prefill-decode disaggregation, CUDA graphs, NCCL tuning, MTP speculative decoding, and every configuration lever available, throughput remains stubbornly stuck at approximately 23 tokens per second at concurrency 16. This is far below the user's target of ~1000 tok/s and, more embarrassingly, far below the 140 tok/s that the same hardware achieved with the much larger Kimi K2.6 model—a 1-trillion-parameter MoE.

The Bottleneck That Wouldn't Budge

The root cause of this performance regression was identified through GPU profiling in the preceding messages ([msg 12490]–[msg 12492]). Two kernels consumed 77% of decode time: the MoE slot-GEMV (_mxfp4_slot_gemv_kernel) at 39% and the sparse-MLA attention kernel (_tiled_sparse_decode_kernel) at 38%. Both were running on CUDA cores (SIMT) rather than the sm_120a FP4/FP8 tensor cores. Tensor-pipe utilization was below 1%. The hardware was capable of far more, but the software stack was failing to leverage it.

The user's comment in [msg 12493] crystallized the frustration: "this machine with MTP was running Kimi, a 1T MoE at 140t/s (tp8 long mtp), so 15t/s at a much smaller model is really really really terrible." This observation reframed the problem. The hardware wasn't the issue—the kernel dispatch was. The same GPUs that drove 140 tok/s on a trillion-parameter model were delivering only 15 tok/s on a much smaller model because the wrong kernels were being selected.

The Insight: Format Determines Fate

The key insight, developed in the assistant's reasoning in [msg 12494], was that the bottleneck was fundamentally a quantization format mismatch. The stock DeepSeek-V4-Flash checkpoint uses MXFP4 (Microscaling FP4, group-32 with UE8M0 scales), which forces the MoE computation through a CUDA-core slot-GEMV kernel. The in-tree tensor-core grouped GEMM (cutlass_fp4_group_mm) is designed for NVFP4 (NVIDIA FP4, group-16 with E4M3 scales). The assistant recognized that switching to the official NVIDIA NVFP4 checkpoint could route MoE execution through tensor-core paths—either MARLIN W4A16 (the same kernel class that drove Kimi at 140 tok/s) or the native FP4 tensor-core grouped GEMM.

A parallel subagent task verified this routing logic definitively. On sm_120 hardware with --moe-runner-backend auto, an NVFP4 checkpoint triggers MARLIN W4A16, which dequantizes FP4 weights and runs them through tensor cores in FP16/BF16 precision. With --moe-runner-backend triton, it routes to cutlass_fp4_group_mm, the native FP4 tensor-core kernel. Either path flips the 39% MoE bottleneck from CUDA-core to tensor-core execution. The user confirmed in [msg 12495] that downloading is "nearly free" with fast internet, removing the last friction point.## The Message Itself: A Pivot in One Bash Command

The subject message ([msg 12497]) is a single bash command executed over SSH on the remote machine at 10.1.230.171. It performs three critical actions in sequence:

  1. Kill the running server: pkill -9 -f "launch_server.*3000[0]" terminates the SGLang inference server that was serving the MXFP4 checkpoint. The -9 signal (SIGKILL) is forceful, ensuring the process is removed immediately regardless of any cleanup handlers. A five-second sleep follows to allow any lingering processes to settle.
  2. Delete the MXFP4 checkpoint: rm -rf /root/models/DeepSeek-V4-Flash removes the 146 GB stock checkpoint from disk. This is a deliberately destructive action—there is no backup, no archive, no safety net. The assistant has calculated that the NVFP4 checkpoint is the correct path forward and that the MXFP4 weights can be re-downloaded if needed. The disk usage drops from 92% to 82%, freeing approximately 107 GB.
  3. Start the NVFP4 download: Using Hugging Face's hf download tool, the assistant initiates a download of the NVIDIA NVFP4 checkpoint (76 files) to /root/models/DeepSeek-V4-Flash-NVFP4. The download runs in the background via nohup, with output logged to /root/nvfp4_dl.log. The initial progress shows files being fetched at 1–3 files per second. The command is wrapped in a 60-second SSH timeout and includes a 12-second sleep to capture initial download progress. The output confirms success: the model was deleted, disk freed to 190 GB available, and the download started with PID 113170, already fetching files.

The Reasoning: Why This Specific Sequence?

The assistant's reasoning reveals a careful cost-benefit analysis. The decision to delete rather than archive the MXFP4 checkpoint reflects a pragmatic tradeoff: the NVFP4 checkpoint is approximately 150 GB, and with only 83 GB free initially, there is no room for both. The MXFP4 checkpoint can be re-downloaded later if needed (the user confirmed fast internet), while the Kimi K2.6 checkpoint (548 GB) is too large to delete casually and may be needed for future comparisons.

The choice of hf download over git clone or direct file transfer is deliberate. Hugging Face's hf download tool handles large model repositories efficiently, supporting parallel file fetching, resume capability, and proper file verification. The nohup wrapper ensures the download survives the SSH session termination.

The 12-second sleep before checking progress is a heuristic: long enough to confirm the download has started and is making progress, short enough to not waste the SSH timeout window. The assistant is balancing the need for confirmation against the risk of the SSH connection timing out.

Assumptions Embedded in the Action

This message rests on several critical assumptions, some explicit and some implicit:

PR #25820 is merged. The NVFP4 checkpoint requires SGLang PR #25820 for proper auto-detection of the moe_quant_algo field in hf_quant_config.json. Without this PR, the checkpoint might be misidentified and routed to a slow fallback path. The assistant assumes that a pull to upstream main (performed earlier) included this PR, which was merged on May 28—plausible given the June 17 timeline.

The NVFP4 checkpoint will route to tensor-core MoE. The subagent task verified that on sm_120, NVFP4 triggers MARLIN W4A16 (with --moe-runner-backend auto) or cutlass_fp4_group_mm (with --moe-runner-backend triton). But this routing depends on the exact SGLang version, the quantization detection logic, and the hardware detection code. Any mismatch could silently fall back to a slow path.

The download will complete successfully. With 76 files totaling ~150 GB, the download is vulnerable to network interruptions, Hugging Face rate limits (the warning about missing HF_TOKEN is visible), and disk space constraints. The assistant freed 107 GB, but the NVFP4 checkpoint is large enough that the exact size matters.

The server can be cleanly restarted. Killing with SIGKILL is forceful and may leave CUDA contexts, GPU memory allocations, or shared-memory segments in an inconsistent state. The assistant assumes that starting a fresh server will cleanly reinitialize everything.## The Thinking Process: A Window into Strategic Decision-Making

The assistant's reasoning in the preceding messages reveals a sophisticated decision-making process. The analysis in [msg 12494] shows the assistant working through multiple layers of inference:

  1. Comparative benchmarking: The user's observation that Kimi achieved 140 tok/s on the same hardware establishes a performance ceiling that the current deployment should approach. This reframes the problem from "how do we optimize the current stack" to "what is fundamentally different about the kernel dispatch."
  2. Format-level diagnosis: The assistant traces the bottleneck not to a specific kernel's implementation quality but to the quantization format that determines which kernel is selected. This is a higher-level diagnosis than typical micro-optimization approaches.
  3. Verification before action: Rather than blindly downloading the NVFP4 checkpoint, the assistant dispatches a subagent task to trace the exact kernel routing path. This verification step prevents wasting the 30+ minute download on a checkpoint that might not actually improve performance.
  4. Risk mitigation: The assistant explicitly notes that the MXFP4 checkpoint can be re-downloaded, that the NVFP4 checkpoint is from an official source (nvidia/), and that PR #25820 is likely merged. These are explicit risk assessments embedded in the reasoning.

Input Knowledge Required

To understand this message fully, one needs:

Output Knowledge Created

This message creates several forms of output knowledge:

Mistakes and Incorrect Assumptions

While the message is well-reasoned, several assumptions warrant scrutiny:

The download speed assumption. The user stated "downloading here is nearly free, we have really fast internet," but the warning about missing HF_TOKEN in the output suggests unauthenticated downloads with lower rate limits. The initial fetch rate of 1–3 files per second for 76 files totaling ~150 GB implies the download could take 30–60 minutes or more. The assistant's 12-second check window is too short to verify the download will complete successfully.

The SIGKILL safety assumption. Using pkill -9 is the nuclear option. While it guarantees process termination, it may leave GPU memory in an inconsistent state, requiring a full CUDA context reset or even a GPU reset. A gentler SIGTERM with a grace period would be safer, though the assistant's five-second sleep may mitigate this.

The PR #25820 merge assumption. The assistant assumes the PR is merged because the timeline suggests it should be, but this is not verified. If the PR is not present, the NVFP4 checkpoint may not be auto-detected, and the server may fall back to the same slow MXFP4 path or fail to load entirely.

The single-bottleneck framing. The NVFP4 switch addresses the 39% MoE bottleneck, but the 38% attention bottleneck remains untouched. Even a perfect MoE fix can at most double throughput (from 23 to ~46 tok/s), still far from the 1000 tok/s target. The assistant acknowledges this implicitly but the message's decisive tone may oversell the expected improvement.## Conclusion

The message in [msg 12497] is a textbook example of a strategic pivot in ML infrastructure optimization. Faced with a throughput ceiling that no configuration lever could break, the assistant correctly identified that the bottleneck was not in how the computation was executed but in which computation path was selected. The quantization format—MXFP4 vs. NVFP4—determined whether the MoE ran on CUDA cores or tensor cores, and that single difference explained the order-of-magnitude gap between the current deployment and the hardware's demonstrated capability.

The execution is clean and decisive: kill the old server, delete the old weights, start the new download. No hesitation, no half-measures. The assistant has done the verification work (the subagent task tracing kernel routing), assessed the risks (PR merge status, download feasibility), and committed to the path forward.

But the message also reveals the limits of configuration-driven optimization. Even if the NVFP4 switch doubles throughput to ~46 tok/s, the attention bottleneck remains, and the 1000 tok/s target requires a multi-week custom kernel effort akin to the earlier K2.6 work. The pivot is necessary but not sufficient—it's the highest-leverage move available without writing new CUDA kernels, but it's not the final answer.

In the broader context of segment 67, this message marks the transition from diagnosis to action. The profiling, the analysis, the subagent verification—all of that was preparation for this single bash command. The download that starts in this message will determine whether the NVFP4 hypothesis is correct, and whether the deployment can escape the sm_120 fallback kernel trap that has plagued it from the start.