The 24% Ceiling: Diagnosing the Limits of NVFP4 Quantization on Blackwell SM120 GPUs

Introduction

In the high-stakes world of large language model inference, a 24% throughput improvement is usually cause for celebration. But when the target is a thousand tokens per second and you're crawling toward thirty, every percentage point gained only sharpens the question: what's still in the way? This is the predicament facing the assistant in message [msg 12509] of a sprawling optimization campaign to deploy DeepSeek-V4-Flash on eight NVIDIA RTX PRO 6000 Blackwell GPUs. The message captures a pivotal moment of diagnostic clarity—the moment when a 24% gain is recognized not as a victory but as a signal, and the assistant pivots from one quantization backend to another in search of the bottleneck that truly matters.

This article examines that single message in depth: the reasoning that produced it, the knowledge it required, the assumptions it made, and the decisions it set in motion. It is a case study in structured performance debugging at the frontier of GPU inference engineering.

The Road to NVFP4

To understand message [msg 12509], one must understand the journey that led to it. The assistant had been working for hours to deploy DeepSeek-V4-Flash—a massive mixture-of-experts (MoE) model with 685B total parameters and approximately 13B active per forward pass—on a machine with eight RTX PRO 6000 GPUs (compute capability sm_120). The stock MXFP4 checkpoint had delivered disappointing throughput: roughly 23 tokens per second at a concurrency of 16 requests, far below the user's target of ~1000 tok/s.

The assistant had methodically exhausted every configuration lever available. CUDA graphs were already enabled. NCCL LL+Ring tuning made no difference. Tilelang indexer fusion failed to JIT-compile on sm_120. Expert parallelism made things worse due to PCIe all-to-all overhead. A GPU profile had traced 63% of decode time to a single kernel—_tiled_sparse_decode_kernel—the sm_120 Triton fallback for sparse MLA attention, launching only 64 blocks on ~170 SMs.

The breakthrough had come in the form of PR #25820, an upstream SGLang pull request that added proper detection and routing for NVIDIA's NVFP4 quantization format. The assistant had fetched and applied this patch (overcoming a shallow-clone merge conflict by using git apply --3way), downloaded the 149 GB NVFP4 checkpoint, and launched the model with --moe-runner-backend marlin. The marlin backend dequantizes FP4 weights to BF16 and executes W4A16 matrix multiplications on the tensor cores—a proven path that had worked for Kimi K2.6 on the same hardware.

The result was a clean 24% improvement: 28.7 tok/s at C=16, with time-to-first-token roughly halved from 830ms to 333ms. Correctness was verified (17×23=391, no NaN). The NVFP4+marlin combination worked.

The Core Insight: Why 24% Is Not Enough

Message [msg 12509] opens with the assistant reviewing these benchmark results. The tone is measured, almost clinical:

"The gains are meaningful but not the dramatic speedup I was hoping for, and I think I understand why."

This sentence is the thesis of the entire message. The assistant then articulates a two-part diagnosis that reveals a sophisticated understanding of the inference pipeline's bottleneck hierarchy.

Part one: Marlin is not native FP4. The marlin backend dequantizes FP4 weights to BF16 before performing matrix multiplication on the tensor cores. This means every MoE layer must stream weights from memory and dequantize them—a memory-bandwidth-bound operation that limits throughput. The assistant recognizes that true FP4×FP4 computation (W4A4) would avoid this overhead entirely, but marlin is W4A16: the weights are stored in FP4 but computed in BF16.

Part two: Attention is now the bottleneck. The assistant references profiling data showing that at C=16, the sparse attention kernel consumes 38% of GPU time, while the MoE layers consume 39%. If marlin roughly halves the MoE cost from 39% to ~20%, that explains the ~24% overall improvement. But attention remains completely unchanged. The bottleneck has simply shifted from MoE to attention.

This is a classic Amdahl's Law moment: optimizing one component reveals another as the new dominant term. The assistant's reasoning demonstrates that they understand not just that the system is faster, but why the speedup is bounded, and what must be addressed next.

The Decision: Pivot to Native FP4

Having diagnosed the situation, the assistant formulates a plan:

"To push further, I should explore the native FP4 path using the Triton backend with cutlass_fp4_group_mm for true W4A4 tensor-core operations, which would avoid the dequantization overhead entirely."

The reasoning here is precise. The assistant knows that the SGLang MoE runner has multiple backends. The triton backend, when used with NVFP4 weights, routes to cutlass_fp4_group_mm—a native FP4×FP4 GEMM that operates directly on the quantized weights without dequantization. This is the path that could theoretically outperform marlin by eliminating the memory bandwidth tax of weight dequantization.

The assistant also considers the flashinfer_trtllm_routed backend (the PR's intended approach) but correctly judges it as SM100-only and therefore non-functional on sm_120. This judgment is based on earlier investigation of the PR's code: the hook forces flashinfer_trtllm_routed for NVFP4 detection, but that backend relies on trtllm-gen kernels that are architecture-gated to SM100 (B200/GB300). On sm_120 (RTX PRO 6000), those kernels would fail.

The assistant then verifies a critical detail about the routing logic:

"I need to verify the hook logic works correctly with the explicit triton backend—the condition checks for auto mode, so with triton specified, the hook won't override and the native FP4 cutlass path should execute as intended."

This demonstrates careful reading of the codebase. The deepseek_v4_hook.py file (patched by PR #25820) only overrides the backend selection when the user specifies auto. By explicitly passing --moe-runner-backend triton, the assistant ensures the hook's SM100-specific routing is bypassed and the native FP4 cutlass path is used.

The Action: Kill, Edit, Launch

The message concludes with a bash command that executes the pivot:

pkill -9 -f "launch_server.*3000[0]" 2>/dev/null; sleep 5
sed "s/--moe-runner-backend marlin/--moe-runner-backend triton/" /root/serve_dsv4_nvfp4.sh > /root/serve_dsv4_nvfp4_cutlass.sh
chmod +x /root/serve_dsv4_nvfp4_cutlass.sh
grep moe-runner /root/serve_dsv4_nvfp4_cutlass.sh
cd /root && nohup bash /root/serve_dsv4_nvfp4_cutlass.sh > /root/dsv4_nvfp4c.log 2>&1 & echo "PID $!"

The command is a model of operational efficiency: kill the old server, create a new launch script with a single sed substitution, make it executable, verify the change, and launch it in the background. The sleep 5 between kill and launch ensures the GPUs are fully released and the old process's CUDA contexts are cleaned up.

The output confirms the new configuration:

exec /root/venv_sglang211/bin/python -m sglang.launch_server \
  --model-path /root/models/DeepSeek-V4-Flash-NVFP4 --trust-remote-code \
  --tp 4 --moe-runner-backend triton \
  --mem-fraction-static 0.70 \
  --cuda-graph-max-bs 32 \
  --host 127.0.0.1 --port 30000
PID 116249

The only difference from the previous launch is --moe-runner-backend triton replacing --moe-runner-backend marlin. Everything else—TP4, memory fraction 0.70, CUDA graph max batch size 32, port 30000—remains identical, ensuring a controlled comparison.

Assumptions and Their Validity

The assistant's reasoning in this message rests on several assumptions, most of which are well-founded but worth examining.

Assumption 1: The triton backend will route to cutlass_fp4_group_mm on sm_120. This is based on the assistant's earlier investigation of the modelopt quantization code, which showed that the triton backend takes the cutlass branch for NVFP4 weights. The assumption is reasonable but not yet verified—the server hasn't started, and the actual kernel selection depends on runtime checks for architecture support.

Assumption 2: Native FP4×FP4 will outperform W4A16 marlin. This is physically plausible: native FP4 avoids the memory bandwidth cost of dequantizing weights from FP4 to BF16. However, the actual performance depends on whether the cutlass_fp4_group_mm kernel achieves good occupancy and tensor-core utilization on sm_120. The kernel may be unoptimized for this architecture, or may have other overheads (e.g., scale factor handling) that offset the bandwidth savings.

Assumption 3: The flashinfer_trtllm_routed backend is SM100-only and will not work. This is supported by the PR's own documentation (tested on B200/GB300) and by the assistant's knowledge that trtllm-gen kernels are architecture-gated. However, the assistant doesn't actually test this assumption—they prioritize the triton path as the safer bet. This is a pragmatic tradeoff, not a proof.

Assumption 4: Attention remains the dominant bottleneck after the MoE optimization. This is well-supported by the profiling data showing 38% attention time at C=16. Even if the triton backend halves the MoE cost again (from ~20% to ~10%), attention would still account for roughly half of the remaining decode time. The assistant's plan to "tackle the remaining 38% with attention split-K" after testing triton is a logical next step.

Input Knowledge Required

To understand and produce this message, the assistant needed a deep and multi-layered knowledge base:

GPU architecture knowledge: Understanding of sm_120 vs sm_100 compute capabilities, tensor-core operation modes (FP4×FP4 vs W4A16), and the concept of CUDA core (SIMT) vs tensor-core execution. The distinction between native FP4 GEMM and dequantized W4A16 marlin is architecture-specific.

Quantization format knowledge: Understanding of MXFP4 vs NVFP4, the role of group_size=16, the difference between quant_algo "NVFP4" and "MIXED_PRECISION", and how different MoE backends (marlin, triton/cutlass, flashinfer_trtllm_routed) handle these formats.

SGLang codebase knowledge: Familiarity with the MoE runner architecture, the deepseek_v4_hook routing logic, the modelopt quantization path, and the interaction between server_args.py and the hook. The assistant knows that the hook only overrides when backend is auto.

Profiling and bottleneck analysis: The ability to interpret GPU kernel profiles, attribute time to specific kernels (sparse attention vs MoE GEMM), and apply Amdahl's Law to predict the impact of optimizing one component.

Operational knowledge: How to manage remote server processes (pkill, nohup, sleep for cleanup), edit configuration files with sed, and verify changes before launching.

Output Knowledge Created

This message creates several pieces of actionable knowledge:

  1. A quantified bottleneck hierarchy for NVFP4+marlin on sm_120: MoE ~20%, attention ~38%, with attention as the new dominant term after the marlin optimization.
  2. A testable hypothesis about native FP4: The triton backend with cutlass_fp4_group_mm may outperform marlin by avoiding dequantization overhead. This hypothesis is immediately tested by the server launch.
  3. A confirmed routing path: The explicit --moe-runner-backend triton flag bypasses the PR's SM100-specific hook logic, providing a working configuration for sm_120 NVFP4 deployment.
  4. A prioritized roadmap: Test triton → add MTP to whichever backend wins → tackle attention with split-K kernel. This roadmap structures the next phase of optimization.
  5. An operational baseline: The NVFP4+marlin configuration at 28.7 tok/s serves as a reference point for evaluating the triton backend. Any improvement or regression will be immediately measurable against this baseline.

The Broader Significance

Message [msg 12509] is significant beyond its immediate context because it illustrates a fundamental pattern in ML inference optimization: the law of diminishing returns meets the reality of bottleneck shifting. The assistant did not simply apply a new quantization format and declare success. Instead, they used the 24% gain as diagnostic information, tracing the improvement back to its source (MoE cost halved) and identifying the new limiting factor (attention unchanged).

This pattern—measure, diagnose, target, pivot—is the essence of systematic performance engineering. The assistant's reasoning shows no magical leaps or lucky guesses. Every conclusion is grounded in data: profiling percentages, throughput numbers, latency measurements. Every decision is justified by reasoning about the underlying hardware and software stack.

The message also demonstrates the importance of understanding abstraction layers. The assistant doesn't just know that marlin is "faster" or "slower"—they know why: because it dequantizes FP4 to BF16, because it runs on tensor cores, because attention is a separate kernel with its own occupancy problem. This multi-level understanding—from quantization formats to GPU architecture to kernel occupancy to system-level throughput—is what enables precise, effective optimization.

Conclusion

Message [msg 12509] captures a moment of diagnostic clarity in a complex optimization campaign. The assistant receives benchmark results showing a 24% improvement from NVFP4+marlin, immediately recognizes that this gain is bounded by an unchanged attention bottleneck, formulates a hypothesis about native FP4×FP4 computation, and executes a controlled experiment to test it. The reasoning is data-driven, the assumptions are explicit, and the action is precise.

The message is a microcosm of what makes effective ML inference engineering: not just knowing which knobs to turn, but understanding why each knob affects performance, how much it can possibly help, and what will become the next bottleneck when it does. In that sense, the 24% gain is not the story—the diagnosis of why it's only 24% is the real contribution.