The Pivot to Proper Quants: A User's Reality Check on DeepSeek-V4-Flash Performance

In any complex engineering collaboration, there comes a moment when raw data collides with lived experience, and the entire trajectory of the work shifts. Message 12493 in this opencode session is exactly such a moment. After the assistant had spent hours methodically deploying DeepSeek-V4-Flash on 8× NVIDIA RTX PRO 6000 Blackwell GPUs (sm_120), exhaustively tuning every configuration lever, profiling kernels, and arriving at a data-driven recommendation, the user delivered a concise but devastating response that reframed the entire problem. This message is not merely a reply—it is a reality injection, a strategic redirection, and a security advisory rolled into three sentences.

The Context: An Optimization Campaign at a Dead End

To understand the weight of this message, one must appreciate what preceded it. The assistant had successfully deployed DeepSeek-V4-Flash (a Mixture-of-Experts model with FP4 quantization) using SGLang with prefill-decode disaggregation across two NUMA domains. The infrastructure was impressive: prefill on GPU 0–3, decode on GPU 4–7, KV cache transfer via NIXL/UCX, and a router on port 8000. Everything worked end-to-end.

But the performance was abysmal. At batch size 1, the model delivered approximately 10 tokens per second. At concurrency 16, it plateaued at roughly 23 tok/s. The user's target was ~1000 tok/s—a gap of roughly 40×.

The assistant had been methodical. Every configuration lever was pulled: NCCL LL+Ring tuning (negligible effect, communication is only 2% of decode), CUDA graphs (already enabled), tilelang indexer fusion (JIT-compile failure on sm_120), non-marlin MoE backends (invalid for FP4 experts), and expert parallelism (worse due to PCIe all-to-all overhead). None moved the needle.

A definitive GPU profile at concurrency 16 revealed the root cause: two kernels consumed 77% of decode time—the _mxfp4_slot_gemv_kernel (MoE, 39%) and the _tiled_sparse_decode_kernel (attention, 38%)—and both were running on standard CUDA cores (SIMT) rather than the sm_120a FP4/FP8 tensor cores. This explained the user's observation of less than 1% tensor-pipe utilization. The hardware's tensor cores were essentially idle while the CUDA cores did all the work.

The assistant's proposed next step, articulated in message 12492, was to "prototype it—re-quantize MXFP4→NVFP4 at load and route the experts through cutlass_fp4_group_mm." This was a custom engineering approach: write code to convert the checkpoint format at load time and patch the MoE routing to use a tensor-core kernel path.

The Message: A Three-Pronged Reality Check

The user's response was brief but dense with meaning:

Go for the highest leverage fixes. Note that this machine with MTP was running Kimi, a 1T MoE at 140t/s (tp8 long mtp), so 15t/s at a much smaller model is really really really terrible. for nvfp we want proper quants like https://huggingface.co/nvidia/DeepSeek-V4-Flash-NVFP4 or https://huggingface.co/canada-quant/DeepSeek-V4-Flash-W4A16-FP8-MTP (note the latter links some random binaries/forked code, assume it's potentially sketchy)

Three distinct messages are packed into this short paragraph, each addressing a different dimension of the problem.

The Performance Bombshell: Kimi at 140 tok/s

The first and most impactful element is the performance reference point. The user reveals that this same machine, with MTP (Multi-Token Prediction) speculative decoding enabled, was previously running Kimi K2.6—a 1 trillion parameter MoE model—at 140 tokens per second using TP8 (tensor parallelism across 8 GPUs) with long-context MTP. This single data point transforms the entire conversation.

The assistant had been operating under the assumption that ~23 tok/s was a hardware limitation of the sm_120 GPUs for this class of model. The profiling had shown that the fallback kernels (Triton-based, running on CUDA cores) were the bottleneck, and the assistant's analysis concluded that "no amount of config tuning or even a single-kernel rewrite (which could yield ~2-3× overall) can close the ~40× gap to 1000 tok/s." The user's data point proves this assumption wrong. The hardware is demonstrably capable of 140 tok/s on a larger model (1T parameters vs DeepSeek-V4-Flash's ~600B). Getting only 15 tok/s on a smaller model is not a hardware ceiling—it is a software configuration failure.

The user's triple "really" ("really really really terrible") is not hyperbole; it is a measured statement of fact backed by empirical evidence from the same machine. This reframes the problem from "how do we squeeze 2-3× out of these kernels" to "something is fundamentally wrong with how we're running this model."

This is the kind of insight that only someone with deep operational experience on the specific hardware can provide. The assistant had been reasoning from first principles (kernel profiles, roofline analysis, occupancy calculations) but lacked the comparative benchmark that the user possessed. The user's knowledge of the Kimi deployment on the same hardware provides an empirical upper bound that no amount of theoretical analysis could have produced.

The Strategic Redirection: Proper Quants vs. Custom Engineering

The second element is the directive to use "proper quants." The assistant had proposed a custom approach: re-quantize the MXFP4 checkpoint to NVFP4 at load time and patch the MoE routing. This is a fragile, bespoke solution that would require significant engineering effort, testing, and maintenance.

The user redirects to two existing HuggingFace repositories:

  1. nvidia/DeepSeek-V4-Flash-NVFP4 — the official NVIDIA NVFP4 quantization, which should have proper tensor-core kernel paths built in
  2. canada-quant/DeepSeek-V4-Flash-W4A16-FP8-MTP — a third-party quantization with a warning about potential security risks This is a classic engineering trade-off: build a custom solution vs. use an existing, well-tested one. The user is advocating for the latter, and for good reason. The official NVIDIA NVFP4 checkpoint is the "proper" quantization because it was created by the hardware vendor themselves, with access to the full cutlass kernel library and the ability to generate tensor-core paths that the assistant's custom re-quantization could only approximate. The user's phrasing—"for nvfp we want proper quants"—implies a philosophy: NVFP4 (NVIDIA's FP4 format) is not just any quantization scheme; it is a specific format that requires specific kernel support. Using a properly quantized checkpoint ensures that the model's weights are laid out in memory exactly as the tensor-core kernels expect them. A load-time re-quantization from MXFP4 (the format used by the stock DeepSeek checkpoint) to NVFP4 might produce numerically identical values but in a memory layout that the fast kernels don't recognize or can't use efficiently.

The Security Advisory: Third-Party Checkpoints

The third element is the security warning about the canada-quant repository: "note the latter links some random binaries/forked code, assume it's potentially sketchy." This is a crucial operational consideration. In the world of large language model deployment, using a third-party checkpoint means trusting that the weights haven't been tampered with, that the quantization was performed correctly, and that any accompanying code or binaries don't contain malware.

The user is making a risk assessment: the official NVIDIA checkpoint is trustworthy but may have limitations (e.g., it might not support MTP or might have different performance characteristics). The third-party checkpoint might offer better features (W4A16-FP8-MTP suggests it supports both weight-only quantization and MTP) but carries unknown risks. The user is not saying "don't use it"; they are saying "assume it's potentially sketchy," which means: proceed with caution, verify the source, inspect the code, and understand the security implications before deploying.

This is a lesson in operational security that many ML engineers learn the hard way. HuggingFace repositories can contain arbitrary code, and "random binaries" could be anything. The user's institutional knowledge—knowing which sources are trustworthy and which require scrutiny—is invaluable.

Decisions Made and Assumptions Corrected

The primary decision made in this message is the rejection of the assistant's proposed custom re-quantization approach in favor of using existing properly quantized checkpoints. This decision is based on:

  1. Empirical evidence: The Kimi performance data proves the hardware is capable of much higher throughput, so the bottleneck is in the software stack, not the hardware.
  2. Engineering efficiency: Using existing checkpoints is faster, safer, and more maintainable than building a custom solution.
  3. Trust and provenance: The official NVIDIA checkpoint is the preferred source; third-party checkpoints require security scrutiny. Several assumptions are corrected: - The assumption that ~23 tok/s is a hardware ceiling: Corrected by the Kimi performance data. The ceiling is in the software configuration. - The assumption that custom re-quantization is the right approach: Corrected by pointing to existing proper quants. - The assumption that any HuggingFace checkpoint is equally trustworthy: Corrected by the security warning about third-party binaries.

Input and Output Knowledge

Input knowledge required to understand this message: The reader must understand the context of the preceding optimization campaign—the kernel profiling, the configuration tuning, the discovery that both MoE and attention kernels were running on CUDA cores instead of tensor cores, and the assistant's proposal to re-quantize at load time. They must also understand what NVFP4 and MXFP4 are (different FP4 quantization formats), what MTP (Multi-Token Prediction) speculative decoding is, and the significance of TP8 vs TP4 tensor parallelism. The reference to Kimi K2.6 requires knowledge of that model's architecture (a 1T parameter MoE model) and its prior deployment on this hardware.

Output knowledge created by this message: The message creates several critical pieces of knowledge:

  1. An empirical performance baseline (Kimi at 140 tok/s on this hardware) that serves as a target for DeepSeek-V4-Flash optimization.
  2. The existence and URLs of two properly quantized checkpoints that should use tensor-core paths.
  3. A security assessment of the third-party checkpoint.
  4. A strategic direction: use existing proper quants rather than building custom solutions.

The Thinking Process Visible

While this message is from the user (not the assistant), the thinking process is still visible in its structure. The user is performing a rapid multi-dimensional analysis:

First, they acknowledge the assistant's recommendation ("Go for the highest leverage fixes") while immediately correcting the direction. Then they provide the critical missing data point (Kimi at 140 tok/s) that reframes the problem. This is presented not as a suggestion but as an undeniable fact—the user knows what this hardware can do because they've seen it. The triple "really" conveys frustration but also certainty.

Next, they provide the solution: use proper NVFP4 quants. The two links represent options with different risk profiles. The first (official NVIDIA) is the safe choice. The second (third-party) is potentially higher-performance but comes with a security caveat. The user is giving the assistant a menu with clear recommendations.

The parenthetical warning about "random binaries/forked code" shows operational maturity. The user isn't just thinking about performance; they're thinking about supply chain security, reproducibility, and operational risk. This is the kind of consideration that distinguishes production deployments from experimental setups.

Conclusion

Message 12493 is a masterclass in concise, high-bandwidth technical communication. In three sentences, the user provides a performance reality check, a strategic redirection, a set of concrete resources, and a security advisory. It corrects the assistant's assumptions, provides missing empirical data, and sets a clear path forward. For anyone working on ML infrastructure optimization, this message demonstrates the value of operational experience—the knowledge of what a specific hardware configuration can actually deliver—as a complement to theoretical analysis and profiling. The assistant's kernel profiling was correct and thorough, but it lacked the one data point that would have contextualized the results: the known performance ceiling of the hardware with a similarly-sized model. The user provided that data point, and the entire optimization strategy shifted as a result.