From Throughput to Accuracy: The Qwen3.5-397B NVFP4 Deployment on 8× Blackwell GPUs

Introduction

In the high-stakes world of production LLM inference, the journey from a working prototype to a reliable production service is rarely a straight line. It is a path marked by false assumptions, silent degradation, and the constant tension between raw throughput and output quality. This article synthesizes a pivotal segment of an opencode coding session — Segment 39 — in which a team deployed the Qwen3.5-397B-A17B-NVFP4 model, a 397-billion-parameter Mixture-of-Experts (MoE) architecture, across eight NVIDIA RTX PRO 6000 Blackwell GPUs connected via PCIe Gen5 with no NVLink fabric. What began as a performance optimization campaign evolved into a deeper investigation of numerical accuracy, culminating in a production deployment that prioritized correctness over marginal throughput gains.

The segment spans roughly thirty messages (indices 5994–6025) and covers: establishing a rigorous throughput baseline, testing and discarding speculative decoding, discovering a silent FP8 KV cache accuracy bug, calculating the memory tradeoffs of the fix, and deploying a production systemd service with BF16 KV cache. The final configuration achieved approximately 172 tok/s at single-request concurrency and over 2,100 tok/s aggregate at high concurrency — numbers that validated the extensive software stack engineering that preceded them.

The Hardware and Software Context

The deployment target was an 8× RTX PRO 6000 Blackwell GPU system. These GPUs, based on NVIDIA's Blackwell architecture (compute capability SM120), each offer 96 GB of VRAM and support advanced quantization formats including NVFP4 — a 4-bit floating-point format that packs two FP4 values into a single byte. However, the GPUs were connected via PCIe Gen5 without NVLink, meaning all inter-GPU communication — including the all-reduce operations required for tensor-parallel inference — traversed the PCIe bus. This topology made communication overhead the dominant bottleneck, a fact that shaped every optimization decision.

The software stack was equally exotic. The assistant had upgraded to nightly builds of PyTorch 2.12.0.dev20260307+cu130, FlashInfer 0.6.5, and the latest SGLang main branch. Critically, the sgl-kernel library — which provides the FP4 GEMM kernels for Blackwell — had to be compiled from source with SM120 support, applying patches from catid for CMake policy guards, CUDA 13 cccl include paths, and FA3 fallback. This was not a turnkey deployment; it was a hands-on, build-from-source operation.

The model itself, Qwen3.5-397B-A17B-NVFP4, is a hybrid architecture combining 60 transformer layers with a Mixture-of-Experts feedforward network. Only 17 billion of its 397 billion parameters are activated per token. Critically for the KV cache analysis that would follow, only 15 of the 60 layers (every 4th layer) use traditional full attention with KV cache; the remaining 45 layers use linear attention with GDN (Gated Differential Network) recurrent state, which does not require KV cache storage. This architectural detail would prove essential to the accuracy investigation.

Establishing the Baseline: The Benchmarking Pivot

The segment opens with the assistant at a decision point. After extensive backend testing — eliminating flashinfer_trtllm (which crashed on SM120) and flashinfer_cutedsl (which produced garbage output) — the assistant had settled on a working configuration: flashinfer_cutlass for the MoE runner backend and flashinfer_cudnn for the FP4 GEMM backend, with triton for attention. A quick smoke test using the chat completions API had reported approximately 72 tok/s, but the assistant immediately distrusted this number, recognizing that API-level measurements include serialization overhead, tokenization, and response formatting that inflate latency.

The assistant's first critical move was to retrieve an existing benchmark script from a prior project directory (benchmark_parallel.py in the eagle3-train folder) and adapt it for the Qwen3.5 model ([msg 5995], [msg 5996]). The adaptation was minimal — changing the model name from "default" to "qwen3.5-397b" — but it was essential. Without this fix, the benchmark would have failed silently against the server's API router.

The benchmark results, when they arrived in [msg 5997], were transformative:

   C |  Agg tok/s |  Per-req tok/s |   Tokens |  Wall(s) |   Ok
-----------------------------------------------------------------
   1 |      172.2 |          176.2 |     4000 |     23.2 |    4
   4 |      542.2 |          159.4 |     7245 |     13.4 |    8
  16 |     1646.7 |          120.3 |    29567 |     18.0 |   32
  32 |     2156.4 |           81.2 |    52337 |     24.3 |   64

The real single-request throughput was 172 tok/s — nearly 2.4× the smoke test's 72 tok/s. At high concurrency, the system delivered 2,156 tok/s aggregate, which the assistant noted was already competitive with published results from catid (1,373 tok/s on 4 GPUs). The assistant's interpretation in [msg 5998] was immediate and precise: "That's much better than the 72 tok/s we saw earlier — the smoke test was measuring with chat completions API overhead."

This moment established the baseline against which all subsequent optimizations would be measured. It also validated the extensive backend testing effort: the flashinfer_cutlass + flashinfer_cudnn combination was not just correct but performant.

The Speculative Decoding Detour: NEXTN MTP

With a strong baseline established, the assistant turned to the next optimization on the todo list: NEXTN Multi-Token Prediction (MTP) speculative decoding ([msg 6000]). The Qwen3.5 checkpoint includes built-in MTP heads — lightweight prediction layers that can speculate multiple tokens ahead in a single forward pass, with the full model verifying them in parallel. This promised to boost single-stream throughput without requiring a separate draft model.

The path to launching NEXTN was not smooth. Three previous attempts had failed: the first crashed with an assertion error about speculative_eagle_topk conflicting with spec_v2 ([msg 5985]); the second required adding --mamba-scheduler-strategy extra_buffer for the hybrid GDN architecture ([msg 5983]); the third was killed by the system ([msg 5989]). In [msg 6001], the assistant assembled the correct incantation — SGLANG_ENABLE_SPEC_V2=1 with --speculative-algorithm NEXTN and --mamba-scheduler-strategy extra_buffer — and launched the server.

The server loaded successfully. Logs showed draft CUDA graphs being captured in under 3 seconds per GPU ([msg 6002]), and the health endpoint returned "healthy" ([msg 6003]). The benchmark was run at the same concurrency levels.

The results, in [msg 6004], were devastating for speculative decoding:

   C |  Agg tok/s |  Per-req tok/s |   Tokens |  Wall(s) |   Ok
-----------------------------------------------------------------
   1 |      172.1 |          176.1 |     4000 |     23.2 |    4
   4 |      541.9 |          159.4 |     7245 |     13.4 |    8
  16 |     1686.1 |          119.5 |    29567 |     17.5 |   32
  32 |     2162.3 |           82.1 |    52593 |     24.3 |   64

These numbers were essentially identical to the baseline — 172.1 vs 172.2 tok/s at C=1, 2,162.3 vs 2,156.4 at C=32. The MTP heads, despite being loaded and operational, contributed nothing to throughput.

The assistant's diagnostic reasoning in [msg 6006] was sharp: "The NEXTN numbers are essentially identical to baseline — the MTP isn't helping because with 1000 random token IDs the draft acceptance rate is probably near zero." The benchmark script used randomly generated token IDs for its prompts, providing no predictable structure for the draft model to exploit. With near-zero acceptance, NEXTN degenerated to baseline decoding. The assistant proposed testing with real text but prioritized the user's concurrent directive to "try up to 1024" concurrency ([msg 6005]), running the benchmark at expanded levels.

This detour into speculative decoding, while ultimately unproductive for throughput, was not wasted. It confirmed that the baseline configuration was already near-optimal for this hardware, and it established that NEXTN could be safely dropped from the production configuration to reduce complexity.

The Accuracy Pivot: Discovering the FP8 KV Cache Bug

The session might have ended with the deployment declared complete — the throughput numbers were excellent, the system was stable, and speculative decoding had been tested and dismissed. But then came the message that changed everything.

In [msg 6007], the user asked a question that revealed deep domain understanding: "Are we doing anything that could be reducing model accuracy? NVFP4 is all we can fit on those GPUs, but things like context should use no quantization because the usecase is long context hard agentic coding."

This question distinguished between two forms of quantization: weight quantization (NVFP4 on model parameters, accepted as necessary) and context quantization (the KV cache, potentially avoidable). The user's intuition was that for long-context agentic coding — where the model must maintain precise representations of code, variable names, and reasoning chains across tens of thousands of tokens — KV cache quantization errors would compound destructively.

The assistant's investigation in [msg 6008] through [msg 6010] confirmed the user's suspicion was correct. The checkpoint's hf_quant_config.json specified kv_cache_quant_algo: "FP8", and SGLang was auto-detecting this and enabling FP8 KV cache. Worse, the server logs contained a damning message: "Using FP8 KV cache but no scaling factors provided. Defaulting to scaling factors of 1.0."

This was a double accuracy hit. First, FP8 (specifically the E5M2 format used for KV cache) provides only 2 bits of mantissa for the key and value vectors — a drastic reduction from BF16's 7 mantissa bits. Each stored value is a coarse approximation of the original. Second, without calibrated scaling factors, the dynamic range was effectively arbitrary: values outside the FP8 representable range would simply clip, introducing non-linear distortions into the attention computation. For a model processing 262K-token contexts, these errors would compound across every attention layer, potentially destroying the model's ability to maintain coherent long-range dependencies.

The assistant's assessment was unequivocal: "For long-context agentic coding, this is very bad."

The Precision Calculus: Can We Afford BF16?

Identifying the problem was only half the battle. The assistant now needed to determine whether the fix — forcing BF16 KV cache via --kv-cache-dtype bf16 — was feasible within the VRAM budget.

The assistant performed a detailed memory calculation in [msg 6011] through [msg 6013]. The Qwen3.5 architecture has 60 layers, but only 15 are full-attention layers (every 4th layer) that store KV cache. With 2 KV heads, a head dimension of 256, and tensor parallelism across 8 GPUs (where KV heads are replicated since 2 < 8), the per-token cost was:

The Deployment: From Service File to Production

The fix required updating the systemd service file that controlled the SGLang server. In [msg 6017], the assistant read the current service file; in [msg 6018], it wrote the updated version with --kv-cache-dtype bf16 added to the server arguments. The file was copied to the remote server via scp in [msg 6019], and the service was started.

But the old server refused to die. In [msg 6015], the assistant discovered that GPU memory was still showing 83 GB consumed per GPU — the kill commands had failed. This required escalation in [msg 6016]: the assistant connected to the Proxmox hypervisor host (10.1.2.6) and used pct exec 129 to force-kill all Python processes with SIGKILL from the container management layer. This revealed an important architectural detail — the SGLang server was running inside a Proxmox container, not directly on bare metal.

The server took approximately 75 seconds to become healthy ([msg 6020]), a startup time consistent with model loading, CUDA graph capture, and memory allocation across 8 GPUs. The assistant then performed a series of verification steps ([msg 6021] through [msg 6023]), using journalctl to grep the service logs for KV cache configuration. The initial query returned empty results because the logs were from the previous launch; the assistant refined the query to check only recent logs.

The final confirmation in [msg 6024] showed:

The Empty Message That Closed the Loop

The segment ends with a remarkable moment: the user sends an empty message at [msg 6025]. No text, no command, no question. In the context of the conversation, this silence was eloquent. It signaled implicit approval of the accuracy fix, a transfer of initiative to the assistant, and satisfaction — or at least acceptance — with the deployment state.

The assistant's response in [msg 6026] was a comprehensive state dump document — "Goal / Instructions / Discoveries / Accomplished" — that consolidated every configuration decision, benchmark result, and lesson learned across the entire deployment effort. This document served as a shared artifact, ensuring both parties had a complete understanding of the system's state before proceeding to the next phase.

Themes and Lessons

Several overarching themes emerge from this segment:

The primacy of measurement. The assistant's first action after establishing a working configuration was to build a proper benchmark. The smoke test had suggested 72 tok/s; the real benchmark revealed 172 tok/s. Without this measurement, the team might have spent hours optimizing a "72 tok/s" baseline that didn't exist. The lesson is universal: measure before you optimize, and use the right measurement tool.

The danger of silent degradation. The FP8 KV cache bug was invisible to throughput benchmarks. The server ran, the model generated text, and the throughput numbers were excellent. But every inference was being silently degraded by uncalibrated quantization. The only reason it was caught was the user's domain knowledge and willingness to ask a hard question at the moment of apparent success. This is a reminder that in production ML, the most dangerous bugs are the ones that don't crash anything.

The value of architectural understanding. The assistant's ability to reason about the KV cache tradeoff depended on deep knowledge of the Qwen3.5 architecture — the hybrid attention mechanism, the layer distribution, the KV head replication under GQA. Without this understanding, the memory calculation would have been wrong, and the decision to force BF16 might have seemed reckless. Architecture-aware reasoning is what separates a surface-level deployment from a deeply informed one.

The discipline of verification. After every configuration change, the assistant verified. The journalctl grep, the nvidia-smi check, the smoke test — these were not optional extras. They were the mechanism by which the assistant confirmed that intended changes had actual effects. In a complex system with multiple layers of abstraction (checkpoint configs, server defaults, service files, runtime behavior), verification is the only way to close the loop between intention and reality.

Knowing when to stop. The assistant could have spent days trying to calibrate FP8 scaling factors, or testing NEXTN with real text, or optimizing all-reduce algorithms. Instead, it recognized that the baseline was already excellent, the accuracy fix was essential and affordable, and further optimization would yield diminishing returns. The decision to deploy — to ship the current configuration and move on — required judgment as much as technical skill.

Conclusion

Segment 39 of this opencode session is a microcosm of production ML engineering. It contains moments of discovery (the real throughput was 2.4× the smoke test), moments of disappointment (NEXTN provided no benefit), moments of alarm (FP8 KV cache with no scaling factors), and moments of resolution (BF16 KV cache deployed and verified). Through it all, the assistant maintained a methodical, evidence-driven approach — forming hypotheses, gathering data, calculating tradeoffs, and verifying outcomes.

The final production configuration — flashinfer_cutlass for MoE, flashinfer_cudnn for FP4 GEMM, BF16 KV cache, no speculative decoding — achieved approximately 172 tok/s at single-request concurrency and over 2,100 tok/s aggregate at high concurrency. More importantly, it achieved these numbers without silently degrading the model's accuracy. For the demanding use case of long-context agentic coding, that distinction matters immensely.

The journey from throughput to accuracy was not a detour. It was the path that led to a production deployment worthy of the name.