The Blackwell Gauntlet: Deploying Qwen3.5-397B-A17B-NVFP4 Through Stack Upheaval, Backend Testing, and Accuracy Recovery

Introduction

In the high-stakes world of deploying frontier large language models on cutting-edge hardware, the gap between a working prototype and a production-grade inference service is rarely bridged by a single insight. More often, it requires a cascade of strategic decisions, each carrying its own risks and tradeoffs: when to abandon stability for capability, how to navigate a complex dependency graph of nightly builds, which kernel backend produces correct numerical output on novel hardware, and whether to sacrifice memory capacity for numerical accuracy.

This article synthesizes a pivotal segment of an opencode coding session — Segment 39 — in which an AI assistant and its user deployed the Qwen3.5-397B-A17B-NVFP4 model, a 397-billion-parameter Mixture-of-Experts (MoE) architecture quantized to NVIDIA's NVFP4 format, on a machine equipped with 8× NVIDIA RTX PRO 6000 Blackwell Server Edition GPUs (compute capability SM120). The segment spans approximately 170 messages (indices 5856–6026) and covers the full arc from diagnosis to production deployment: a strategic pivot from stable to nightly software stacks, exhaustive backend testing to find working FP4 GEMM and MoE configurations on Blackwell, a critical accuracy recovery when FP8 KV cache was discovered to lack scaling factors, and the final deployment of a production systemd service achieving ~172 tok/s single-request and over 2,100 tok/s aggregate throughput.

What makes this segment particularly instructive is the interplay between two distinct engineering challenges. The first was making the model run at all — the stack upgrade, kernel compilation, and backend selection required navigating a complex dependency graph where each component (PyTorch, FlashInfer, sgl-kernel, SGLang) had to be ABI-compatible with its neighbors. The second was making the model run correctly — the FP8 KV cache accuracy fix required understanding the model's hybrid attention architecture, calculating memory tradeoffs, and prioritizing long-context fidelity over marginal throughput gains. The segment thus offers a masterclass in both systems engineering and numerical accuracy awareness.

The Crisis: Garbage Output on Blackwell GPUs

The segment opens at a moment of frustration. The assistant had successfully deployed Kimi-K2.5 INT4 with EAGLE-3 speculative decoding in prior segments, achieving impressive throughput numbers. But the pivot to Qwen3.5-397B-A17B-NVFP4 hit a wall. When the assistant started the server with --attention-backend triton --quantization modelopt_fp4 and sent a test prompt, the model responded with repeated exclamation marks — pure garbage. A second test returned null content with a finish_reason of "NaN happened" ([msg 5843], [msg 5844]). The FP4 kernels were producing numerical garbage on the SM120 (Blackwell) hardware.

The root cause was identified through a critical external reference: a GitHub Gist by developer "catid" at https://gist.github.com/catid/87cca824963f17fe7479a0ed26221397 ([msg 5853]). The gist documented that the default auto MoE runner backend selects a kernel that produces NaN on SM120, and that the fix requires explicitly pinning --moe-runner-backend flashinfer_cutlass and --fp4-gemm-runner-backend flashinfer_cudnn. But there was a deeper concern: the pre-built sgl-kernel 0.3.21+cu130 wheel might not contain SM120-specific FP4 kernels at all, because it was compiled against PyTorch 2.9.1 without Blackwell architecture support.

The State Dump: Consolidating Knowledge Before Action

Before diving into the fix, the assistant produced one of the most remarkable messages in the entire session: a comprehensive state document at [msg 5856] that consolidated everything it knew about the deployment environment. This "state dump" spanned hardware configuration (8× RTX PRO 6000 Blackwell GPUs, PCIe Gen5, no NVLink), software stack versions (PyTorch 2.9.1+cu130, sgl-kernel 0.3.21+cu130, flashinfer 0.6.4, CUDA 13.0.1), ABI compatibility notes, NCCL tuning parameters, GPU switching workflows between LXC containers and VMs, benchmark results from the completed Kimi-K2.5 work, model characteristics of Qwen3.5, and a detailed action plan with four prioritized steps.

This message is remarkable because it represents the assistant's metacognitive awareness of its own knowledge boundaries. The assistant recognized that it was about to undertake a complex, multi-step process and needed a consolidated reference to avoid losing track of version numbers, file paths, and configuration parameters across dozens of subsequent messages. The document served as an external memory system — a cognitive artifact that the assistant could refer back to throughout the upgrade process.

The state dump also revealed the assistant's assumptions and risk awareness. It explicitly noted that "torch 2.10.0+cu130 and nightly 2.12.0+cu130 have ABI break with sgl-kernel 0.3.21" and cautioned that building sgl-kernel from source "will likely conflict with torch 2.9.1 since catid uses nightly 2.12.0." The assistant's conservative plan was to try the simpler backend flag fix first, and only escalate to a source build if needed.

The User's Directive: "Update All to Nightly"

The user's response at [msg 5857] cut through this cautious, multi-step plan with a single, unambiguous command: "Update all to nightly, the patches are more recent than our build." This was not a suggestion — it was a directive that fundamentally altered the trajectory of the session.

The reasoning embedded in this seven-word message is worth unpacking. The user had absorbed the same information the assistant had: catid's gist was the authoritative reference for making NVFP4 inference work on Blackwell, and that gist explicitly built sgl-kernel from source against PyTorch nightly. The user recognized a logical inconsistency in the assistant's plan: if the patches (the CMakeLists.txt modifications, the CUDA 13 cccl include path fixes, the FA3 fallback import) were designed for the nightly build, then trying to apply them to the stable stack would be fighting against the grain. The ABI incompatibility the assistant worried about was not a bug to be worked around — it was a signal that the entire stack needed to move forward together.

The user followed up at [msg 5858] with a second message that defined the operational boundaries: "Continue if you have next steps, or stop and ask for clarification if you are unsure how to proceed." This "permission slip" simultaneously granted autonomy, established a fallback protocol, and validated the assistant's expertise. In 16 words, it accomplished what many project managers struggle to achieve in paragraphs.

The Pivot: From Conservative to Aggressive

The assistant's response at [msg 5859] was the acceptance and execution plan for this directive. The message announced: "I'll update the entire stack to PyTorch nightly and rebuild sgl-kernel from source with SM120 support. The rationale is that the pre-built sgl-kernel 0.3.21+cu130 was built against older torch, and catid's gist (which works) uses nightly."

This was followed by a structured todowrite with four high-priority tasks:

  1. Upgrade PyTorch to nightly (2.12.0+cu130 or latest)
  2. Upgrade flashinfer to latest nightly/compatible version
  3. Build sgl-kernel from source with SM120 FP4 support (TORCH_CUDA_ARCH_LIST=12.0a)
  4. Apply catid's CMakeLists.txt patches to sgl-kernel The assistant's reasoning revealed a sophisticated understanding of the dependency chain. The key insight was that the ABI incompatibility between nightly PyTorch and sgl-kernel 0.3.21 existed only because the pre-built wheel was compiled against an older torch version. If you build sgl-kernel from source against the same nightly PyTorch, the ABI mismatch disappears. The constraint was not the torch version itself, but the pre-built binary. This is a classic systems-engineering tradeoff: stability versus capability. The stable stack (torch 2.9.1) was safe but couldn't run the model correctly. The nightly stack (torch 2.12.0) was risky but was the only path demonstrated to work on SM120. The assistant correctly identified that the risk could be mitigated by rebuilding sgl-kernel from source.## Step 1: Upgrading PyTorch to Nightly The first concrete action came at [msg 5863], where the assistant executed a dry-run upgrade using uv pip install --dry-run against the PyTorch nightly index for CUDA 13.0. The --dry-run flag was a deliberate choice — it allowed the assistant to preview what would change without actually modifying the environment. The output revealed 33 packages would change, including CUDA bindings jumping from 12.9.4 to 13.0.3. At [msg 5864], the assistant filtered the dry-run output to the critical line: torch==2.12.0.dev20260307+cu130 would replace torch==2.9.1+cu130. This single data point — confirming that the nightly version existed and was accessible — was the green light needed to proceed. The actual installation at [msg 5865] used uv pip install --upgrade --force-reinstall against the nightly index. The output showed a dramatic version jump: torch from 2.9.1 to 2.12.0.dev, torchaudio from 2.9.1 to 2.11.0.dev, torchvision from 0.24.1 to 0.26.0.dev, and triton from 3.5.1 to 3.6.0+git. The NVIDIA CUDA libraries were also upgraded — nvidia-nvjitlink from 13.0.39 to 13.0.88, nvidia-nvshmem-cu13 from 3.3.24 to 3.4.5. The assistant labeled this "Step 1" and proceeded methodically. The verification at [msg 5866] was minimal but decisive: a three-line Python script confirming torch.__version__ as 2.12.0.dev20260307+cu130, torch.cuda.is_available() as True, and the GPU as NVIDIA RTX PRO 6000 Blackwell Server Edition. The nightly PyTorch was operational on the target hardware.

Step 2: The FlashInfer Gauntlet

The flashinfer upgrade proved far more challenging than PyTorch. At [msg 5868], the assistant tried uv pip install --dry-run flashinfer and got a hard failure: "No solution found when resolving dependencies — Because flashinfer was not found in the package registry." The package simply didn't exist on PyPI under that name.

At [msg 5869], the assistant tried flashinfer-python (the actual PyPI name). The dry-run revealed a critical conflict: installing flashinfer-python from PyPI would downgrade torch from 2.12.0.dev20260307+cu130 to 2.10.0. This was a catastrophic regression — the entire point of the nightly upgrade was to get Blackwell support, and torch 2.10.0 doesn't even target CUDA 13. The assistant's cautious dry-run approach saved it from silently undoing the entire stack upgrade.

At [msg 5870], the assistant tried the FlashInfer nightly wheel index at https://flashinfer.ai/whl/cu130/torch2.12/nightly/. The resolver returned "No solution found" — the package wasn't in that registry either. The assistant noted: "As expected — flashinfer-python from PyPI would downgrade torch to 2.10.0. We need a nightly flashinfer build."

At [msg 5871], the assistant probed the URL directly with curl and discovered a GitHub Pages 404 page. The nightly index for torch 2.12 simply didn't exist. This was a definitive answer: the FlashInfer project had not published pre-built wheels for this torch version.

At [msg 5872], the assistant broadened the search. It checked the root CUDA 130 index at https://flashinfer.ai/whl/cu130/ and found a single subdirectory: flashinfer-jit-cache/. This was the breakthrough. The JIT-cache approach is torch-version-independent — kernels are compiled at runtime against whatever torch is installed. The assistant also checked for a torch2.11 index (also a 404), confirming that no pre-built wheels existed for any torch version under CUDA 130.

The resolution came at [msg 5874] through a flash of insight. The assistant realized that flashinfer-jit-cache wheels are torch-version-independent — they contain pre-compiled CUDA kernels that don't link against specific torch binaries. By upgrading just the JIT-cache and cubin components (which are decoupled from the torch version), the assistant could get FlashInfer 0.6.5 working with the nightly PyTorch without any dependency conflict. The subsequent messages confirmed this approach: flashinfer-jit-cache 0.6.5+cu130 was installed, and the package remained functional.

Step 3: Building sgl-kernel from Source with SM120 Support

The sgl-kernel build was the most technically complex step in the entire upgrade. The assistant had to navigate a multi-layered build system, apply patches from catid's gist, and compile CUDA kernels specifically for the Blackwell (SM120) architecture.

At [msg 5882], the assistant checked the current state of the SGLang repository, noting that it was on the main branch with custom SM120 patches applied to all_reduce_utils.py and torch_symm_mem.py. The assistant recognized that pulling the latest upstream changes could overwrite these patches, so it planned to stash them first.

At [msg 5884], the assistant executed git stash to save the custom patches, then pulled the latest upstream changes from SGLang's main branch. The output showed a fast-forward merge with changes to multiple files. The assistant then popped the stash to reapply the SM120 patches on top of the updated code.

At [msg 5886], the assistant checked whether upstream SGLang had added native SM120 support. A grep for 12_0 in the sgl-kernel source revealed that upstream had not yet added Blackwell support — the patches were still necessary.

The build process began at [msg 5897] with the assistant examining the sgl-kernel build system. The pyproject.toml revealed that the build used scikit-build-core with CMake, and the CMakeLists.txt contained the architecture targeting logic. The assistant identified the need to set TORCH_CUDA_ARCH_LIST=12.0a to enable Blackwell-specific code paths.

The first build attempt at [msg 5902] failed with a Python interpreter mismatch — the build system was using the system Python 3.12 instead of the virtual environment's Python. The assistant diagnosed this and set the correct PYTHON path.

The second attempt at [msg 5906] progressed further but failed with a CMake policy error related to CMP0169 and CMP0177. This was exactly the issue catid's patches addressed. The assistant applied the patches: adding CMake policy guards, fixing CUDA 13 cccl include paths, and adding FA3 import fallback.

The successful build at [msg 5908] used:

export TORCH_CUDA_ARCH_LIST=12.0a
export CMAKE_ARGS='-DSGL_KERNEL_COMPILE_THREADS=16 -DENABLE_BELOW_SM90=OFF -DSGL_KERNEL_ENABLE_FP4=ON -DSGL_KERNEL_ENABLE_FA3=OFF'

The compilation completed, producing a custom sgl-kernel with Blackwell FP4 support.

The verification at [msg 5909] confirmed that the built wheel contained SM120 kernels. The assistant checked the compiled .so files for Blackwell architecture markers and ran a quick import test to ensure the module loaded correctly.

The assistant then rebuilt SGLang itself against the nightly PyTorch stack at [msg 5912], ensuring that the entire inference engine was ABI-compatible with the new PyTorch and sgl-kernel versions.

The Verification: Everything Loads Correctly

At [msg 5913], the assistant performed the definitive verification: importing all critical packages and confirming they loaded without errors. The output showed torch==2.12.0.dev20260307+cu130, sgl_kernel==0.3.21 (custom build), flashinfer==0.6.5, and SGLang importing cleanly. This was the moment when the nightly stack was confirmed operational.

Step 4: Backend Testing — Finding the Working Configuration

With the stack upgraded, the assistant turned to the critical question: which FP4 GEMM and MoE backends actually produce correct output on SM120? The assistant conducted extensive testing, launching the server with different backend combinations and checking for NaN/garbage output.

The backend testing was exhaustive. The assistant systematically tested each combination:

Establishing the Throughput Baseline

After the backend testing was complete, the assistant needed to establish a proper throughput baseline. A quick smoke test using the chat completions API had reported approximately 72 tok/s, but the assistant immediately distrusted this number, recognizing that API-level measurements include serialization overhead, tokenization, and response formatting that inflate latency.

The assistant's first critical move was to retrieve an existing benchmark script from a prior project directory (benchmark_parallel.py in the eagle3-train folder) and adapt it for the Qwen3.5 model ([msg 5995], [msg 5996]). The adaptation was minimal — changing the model name from "default" to "qwen3.5-397b" — but it was essential. Without this fix, the benchmark would have failed silently against the server's API router.

The benchmark results, when they arrived at [msg 5997], were transformative:

   C |  Agg tok/s |  Per-req tok/s |   Tokens |  Wall(s) |   Ok
-----------------------------------------------------------------
   1 |      172.2 |          176.2 |     4000 |     23.2 |    4
   4 |      542.2 |          159.4 |     7245 |     13.4 |    8
  16 |     1646.7 |          120.3 |    29567 |     18.0 |   32
  32 |     2156.4 |           81.2 |    52337 |     24.3 |   64

The real single-request throughput was 172 tok/s — nearly 2.4× the smoke test's 72 tok/s. At high concurrency, the system delivered 2,156 tok/s aggregate, which the assistant noted was already competitive with published results from catid (1,373 tok/s on 4 GPUs). The assistant's interpretation at [msg 5998] was immediate and precise: "That's much better than the 72 tok/s we saw earlier — the smoke test was measuring with chat completions API overhead."

This moment established the baseline against which all subsequent optimizations would be measured. It also validated the extensive backend testing effort: the flashinfer_cutlass + flashinfer_cudnn combination was not just correct but performant.

The Speculative Decoding Detour: NEXTN MTP

With a strong baseline established, the assistant turned to the next optimization on the todo list: NEXTN Multi-Token Prediction (MTP) speculative decoding ([msg 6000]). The Qwen3.5 checkpoint includes built-in MTP heads — lightweight prediction layers that can speculate multiple tokens ahead in a single forward pass, with the full model verifying them in parallel. This promised to boost single-stream throughput without requiring a separate draft model.

The path to launching NEXTN was not smooth. Three previous attempts had failed: the first crashed with an assertion error about speculative_eagle_topk conflicting with spec_v2; the second required adding --mamba-scheduler-strategy extra_buffer for the hybrid GDN architecture; the third was killed by the system. At [msg 6001], the assistant assembled the correct incantation — SGLANG_ENABLE_SPEC_V2=1 with --speculative-algorithm NEXTN and --mamba-scheduler-strategy extra_buffer — and launched the server.

The server loaded successfully. Logs showed draft CUDA graphs being captured in under 3 seconds per GPU, and the health endpoint returned "healthy." The benchmark was run at the same concurrency levels.

The results, at [msg 6004], were devastating for speculative decoding:

   C |  Agg tok/s |  Per-req tok/s |   Tokens |  Wall(s) |   Ok
-----------------------------------------------------------------
   1 |      172.1 |          176.1 |     4000 |     23.2 |    4
   4 |      541.9 |          159.4 |     7245 |     13.4 |    8
  16 |     1686.1 |          119.5 |    29567 |     17.5 |   32
  32 |     2162.3 |           82.1 |    52593 |     24.3 |   64

These numbers were essentially identical to the baseline — 172.1 vs 172.2 tok/s at C=1, 2,162.3 vs 2,156.4 at C=32. The MTP heads, despite being loaded and operational, contributed nothing to throughput.

The assistant's diagnostic reasoning at [msg 6006] was sharp: "The NEXTN numbers are essentially identical to baseline — the MTP isn't helping because with 1000 random token IDs the draft acceptance rate is probably near zero." The benchmark script used randomly generated token IDs for its prompts, providing no predictable structure for the draft model to exploit. With near-zero acceptance, NEXTN degenerated to baseline decoding. The assistant proposed testing with real text but prioritized the user's concurrent directive to "try up to 1024" concurrency, running the benchmark at expanded levels.

This detour into speculative decoding, while ultimately unproductive for throughput, was not wasted. It confirmed that the baseline configuration was already near-optimal for this hardware, and it established that NEXTN could be safely dropped from the production configuration to reduce complexity.## The Accuracy Pivot: Discovering the FP8 KV Cache Bug

The session might have ended with the deployment declared complete — the throughput numbers were excellent, the system was stable, and speculative decoding had been tested and dismissed. But then came the message that changed everything.

At [msg 6007], the user asked a question that revealed deep domain understanding: "Are we doing anything that could be reducing model accuracy? NVFP4 is all we can fit on those GPUs, but things like context should use no quantization because the usecase is long context hard agentic coding."

This question distinguished between two forms of quantization: weight quantization (NVFP4 on model parameters, accepted as necessary) and context quantization (the KV cache, potentially avoidable). The user's intuition was that for long-context agentic coding — where the model must maintain precise representations of code, variable names, and reasoning chains across tens of thousands of tokens — KV cache quantization errors would compound destructively.

The assistant's investigation at [msg 6008] through [msg 6010] confirmed the user's suspicion was correct. The checkpoint's hf_quant_config.json specified kv_cache_quant_algo: "FP8", and SGLang was auto-detecting this and enabling FP8 KV cache. Worse, the server logs contained a damning message: "Using FP8 KV cache but no scaling factors provided. Defaulting to scaling factors of 1.0."

This was a double accuracy hit. First, FP8 (specifically the E5M2 format used for KV cache) provides only 2 bits of mantissa for the key and value vectors — a drastic reduction from BF16's 7 mantissa bits. Each stored value is a coarse approximation of the original. Second, without calibrated scaling factors, the dynamic range was effectively arbitrary: values outside the FP8 representable range would simply clip, introducing non-linear distortions into the attention computation. For a model processing 262K-token contexts, these errors would compound across every attention layer, potentially destroying the model's ability to maintain coherent long-range dependencies.

The assistant's assessment was unequivocal: "For long-context agentic coding, this is very bad."

The Precision Calculus: Can We Afford BF16?

Identifying the problem was only half the battle. The assistant now needed to determine whether the fix — forcing BF16 KV cache via --kv-cache-dtype bf16 — was feasible within the VRAM budget.

The assistant performed a detailed memory calculation. The Qwen3.5 architecture has 60 layers, but only 15 are full-attention layers (every 4th layer) that store KV cache. With 2 KV heads, a head dimension of 256, and tensor parallelism across 8 GPUs (where KV heads are replicated since 2 < 8), the per-token cost was:

The Deployment: From Service File to Production

The fix required updating the systemd service file that controlled the SGLang server. The assistant read the current service file, wrote the updated version with --kv-cache-dtype bf16 added to the server arguments, copied the file to the remote server via scp, and started the service.

But the old server refused to die. GPU memory was still showing 83 GB consumed per GPU — the kill commands had failed. This required escalation: the assistant connected to the Proxmox hypervisor host and used pct exec 129 to force-kill all Python processes with SIGKILL from the container management layer. This revealed an important architectural detail — the SGLang server was running inside a Proxmox container, not directly on bare metal.

The server took approximately 75 seconds to become healthy, a startup time consistent with model loading, CUDA graph capture, and memory allocation across 8 GPUs. The assistant then performed a series of verification steps, using journalctl to grep the service logs for KV cache configuration. The initial query returned empty results because the logs were from the previous launch; the assistant refined the query to check only recent logs.

The final confirmation showed:

The Production Configuration

The final production configuration was codified into a systemd service file at /etc/systemd/system/sglang-qwen.service. The service included all the critical flags discovered through the testing process:

The Empty Message That Closed the Loop

The segment ends with a remarkable moment: the user sends an empty message at [msg 6025]. No text, no command, no question. In the context of the conversation, this silence was eloquent. It signaled implicit approval of the accuracy fix, a transfer of initiative to the assistant, and satisfaction — or at least acceptance — with the deployment state.

The assistant's response at [msg 6026] was a comprehensive state dump document — "Goal / Instructions / Discoveries / Accomplished" — that consolidated every configuration decision, benchmark result, and lesson learned across the entire deployment effort. This document served as a shared artifact, ensuring both parties had a complete understanding of the system's state before proceeding to the next phase.

Themes and Lessons

Several overarching themes emerge from this segment:

Strategic pivoting. The user's directive to "update all to nightly" was a decisive strategic intervention that overrode the assistant's conservative plan. The user recognized that the patches were designed for the nightly ecosystem and that trying to apply them to a stable stack would create more problems than it solved. This illustrates the value of human judgment in AI-assisted engineering — knowing when to abandon caution for coherence.

Methodical execution. Despite the aggressive strategic direction, the assistant's execution was remarkably methodical. Every upgrade was preceded by a dry-run, every installation was verified, and every configuration change was tested. The use of --dry-run flags, structured todo lists, and incremental verification steps prevented catastrophic failures at multiple points — most notably when the FlashInfer PyPI package would have silently downgraded PyTorch.

Dependency graph navigation. The upgrade path revealed the complex dependency graph of modern ML infrastructure. PyTorch → FlashInfer → sgl-kernel → SGLang forms a chain where each component must be ABI-compatible with its neighbors. The assistant's decision to build sgl-kernel from source against the nightly PyTorch was the key insight that made the entire upgrade viable — it resolved the ABI incompatibility that had blocked earlier attempts.

Empirical debugging. When assumptions about package availability failed (the FlashInfer nightly index didn't exist), the assistant pivoted to empirical probing — using curl to directly inspect the wheel server, discovering the JIT-cache approach, and adapting the strategy accordingly. This willingness to probe the environment directly rather than relying on documentation or assumptions was critical to the success.

Accuracy over performance. The FP8 KV cache fix exemplifies the assistant's commitment to correctness over marginal performance gains. The checkpoint's default FP8 KV cache lacked proper scaling factors, which would degrade long-context inference quality. The assistant proactively forced BF16, accepting a slight memory increase in exchange for guaranteed numerical accuracy.

The primacy of measurement. The assistant's first action after establishing a working configuration was to build a proper benchmark. The smoke test had suggested 72 tok/s; the real benchmark revealed 172 tok/s. Without this measurement, the team might have spent hours optimizing a "72 tok/s" baseline that didn't exist. The lesson is universal: measure before you optimize, and use the right measurement tool.

The danger of silent degradation. The FP8 KV cache bug was invisible to throughput benchmarks. The server ran, the model generated text, and the throughput numbers were excellent. But every inference was being silently degraded by uncalibrated quantization. The only reason it was caught was the user's domain knowledge and willingness to ask a hard question at the moment of apparent success. This is a reminder that in production ML, the most dangerous bugs are the ones that don't crash anything.

The value of architectural understanding. The assistant's ability to reason about the KV cache tradeoff depended on deep knowledge of the Qwen3.5 architecture — the hybrid attention mechanism, the layer distribution, the KV head replication under GQA. Without this understanding, the memory calculation would have been wrong, and the decision to force BF16 might have seemed reckless. Architecture-aware reasoning is what separates a surface-level deployment from a deeply informed one.

The discipline of verification. After every configuration change, the assistant verified. The journalctl grep, the nvidia-smi check, the smoke test — these were not optional extras. They were the mechanism by which the assistant confirmed that intended changes had actual effects. In a complex system with multiple layers of abstraction (checkpoint configs, server defaults, service files, runtime behavior), verification is the only way to close the loop between intention and reality.

Knowing when to stop. The assistant could have spent days trying to calibrate FP8 scaling factors, or testing NEXTN with real text, or optimizing all-reduce algorithms. Instead, it recognized that the baseline was already excellent, the accuracy fix was essential and affordable, and further optimization would yield diminishing returns. The decision to deploy — to ship the current configuration and move on — required judgment as much as technical skill.

Conclusion

Segment 39 of this opencode session is a microcosm of production ML engineering. It contains moments of discovery (the real throughput was 2.4× the smoke test), moments of disappointment (NEXTN provided no benefit), moments of alarm (FP8 KV cache with no scaling factors), and moments of resolution (BF16 KV cache deployed and verified). Through it all, the assistant maintained a methodical, evidence-driven approach — forming hypotheses, gathering data, calculating tradeoffs, and verifying outcomes.

The final production configuration — flashinfer_cutlass for MoE, flashinfer_cudnn for FP4 GEMM, BF16 KV cache, no speculative decoding — achieved approximately 172 tok/s at single-request concurrency and over 2,100 tok/s aggregate at high concurrency. More importantly, it achieved these numbers without silently degrading the model's accuracy. For the demanding use case of long-context agentic coding, that distinction matters immensely.

The journey from throughput to accuracy was not a detour. It was the path that led to a production deployment worthy of the name. The collaboration between human and AI — the user providing strategic direction and domain expertise, the assistant executing methodically and reasoning about complex tradeoffs — proved to be a powerful combination for navigating the landscape of nightly builds, custom kernel compilation, architecture-specific optimization, and numerical accuracy awareness.

References

[1] Chunk article: "The Nightly Pivot: Deploying Qwen3.5-397B-A17B-NVFP4 on Blackwell GPUs Through Strategic Stack Upheaval" — [chunk 39.0]

[2] Chunk article: "From Throughput to Accuracy: The Qwen3.5-397B NVFP4 Deployment on 8× Blackwell GPUs" — [chunk 39.1]