From Kimi-K2.5 to Qwen3.5: A Production Odyssey Through Model Deployment, GPU Passthrough, and Blackwell Compatibility

Introduction

The deployment of large language models at production scale is rarely a linear journey. It is a series of pivots, tangents, and discoveries — each revealing something new about the system, the hardware, or the operator's own assumptions. This article synthesizes a remarkable stretch of an opencode coding session that spans the full lifecycle of production ML engineering: from hardening a Kimi-K2.5 INT4 deployment with hierarchical KV cache, through a tangent into GPU passthrough on Proxmox, to a complete pivot toward deploying the newer Qwen3.5-397B-A17B-NVFP4 model on Blackwell GPUs. The narrative arc encompasses systemd service creation, memory hierarchy optimization, kernel module debugging, source code patching, and numerical stability diagnosis — all within a single, continuous conversation between a user and an AI assistant.

This chunk of the session (messages 5738–5855, as documented across 118 articles) represents one of the most densely packed stretches of production ML engineering work imaginable. It is a case study in how modern AI infrastructure is built: not through grand architectural designs, but through iterative, context-aware problem-solving where each discovery feeds the next.

Part I: Hardening the Kimi-K2.5 INT4 Deployment

The session opens with the Kimi-K2.5 INT4 model already running, but not yet hardened for production. The assistant had spent earlier segments benchmarking EAGLE-3 speculative decoding, tuning the optimal topk=1 spec_v2 configuration, and fixing a crash caused by a missing attribute in the dynamic speculation disable patch. Now came the operational work: codifying the setup into a systemd service, adding proper tool call and reasoning parsers, and optimizing memory utilization.

The Systemd Service

The creation of sglang-kimi.service was the first major milestone. This was not a simple process launch — it was a production-grade service definition with TimeoutStartSec=900 (accounting for the 547 GB model's ~11-minute load time), Restart=on-failure for resilience, and carefully tuned NCCL and CUDA environment variables. The service file encoded weeks of operational learning about what it takes to run a 547-billion-parameter model across eight RTX PRO 6000 Blackwell GPUs reliably [7].

The assistant added --tool-call-parser kimi_k2 and --reasoning-parser kimi_k2 to properly structure the API output, ensuring that tool calls and reasoning content were correctly extracted from the model's generation. These seemingly minor flags were critical for production usability — without them, downstream applications consuming the API would receive unstructured text that required custom parsing [7].

The Hierarchical KV Cache Breakthrough

The most impactful optimization in this phase was the hierarchical KV cache (HiCache). The user had noticed a capacity constraint — they were "low on max parallel request context" — and asked about RAM offload options. The assistant launched a deep diagnostic investigation, tracing through SGLang's profile_max_num_token function to understand exactly how KV cache memory was being allocated [1].

The discovery was striking. With mem_fraction_static=0.88, SGLang reserved 12% of total GPU memory as static headroom regardless of actual usage, leaving 7.4 GB per GPU effectively wasted. Meanwhile, the machine had 396 GB of free system RAM — an enormous reservoir sitting completely idle. The assistant presented four options, and the user's immediate response — "What about hicache?" — zeroed in on the hierarchical cache approach [1].

The assistant acted decisively, updating the systemd service file with four new flags: --enable-hierarchical-cache, --hicache-ratio 2.0, --hicache-write-policy write_through, and --hicache-io-backend kernel. Then came the user's four-word question that would double the allocation: "can we do ratio 4?" [4]. This single question reflected a sophisticated mental calculation: with 396 GB free and ratio 2.0 using ~166 GB, the user intuitively recognized that more capacity was available. The assistant immediately updated the ratio to 4.0, and the server restarted with 358 GB of pinned host memory allocated as a secondary KV cache tier [12].

The verification of this deployment was itself a learning experience. The assistant had to navigate a timed-out restart command (the model load exceeded the bash tool's 15-second timeout), a "deactivating" status indicating the old process was still shutting down, and finally the confirmation that the service was "active (running)" with the new configuration [7]. The journal logs showed Allocating 44.77 GB host memory for hierarchical KV cache on each TP thread — confirmation that the massive allocation was succeeding [12].

Part II: The GPU Passthrough Tangent

Just as the Kimi-K2.5 deployment reached a stable, hardened state, the user pivoted to a "tangent experiment": passing all eight GPUs to a Proxmox VM instead of using the LXC container. This seemingly simple request would expose layers of complexity about NVIDIA driver architecture, Proxmox PCI passthrough mechanics, and the Blackwell GPU's unique requirements [15].

The Clarifying Question

The user's initial question was straightforward: "Because it's now a VM and not a container do we need to unload nvidia driver on host? Blacklist and reboot? How can we make this sort of switch reasonably painless?" The assistant's response was a masterclass in structured clarification. Rather than answering the surface-level "how" question, it reframed the problem at a strategic level: "are you passing through all 8 GPUs to the VM, or just some of them? And do you need the LXC container to keep running simultaneously?" [16].

This was not pedantry — it was essential risk management. The GPUs could only be bound to one driver at a time (either the host's nvidia driver for LXC access, or vfio-pci for VM passthrough). Without clarifying the allocation model, any advice the assistant gave could break the production setup. The user answered "All 8 GPUs to VM," and the assistant proceeded with a clear understanding of the constraints [16].

The Discovery That Proxmox Already Solved It

The assistant's investigation revealed a much more favorable reality than expected. The Proxmox host already had a working PCI mapping configuration (/etc/pve/mapping/pci.cfg with a mapping called pro6000), and VM 131 (ml-pipelines) was already configured with eight hostpci entries. The key insight was that Proxmox handles the driver switch automatically — starting a VM with hostpci entries unbinds the NVIDIA driver and binds vfio-pci, and stopping the VM reverses the process. No blacklisting, no rebooting, no manual module unloading [22].

The assistant confirmed this by checking lspci -ks 01:00.0 on the host, which showed "Kernel driver in use: vfio-pci" — the GPUs were already detached from the NVIDIA driver. The NVIDIA kernel modules were still loaded in memory (as shown by lsmod), but they had zero references to any devices [22].

The Blackwell Open Kernel Module Requirement

But when the assistant checked the VM guest, a different problem emerged. The GPUs were visible in lspci and /dev/nvidia* devices existed, but nvidia-smi returned "No devices were found." The kernel messages revealed the culprit: NVRM: installed in this system requires use of the NVIDIA open kernel modules [29].

This was a critical discovery about Blackwell GPU architecture. The proprietary NVIDIA kernel module (nvidia-dkms-590) cannot initialize Blackwell GPUs — they require the open kernel module (nvidia-dkms-590-open). This is a relatively recent change in NVIDIA's driver strategy: starting with the SM120 architecture, the open kernel module is mandatory. The fix was a single package installation: apt-get install nvidia-dkms-590-open, which replaces the proprietary module via DKMS rebuild [29][31].

After a VM reboot, all eight GPUs came to life. The nvidia-smi output showed all devices with correct driver versions, and the GPUs were ready for inference [34].

Establishing the Rebinding Workflow

The assistant then established a workflow for switching between LXC container and VM usage. When switching from VM to LXC, the GPUs needed to be rebinded from vfio-pci back to the NVIDIA driver. This required writing the driver override to the PCI device's sysfs path (/sys/bus/pci/drivers/vfio-pci/unbind followed by /sys/bus/pci/drivers/nvidia/bind) and then reloading the nvidia_uvm module [46][47].

The assistant documented this process meticulously, creating a reusable workflow that would allow relatively painless switching between the two environments. The only real pain point was the ~10-minute model load time when switching back to the LXC container [48].

Part III: The Pivot to Qwen3.5-397B-A17B-NVFP4

Just as the GPU rebinding workflow was established and the Kimi-K2.5 service was loading back into production, the user dropped a concise but consequential message that redirected the entire trajectory of the work: "Swapping the model on the llm server; Get and setup nvidia/Qwen3.5-397B-A17B-NVFP4; use latest upstream / main SGLang (cuda13 nvfp is way faster, also newish model so need latestest build)" [52].

This was not a casual suggestion — it was a strategic decision driven by the recognition that NVFP4 (NVIDIA's FP4 quantization) on CUDA 13 offered dramatically better performance, and that the Qwen3.5 architecture (a 397B-parameter Mixture-of-Experts model with only 17B active parameters per token) was significantly more efficient than the dense Kimi-K2.5 [52].

Building SGLang from Source

The assistant's first action was to stop the Kimi service, check disk space, and fetch the model card to understand the deployment requirements. The model card revealed critical details: the model uses modelopt_fp4 quantization, requires --tensor-parallel-size 4 (not 8, since only 17B parameters are active per token), and needs SGLang with PR #18937 merged [55].

Building the latest SGLang main branch from source proved challenging. The environment used CUDA 13 with custom PyTorch 2.9.1 and hand-tuned dependencies — a fragile setup that a naive pip install could easily break. The assistant discovered that SGLang's pyproject.toml lived in a python/ subdirectory (a non-standard layout common in projects with C++ extensions) and performed a careful --no-deps editable install to preserve the existing dependency versions [62][64][67].

Reapplying SM120 Patches

A critical moment came when the assistant realized that the fresh SGLang main branch clone had lost the SM120 patches applied in the previous build. The assistant ran a diagnostic grep to check whether the new codebase included Blackwell support in two critical files: all_reduce_utils.py (for the TORCH_SYMM_MEM_ALL_REDUCE_MAX_SIZES dictionary) and torch_symm_mem.py (for the _WORLD_SIZES_MULTIMEM dictionary) [80].

The diagnostic confirmed the gap: SM120 entries were missing. Without them, the system would silently fall back to suboptimal communication paths, losing the 77.6% improvement that FlashInfer allreduce fusion and Torch symmetric memory had unlocked for Blackwell. The assistant re-applied the patches using a Python script (since sed proved inadequate for the multi-line insertions needed) [82][83][84].

The NaN Crisis

With SGLang built, SM120 patches applied, and the model downloaded (223 GB across 6 safetensor shards), the assistant started the server. The model loaded in about two minutes — and then the first inference test produced catastrophic output: nothing but repeated exclamation marks, captured as reasoning_content by the Qwen3 reasoning parser. A second test with different parameters confirmed the worst: "finish_reason": "stop" with "matched_stop": "NaN happened" [106].

The model was producing numerical garbage. The assistant's diagnostic process was methodical: it ruled out the reasoning parser as the cause, tested with higher temperature to escape potential degenerate modes, and examined the server logs for FP4-related warnings. The hypothesis formed: the default FP4 GEMM runner backend (flashinfer_cutlass) and MoE runner backend were incompatible with Blackwell's SM120 architecture [106][107].

The assistant inspected the available FP4 GEMM backends in SGLang's source code, finding four choices: auto, flashinfer_cudnn, flashinfer_cutlass, and flashinfer_trtllm. The default was flashinfer_cutlass, and the question was whether it supported SM120 [110].

The Gist That Saved the Deployment

At this critical juncture, the user intervened with a single URL: https://gist.github.com/catid/87cca824963f17fe7479a0ed26221397 [113]. This GitHub Gist, authored by a developer who had already solved this exact problem, documented the precise configuration needed to run Qwen3.5 NVFP4 on Blackwell GPUs.

The assistant fetched the Gist and extracted the critical fix: the NaN outputs were caused by the wrong MoE and FP4 GEMM backends. The solution required explicitly setting --moe-runner-backend flashinfer_cutlass and --fp4-gemm-runner-backend flashinfer_cudnn — an asymmetric configuration where the MoE runner uses CUTLASS while the FP4 GEMM runner uses cuDNN [117].

The assistant applied the fix, stopping the service, updating the systemd configuration, and restarting. The model now produced correct, coherent responses. The deployment was finally successful [117].

Themes and Lessons

The Value of Configuration-Driven Optimization

Throughout this chunk, the most impactful changes were not code modifications but configuration flags. The hierarchical KV cache was enabled with four flags. The FP4 backend fix was two flags. The SM120 patches were data structure entries. This pattern underscores a key insight about modern ML infrastructure: the hard work of implementing features is done by framework developers; the operator's skill lies in knowing which flags to set and when.

The Importance of Structured Clarification

The assistant's response to the GPU passthrough question — refusing to answer the "how" until the "what" was clarified — exemplifies a critical design principle for AI assistants in infrastructure management. By surfacing the unresolved constraint (driver exclusivity between LXC and VM) and forcing a strategic decision before providing tactical advice, the assistant prevented what could have been a destructive misstep [16].

Community Knowledge as a Debugging Accelerator

The GitHub Gist that resolved the NaN crisis is a powerful reminder that in the fast-moving world of ML infrastructure, community knowledge often outpaces official documentation. The user's ability to connect the assistant to this external resource — and the assistant's ability to fetch, parse, and act on it — represents the ideal human-AI collaboration model: the human provides context and connections, the AI executes with precision [113].

The Asymmetric Backend Problem

The NaN debugging revealed a subtle but critical pattern: on Blackwell GPUs, the FP4 GEMM runner and the MoE runner require different backends. The FP4 GEMM needs cuDNN (flashinfer_cudnn), while the MoE runner needs CUTLASS (flashinfer_cutlass). This asymmetric requirement is the kind of detail that is almost impossible to discover through reasoning alone — it requires either empirical testing or community knowledge [117].

Conclusion

This chunk of the opencode session is a microcosm of production ML engineering in the age of frontier models. It spans the full lifecycle: from hardening an existing deployment, through optimizing memory utilization with hierarchical caching, to navigating the complexities of GPU virtualization on Proxmox, to pivoting to a newer model with a completely different quantization format and debugging the resulting numerical instability. Each phase required different skills: system administration for the systemd service, memory architecture analysis for the KV cache, kernel module debugging for the GPU passthrough, and numerical analysis for the FP4 backend issue.

The thread running through all of this is the collaboration between human and AI. The user provided strategic direction, domain knowledge, and community connections; the assistant provided relentless execution, systematic debugging, and the ability to trace problems through source code, system logs, and configuration files. Together, they transformed a machine with eight Blackwell GPUs from a promising hardware platform into a production-ready inference server running one of the most advanced open-weight models available — and they documented every step of the journey.