From Production Hardening to Model Migration: The Full Lifecycle of Multi-GPU LLM Deployment on Blackwell

Introduction

This segment of the opencode session captures one of the most densely packed stretches of production ML engineering imaginable. Spanning over 120 messages, it represents the culmination of weeks of optimization work — CUDA toolkit upgrades, NCCL tuning, speculative decoding configuration, crash debugging, and systemd service creation — all converging into a reliable, boot-persistent production system. But the story does not end there. Having achieved a hardened deployment of the Kimi-K2.5 INT4 model, the session pivots to deploying a completely different model architecture — the Qwen3.5-397B-A17B-NVFP4 — demonstrating the flexibility and depth of the infrastructure that was built.

The narrative arc encompasses four major phases: the finalization of EAGLE-3 speculative decoding with the topk=1 spec_v2 overlap configuration, the production hardening process that codified the setup into a systemd service, a tangent into GPU passthrough on Proxmox that revealed critical details about Blackwell's open kernel module requirement, and finally a complete model migration to Qwen3.5 NVFP4 that required building the latest SGLang from source and debugging NaN outputs through careful backend configuration. Each phase reveals something important about the practice of deploying large language models in production on cutting-edge hardware.

Phase I: The Final Breakthrough in Speculative Decoding

The segment opens at a critical juncture. The assistant had just produced a comprehensive 3,000-word status document ([msg 5615]) that captured the full state of the optimization campaign. This document catalogued every discovery from the preceding weeks: the CUDA 13 upgrade that unblocked FlashInfer allreduce fusion on Blackwell GPUs, the transformation of EAGLE-3 from a net-negative 54.1 tok/s to a net-positive 96.1 tok/s, the exhaustive parallel throughput benchmarks, the failed dynamic speculation disable attempt on the v1 worker path, and the pivot to the spec_v2 overlap scheduling approach.

The document was written while the assistant waited for a new server to load — a server configured with SGLANG_ENABLE_SPEC_V2=True, --speculative-eagle-topk 1, and --speculative-num-steps 2. This configuration represented a bet: that the overlap scheduling in spec_v2, which runs scheduler batch preparation in parallel with GPU forward passes, could hide the draft model overhead that had plagued earlier attempts. The trade-off was that spec_v2 required topk=1 (chain speculation, producing only 3 draft tokens) instead of the topk=4 tree speculation (16 draft tokens) used in earlier benchmarks. Lower acceptance rate, but lower overhead and cleaner architecture.

But before the assistant could validate this hypothesis, the user delivered a laconic reality check ([msg 5616]): "continue; Seems server crashed btw." This seven-word message shattered the assistant's assumption that the server was still loading and forced an immediate pivot from planning to debugging.

The Phantom AttributeError

The crash traceback pointed to the overlap scheduling event loop — event_loop_overlap in the scheduler — confirming that the spec_v2 path had failed during its first batch processing attempt. What followed was one of the most detailed debugging sequences in the entire session. The crash was caused by an AttributeError — a missing spec_disable_batch_threshold attribute on the EAGLE worker object.

The fix was not as simple as adding the attribute, because the object did have the attribute in its __init__ method. The problem was that the object was only partially initialized when the crash occurred. The root cause was a subtle initialization ordering issue: the EAGLE worker's __init__ method set self.spec_disable_batch_threshold partway through initialization, but a CUDA graph capture step that occurred earlier in __init__ could fail, leaving the object in an inconsistent state where the attribute was never set. The fix — initializing the attribute to 0 at the very top of __init__ — was a defensive programming pattern that ensured the attribute existed regardless of where initialization failed.

This debugging sequence is a masterclass in systematic diagnosis. The assistant traced the error through multiple hypotheses: perhaps the attribute was never set (contradicted by reading the source), perhaps it was being deleted (contradicted by code analysis), perhaps the wrong class was being instantiated (contradicted by module path checks), perhaps the object was partially initialized (confirmed by traceback analysis). Each hypothesis was tested, each contradiction was documented, and the final fix was minimal but correct.

Phase II: Production Hardening — From nohup to Systemd

With the crash fixed and the configuration validated — benchmarks showed topk=1 + spec_v2 matching or beating baseline throughput at high concurrency — the user gave the decisive instruction ([msg 5659]): "Save findings, on the machine - save /root/production_v2.md with details + update prod deployment (systemd and all) to run this exact setup, start on boot etc."

This instruction marked the transition from experimentation to production. The assistant's response was methodical and thorough. First, it created /root/production_v2.md ([msg 5662]), a comprehensive production deployment document that captured not just the launch command but the entire rationale: benchmark comparisons across four configurations, explanations for each flag, NCCL tuning parameters, and a catalog of all patches applied to SGLang. This document is not merely a configuration reference; it is the crystallization of an intensive optimization campaign spanning multiple days, multiple CUDA toolkit versions, and countless server restarts.

The Systemd Service

Then came the systemd service creation. The assistant verified that systemd was available on the Ubuntu 24.04 container, checked for existing service files, and confirmed the running process before creating /etc/systemd/system/sglang-kimi.service. The service file encoded every optimization decision from the preceding weeks:

The Operational Fixes: Tool Call Parsers and Model Names

Having achieved a running systemd service, the session then addressed two critical operational gaps that the user discovered through actual usage.

The first was the tool call parsing problem ([msg 5690]). The user reported that the server was generating raw special tokens — <|tool_calls_section_begin|>, <|tool_call_begin|>, <|tool_call_argument_begin|> — in the API response content field, rather than parsing them into the structured tool_calls array that the OpenAI-compatible API specification demands. This was the difference between a model that can generate tool calls and a serving stack that understands them.

The fix required adding --tool-call-parser kimi_k2 and --reasoning-parser kimi_k2 flags to the launch command. These flags tell SGLang's post-processing layer how to interpret the Kimi-K2.5 model's specific special token vocabulary. Without them, the server was technically operational — returning HTTP 200, generating tokens — but functionally broken for its intended use as an agentic coding assistant.

The second fix was the model name exposure. The user asked: "Can you make sglang expose this model as 'kimi-k2.5'?" By default, SGLang exposes the model under its Hugging Face path (/shared/kimi-k2.5-int4), which is not a user-friendly identifier. Adding --served-model-name kimi-k2.5 to the launch command made the OpenAI-compatible API endpoint return the correct model name, enabling downstream tools and libraries to identify the model properly.

Both fixes required editing the systemd service file, reloading systemd, and restarting the server — each restart incurring the ~9.5 minute cold start penalty. The assistant handled this gracefully, batching the changes to minimize restarts.

The Capacity Challenge: Hierarchical KV Cache Offload

With the server running and correctly parsing tool calls, the user identified the next bottleneck ([msg 5720]): "seems like I'm kinda low on max parallel request context, what options do we have for ram offload of kv cache?"

This question revealed a sophisticated understanding of the memory hierarchy in GPU inference. The Kimi-K2.5 INT4 model consumed approximately 72.3 GB per GPU (across 8 GPUs), leaving only ~21.7 GB per GPU for KV cache, activations, and the EAGLE-3 draft model. With --mem-fraction-static 0.88, the KV cache pool was allocated at roughly 7.9 GB per GPU — enough for perhaps 159K tokens of cache, but severely limiting concurrent request capacity.

The assistant's diagnostic response was a model of disciplined engineering: three parallel commands to establish ground truth. nvidia-smi showed ~5 GB free per GPU. free -h revealed 396 GB of available system RAM — a massive untapped resource. journalctl confirmed the KV cache pool allocation. These three commands are not random; they form a deliberate diagnostic triad that captures GPU memory, system memory, and current allocation state simultaneously.

The assistant then embarked on a deep investigation of SGLang's KV cache allocation logic, tracing the exact formula used to compute max_total_tokens and the mem_fraction_static parameter. This investigation involved reading SGLang source code, deriving the allocation formula, and understanding why 7.43 GB remained free per GPU despite the 0.88 fraction.

The solution was SGLang's hierarchical cache (HiCache) feature, which uses system RAM as an L2 prefix cache. The user's decisive three-word message — "What about hicache?" ([msg 5737]) — selected this option from the assistant's menu of four alternatives. This was a decision signal, a steering command, and an implicit evaluation of the assistant's prior analysis — all compressed into three syllables.

The assistant configured HiCache with --enable-hierarchical-cache --hicache-ratio 4.0, effectively adding ~358 GB of system RAM as an L2 KV cache tier. This transformed the system's effective memory capacity from ~63 GB (GPU-only) to over 358 GB, directly addressing the user's concern about parallel request context. The verification showed Allocating 44.77 GB host memory for hierarchical KV cache on each TP thread — confirmation that the massive allocation was succeeding.## Phase III: The GPU Passthrough Tangent

Just as the Kimi-K2.5 deployment reached a stable, hardened state, the user pivoted to a tangent experiment: passing all eight GPUs to a Proxmox VM instead of using the LXC container. This seemingly simple request would expose layers of complexity about NVIDIA driver architecture, Proxmox PCI passthrough mechanics, and the Blackwell GPU's unique requirements.

The Clarifying Question

The user's initial question was straightforward: "Because it's now a VM and not a container do we need to unload nvidia driver on host? Blacklist and reboot? How can we make this sort of switch reasonably painless?" The assistant's response was a masterclass in structured clarification. Rather than answering the surface-level "how" question, it reframed the problem at a strategic level: "are you passing through all 8 GPUs to the VM, or just some of them? And do you need the LXC container to keep running simultaneously?"

This was not pedantry — it was essential risk management. The GPUs could only be bound to one driver at a time (either the host's nvidia driver for LXC access, or vfio-pci for VM passthrough). Without clarifying the allocation model, any advice the assistant gave could break the production setup. The user answered "All 8 GPUs to VM," and the assistant proceeded with a clear understanding of the constraints.

The Discovery That Proxmox Already Solved It

The assistant's investigation revealed a much more favorable reality than expected. The Proxmox host already had a working PCI mapping configuration (/etc/pve/mapping/pci.cfg with a mapping called pro6000), and VM 131 (ml-pipelines) was already configured with eight hostpci entries. The key insight was that Proxmox handles the driver switch automatically — starting a VM with hostpci entries unbinds the NVIDIA driver and binds vfio-pci, and stopping the VM reverses the process. No blacklisting, no rebooting, no manual module unloading.

The assistant confirmed this by checking lspci -ks 01:00.0 on the host, which showed "Kernel driver in use: vfio-pci" — the GPUs were already detached from the NVIDIA driver. The NVIDIA kernel modules were still loaded in memory (as shown by lsmod), but they had zero references to any devices.

The Blackwell Open Kernel Module Requirement

But when the assistant checked the VM guest, a different problem emerged. The GPUs were visible in lspci and /dev/nvidia* devices existed, but nvidia-smi returned "No devices were found." The kernel messages revealed the culprit: NVRM: installed in this system requires use of the NVIDIA open kernel modules.

This was a critical discovery about Blackwell GPU architecture. The proprietary NVIDIA kernel module (nvidia-dkms-590) cannot initialize Blackwell GPUs — they require the open kernel module (nvidia-dkms-590-open). This is a relatively recent change in NVIDIA's driver strategy: starting with the SM120 architecture, the open kernel module is mandatory. The fix was a single package installation: apt-get install nvidia-dkms-590-open, which replaces the proprietary module via DKMS rebuild.

After a VM reboot, all eight GPUs came to life. The nvidia-smi output showed all devices with correct driver versions, and the GPUs were ready for inference.

Establishing the Rebinding Workflow

The assistant then established a workflow for switching between LXC container and VM usage. When switching from VM to LXC, the GPUs needed to be rebinded from vfio-pci back to the NVIDIA driver. This required writing the driver override to the PCI device's sysfs path (/sys/bus/pci/drivers/vfio-pci/unbind followed by /sys/bus/pci/drivers/nvidia/bind) and then reloading the nvidia_uvm module.

The assistant documented this process meticulously, creating a reusable workflow that would allow relatively painless switching between the two environments. The only real pain point was the ~10-minute model load time when switching back to the LXC container.

While this tangent did not ultimately change the deployment approach, it demonstrated the flexibility of the infrastructure and provided a fallback option if the container approach proved insufficient. It also revealed the critical open kernel module requirement for Blackwell GPUs — knowledge that would prove essential when deploying the next model.

Phase IV: The Pivot to Qwen3.5-397B-A17B-NVFP4

The final major arc of this segment was a pivot to a newer, more efficient model: nvidia/Qwen3.5-397B-A17B-NVFP4. This model uses NVFP4 quantization (NVIDIA's native FP4 format) and a Mixture-of-Experts architecture with 397B total parameters and only 17B active parameters per token — significantly more efficient than the 1T-parameter Kimi-K2.5.

Just as the GPU rebinding workflow was established and the Kimi-K2.5 service was loading back into production, the user dropped a concise but consequential message that redirected the entire trajectory of the work: "Swapping the model on the llm server; Get and setup nvidia/Qwen3.5-397B-A17B-NVFP4; use latest upstream / main SGLang (cuda13 nvfp is way faster, also newish model so need latestest build)."

This was not a casual suggestion — it was a strategic decision driven by the recognition that NVFP4 on CUDA 13 offered dramatically better performance, and that the Qwen3.5 architecture was significantly more efficient than the dense Kimi-K2.5.

Building SGLang from Source

The assistant's first action was to stop the Kimi service, check disk space, and fetch the model card to understand the deployment requirements. The model card revealed critical details: the model uses modelopt_fp4 quantization, requires --tensor-parallel-size 4 (not 8, since only 17B parameters are active per token), and needs SGLang with PR #18937 merged.

Building the latest SGLang main branch from source proved challenging. The environment used CUDA 13 with custom PyTorch 2.9.1 and hand-tuned dependencies — a fragile setup that a naive pip install could easily break. The assistant discovered that SGLang's pyproject.toml lived in a python/ subdirectory (a non-standard layout common in projects with C++ extensions) and performed a careful --no-deps editable install to preserve the existing dependency versions.

Reapplying SM120 Patches

A critical moment came when the assistant realized that the fresh SGLang main branch clone had lost the SM120 patches applied in the previous build. The assistant ran a diagnostic grep to check whether the new codebase included Blackwell support in two critical files: all_reduce_utils.py (for the TORCH_SYMM_MEM_ALL_REDUCE_MAX_SIZES dictionary) and torch_symm_mem.py (for the _WORLD_SIZES_MULTIMEM dictionary).

The diagnostic confirmed the gap: SM120 entries were missing. Without them, the system would silently fall back to suboptimal communication paths, losing the 77.6% improvement that FlashInfer allreduce fusion and Torch symmetric memory had unlocked for Blackwell. The assistant re-applied the patches using a Python script (since sed proved inadequate for the multi-line insertions needed).

The NaN Crisis

With SGLang built, SM120 patches applied, and the model downloaded (223 GB across 6 safetensor shards), the assistant started the server. The model loaded in about two minutes — and then the first inference test produced catastrophic output: nothing but repeated exclamation marks, captured as reasoning_content by the Qwen3 reasoning parser. A second test with different parameters confirmed the worst: "finish_reason": "stop" with "matched_stop": "NaN happened".

The model was producing numerical garbage. The assistant's diagnostic process was methodical: it ruled out the reasoning parser as the cause, tested with higher temperature to escape potential degenerate modes, and examined the server logs for FP4-related warnings. The hypothesis formed: the default FP4 GEMM runner backend (flashinfer_cutlass) and MoE runner backend were incompatible with Blackwell's SM120 architecture.

The assistant inspected the available FP4 GEMM backends in SGLang's source code, finding four choices: auto, flashinfer_cudnn, flashinfer_cutlass, and flashinfer_trtllm. The default was flashinfer_cutlass, and the question was whether it supported SM120.

The Gist That Saved the Deployment

At this critical juncture, the user intervened with a single URL: https://gist.github.com/catid/87cca824963f17fe7479a0ed26221397. This GitHub Gist, authored by a developer who had already solved this exact problem, documented the precise configuration needed to run Qwen3.5 NVFP4 on Blackwell GPUs.

The assistant fetched the Gist and extracted the critical fix: the NaN outputs were caused by the wrong MoE and FP4 GEMM backends. The solution required explicitly setting --moe-runner-backend flashinfer_cutlass and --fp4-gemm-runner-backend flashinfer_cudnn — an asymmetric configuration where the MoE runner uses CUTLASS while the FP4 GEMM runner uses cuDNN.

The assistant applied the fix, stopping the service, updating the systemd configuration, and restarting. The model now produced correct, coherent responses. The deployment was finally successful.

Additionally, the Gist revealed that hybrid GDN models (which combine linear attention layers with full attention layers) require --attention-backend triton rather than flashinfer on Blackwell. The assistant had initially used flashinfer and encountered an AssertionError at startup — a clear error message that pointed directly to the fix. After switching to triton, the server started successfully.

Themes and Lessons

Several themes emerge from this segment that are relevant to anyone deploying large language models in production.

The importance of systematic debugging. The phantom AttributeError debugging sequence demonstrates that the most elusive bugs are often not where they first appear. The assistant's methodical approach of forming hypotheses, gathering evidence, and revising assumptions is a template for debugging complex distributed systems.

The gap between "server is running" and "server is usable." The tool call parsing problem revealed that a server can pass health checks and generate tokens while being fundamentally broken for its intended use case. Production verification must test the full API contract, not just basic connectivity.

The value of documentation. The /root/production_v2.md document created in [msg 5662] is not a luxury — it is the single source of truth for the deployment. It captures not just what is running but why each parameter was chosen, enabling future engineers to understand, modify, or replicate the configuration.

The discipline of diagnosis before prescription. When the user asked about KV cache offload, the assistant did not immediately propose solutions. It ran three diagnostic commands to establish ground truth, then researched the options, then presented a menu of alternatives. This pattern — diagnose, research, present options, let the user decide — is a hallmark of mature engineering.

The inevitability of pivots. The session began with EAGLE-3 speculative decoding optimization, transitioned to production hardening, and ended with deploying a completely different model. The infrastructure built for the first model — the systemd service, the NCCL tuning, the operational procedures — was reusable for the second. Building for flexibility pays off.

Community knowledge as a debugging accelerator. The GitHub Gist that resolved the NaN crisis is a powerful reminder that in the fast-moving world of ML infrastructure, community knowledge often outpaces official documentation. The user's ability to connect the assistant to this external resource — and the assistant's ability to fetch, parse, and act on it — represents the ideal human-AI collaboration model: the human provides context and connections, the AI executes with precision.

The asymmetric backend problem. The NaN debugging revealed a subtle but critical pattern: on Blackwell GPUs, the FP4 GEMM runner and the MoE runner require different backends. The FP4 GEMM needs cuDNN (flashinfer_cudnn), while the MoE runner needs CUTLASS (flashinfer_cutlass). This asymmetric requirement is the kind of detail that is almost impossible to discover through reasoning alone — it requires either empirical testing or community knowledge.

Conclusion

This segment of the opencode session captures the full lifecycle of a production ML deployment: from experimental benchmarking through crash debugging, production hardening, operational fixes, capacity planning, GPU virtualization, and finally model migration. The 120+ messages in this segment represent hundreds of hours of engineering work compressed into a narrative that reveals both the technical depth and the human judgment required to deploy large language models on cutting-edge hardware.

The final state of the system — a systemd-managed SGLang server running Qwen3.5-397B-A17B-NVFP4 with hierarchical KV cache, proper tool call parsing, and NCCL tuning for PCIe-bound Blackwell GPUs — is the product of systematic optimization, disciplined debugging, and the kind of operational knowledge that can only be earned through experience. The journey from "the server crashed" to "the server is healthy, parsing tool calls, and has 358 GB of KV cache capacity" is the story of this segment, and it is a story worth studying for anyone who works at the intersection of machine learning and systems engineering.

The thread running through all of this is the collaboration between human and AI. The user provided strategic direction, domain knowledge, and community connections; the assistant provided relentless execution, systematic debugging, and the ability to trace problems through source code, system logs, and configuration files. Together, they transformed a machine with eight Blackwell GPUs from a promising hardware platform into a production-ready inference server running one of the most advanced open-weight models available — and they documented every step of the journey.## References

[1] "From Optimization to Production: The Hardening and Evolution of a Multi-GPU LLM Deployment" — Chunk 0 article covering the Kimi-K2.5 production hardening, EAGLE-3 debugging, systemd service creation, tool call parsers, hierarchical KV cache, and the pivot to Qwen3.5.

[2] "From Kimi-K2.5 to Qwen3.5: A Production Odyssey Through Model Deployment, GPU Passthrough, and Blackwell Compatibility" — Chunk 1 article covering the GPU passthrough tangent, open kernel module requirement, SGLang main branch build, SM120 patches, and NaN debugging.

[3] The comprehensive status document at [msg 5615] — captures the full state of the optimization campaign including CUDA 13 upgrade, EAGLE-3 benchmarks, NCCL tuning, and the spec_v2 exploration plan.

[4] The user's crash notification at [msg 5616] — "continue; Seems server crashed btw" — the seven-word message that forced an immediate pivot from planning to debugging.

[5] The production deployment instruction at [msg 5659] — "Save findings... update prod deployment (systemd and all) to run this exact setup, start on boot etc."

[6] The production_v2.md creation at [msg 5662] — comprehensive deployment document capturing benchmark comparisons, configuration rationale, NCCL tuning, and patch catalog.

[7] The tool call parsing bug report at [msg 5690] — user reports unparsed special tokens in API output, leading to the --tool-call-parser kimi_k2 and --reasoning-parser kimi_k2 fix.

[8] The KV cache capacity question at [msg 5720] — user identifies the parallel request context bottleneck, triggering the hierarchical cache investigation.

[9] The decisive "What about hicache?" at [msg 5737] — three words that selected the hierarchical cache approach from the assistant's menu of four alternatives.

[10] The GitHub Gist at [msg 5850] — catid's SM120 instructions that resolved the NaN crisis with the asymmetric FP4 GEMM and MoE backend configuration.