Segment 38

This sub-session saw the Kimi-K2.5 INT4 model move from experimental benchmarking to a hardened production deployment. The optimal EAGLE-3 topk=1 spec_v2 configuration was finalized and confirmed to match baseline throughput at high concurrency. A crash from a missing attribute in the dynamic speculation disable patch was fixed, and the setup was codified into a systemd service with auto-start on boot. Critical operational improvements included adding tool call and reasoning parsers to the API output and enabling the hierarchical KV cache to utilize system RAM as a prefix cache. A tangent experiment involved passing the GPUs to a VM, which required installing the open kernel module for Blackwell support and establishing a rebinding workflow. The user then pivoted to deploying the newer, more efficient Qwen3.5-397B-A17B-NVFP4 model, which required building the latest SGLang main branch from source, applying SM120 patches, and resolving NaN outputs by explicitly configuring the FP4 GEMM and MoE backends for Blackwell.

From Production Hardening to Model Migration: The Full Lifecycle of Multi-GPU LLM Deployment on Blackwell 4009 words

Chunks

• From Optimization to Production: The Hardening and Evolution of a Multi-GPU LLM Deployment
  2819 words
• From Kimi-K2.5 to Qwen3.5: A Production Odyssey Through Model Deployment, GPU Passthrough, and Blackwell Compatibility
  2444 words