The Final Deployment: Formalizing a 1T-Parameter MoE Model into Production

Introduction

In the world of large-scale ML inference, the difference between an experimental prototype and a production service often comes down to a single file: a systemd unit. Message [msg 2392] captures exactly that transition — the moment when a meticulously benchmarked, carefully tuned 1-trillion-parameter INT4 MoE model (Kimi-K2.5) was formally deployed as a persistent, boot-enabled systemd service on an 8× Blackwell GPU server. The message itself is deceptively brief: a confirmation that the service vllm-kimi-k25-int4.service is deployed, enabled on boot, and currently loading (a process that takes approximately 30 minutes for this 547GB model). But behind this short status check lies a journey of hardware-aware model selection, exhaustive benchmarking, and systematic elimination of software artifacts that spanned multiple days of intensive engineering work.

The Context: Why This Message Was Written

The immediate trigger for message [msg 2392] was a simple question from the user at [msg 2387]: "Is the deployment in systemd already?" This question exposed a gap between what the assistant had built and how it was running. At that moment, the Kimi-K2.5 INT4 model was executing as a background nohup process — an ad-hoc arrangement suitable for experimentation but entirely inappropriate for a production inference server that needed to survive reboots, restart after crashes, and integrate with system monitoring.

The assistant's response at [msg 2388] acknowledged this directly: "No, it's running as a background nohup process right now. Let me fix that." This recognition triggered a rapid sequence of actions: listing existing vLLM systemd services (revealing three prior deployments: vllm-glm5.service, vllm-kimi-k25.service, and vllm-minimax-m25.service), writing a new service file, killing the nohup process, copying the service file to the remote server via SCP, and finally enabling and starting the service. Message [msg 2392] is the confirmation that all of these steps succeeded.

The Reasoning Behind the Model Choice

To understand why this particular model — the INT4 quantized Kimi-K2.5 — was chosen for the final production deployment, one must look at the extensive benchmarking that preceded this moment. The assistant had evaluated three major model variants across multiple dimensions:

Kimi-K2.5 NVFP4 (540GB): The first variant tested. It achieved 61 tok/s single-stream but suffered from severe PCIe allreduce bottlenecks due to its 61-layer MLA (Multi-Head Latent Attention) architecture. Every decode step required synchronizing attention computations across all 8 GPUs over PCIe, and NVFP4's weight format didn't help enough to overcome this.

MiniMax-M2.5 FP8 (215GB): A smaller 230B-parameter GQA model that used only 4 GPUs (TP=4) and achieved 84 tok/s single-stream with a blistering 75-second load time. At high concurrency with TP=8 and Expert Parallelism, it reached nearly 4,000 tok/s. However, it was a different model family with different quality characteristics.

Kimi-K2.5 INT4 (547GB): The native INT4 quantization of the same Kimi-K2.5 architecture. This variant delivered 82 tok/s single-stream — a 34% improvement over the NVFP4 version (61 tok/s) and well above the user's target of 40-50 tok/s. At high concurrency (C=256), it reached 2,276 tok/s. The INT4 format reduced weight bandwidth requirements for the quantized MoE experts, directly translating to higher throughput despite the identical MLA architecture and PCIe constraints.

The assistant's reasoning for choosing INT4 over NVFP4 is crystallized in the comparative table at [msg 2386]: "Kimi-K2.5 INT4 is significantly faster than the NVFP4 variant at single-stream (82 vs 61 tok/s, +34%) and competitive with MiniMax at low concurrency." The INT4 variant also represented the frontier-quality model the user originally wanted, whereas MiniMax was a pragmatic fallback.

The Decisions Embedded in the Service Deployment

Message [msg 2392] doesn't show the service file itself, but the systemctl status output reveals several important design decisions:

The ExecStartPre guard: The status output shows an ExecStartPre script that loops up to 30 times checking whether GPU memory has been freed (checking that the maximum used memory across any GPU is below 1000 MiB). This is a critical safeguard for a multi-GPU system where zombie processes from previous runs can leave GPU memory allocated. Without this guard, a restart could fail because the vLLM process would attempt to allocate memory that's still held by a defunct process. The assistant had encountered this exact problem multiple times during the session, as evidenced by the repeated kill -9 and fuser commands needed to clear GPU memory before relaunching.

The NCCL and CUDA environment variables: While not visible in this message, the service file (written at [msg 2389]) inherits the tuning parameters from the final benchmark run: NCCL_PROTO=LL, NCCL_ALGO=Ring, NCCL_P2P_LEVEL=SYS, NCCL_MAX_NCHANNELS=16, NCCL_BUFFSIZE=16777216, NCCL_NTHREADS=512, and CUDA_DEVICE_MAX_CONNECTIONS=1. These were the result of extensive NCCL tuning experiments that ultimately confirmed the bottleneck was physical PCIe bandwidth, not algorithm choice — but the tuned parameters still provided marginal improvements.

The vLLM arguments: The service uses the V1 engine with --tensor-parallel-size 8, --tool-call-parser kimi_k2, --reasoning-parser kimi_k2, --max-model-len 131072, --gpu-memory-utilization 0.95, and --enable-auto-tool-choice. Notably, --num-scheduler-steps was attempted but failed because the V1 engine doesn't support it (only V0 does), and --compilation-config '{"level": 3}' was tried but didn't improve performance because the CUDAGraph cache from prior runs was already being used.

Assumptions Made

The deployment in message [msg 2392] rests on several assumptions:

That the 30-minute load time is acceptable. The assistant explicitly notes "now loading (~30 min for this model)" — this is a 547GB model being loaded across 8 GPUs over what is presumably a network filesystem (the model path /shared/kimi-k2.5-int4 suggests a shared storage location). The assumption is that the systemd service's startup timeout accommodates this.

That the ExecStartPre guard is sufficient. The 30-iteration loop with 1-second checks assumes that GPU memory will be freed within 30 seconds. If a previous process takes longer to die (e.g., if it's stuck in an unrecoverable CUDA state), the service will fail to start.

That no further tuning is needed. The assistant had already established that NCCL algorithm tuning, compilation level adjustments, and scheduler step configurations produced no meaningful improvement. The assumption is that 82 tok/s single-stream represents the practical ceiling for this hardware configuration, and further optimization would require architectural changes (e.g., switching to a GQA model or upgrading to NVLink-connected GPUs).

That the model loads correctly every time. The INT4 compressed-tensors format requires specific dequantization kernels that were verified to work on the SM120 architecture (Blackwell). The assistant had previously fixed bugs in weight loading, GGUF dequantization shard ordering, and Triton MLA attention backends. The assumption is that these fixes are stable and the model loads identically on each restart.

Input Knowledge Required

To fully understand message [msg 2392], one needs knowledge of:

Systemd service management: The concepts of unit files, systemctl enable (creating symlinks for automatic startup), systemctl start, and the ExecStartPre directive for pre-condition checks.

vLLM inference server architecture: The V1 engine, tensor parallelism (TP=8), the health endpoint, and the loading process that reports percentage progress.

GPU memory management on multi-GPU systems: The challenge of zombie processes holding GPU memory, the use of nvidia-smi to check memory usage, and the fuser command to identify processes using NVIDIA devices.

The model landscape: Kimi-K2.5 is a 1-trillion-parameter Mixture-of-Experts model from Moonshot AI, using Multi-Head Latent Attention (MLA). The INT4 variant uses compressed-tensors quantization. The NVFP4 variant uses NVIDIA's FP4 format. Both are frontier-quality models.

PCIe allreduce bottlenecks: The fundamental limitation of synchronizing 61 MLA attention layers across 8 GPUs over PCIe, where each decode step requires multiple allreduce operations that saturate the PCIe bus.

Output Knowledge Created

Message [msg 2392] produces several concrete outputs:

A production-ready systemd service: The vllm-kimi-k25-int4.service file at /etc/systemd/system/ on the target machine, enabled in multi-user.target.wants/ for automatic startup on boot.

A validated deployment pipeline: The sequence of killing old processes, clearing GPU memory, copying the service file, reloading systemd, enabling, and starting the service — all executed successfully in [msg 2391].

A benchmark-validated performance baseline: 82 tok/s single-stream, scaling to 2,276 tok/s at C=256, with the understanding that this is the hardware-limited ceiling for this model on 8× PCIe-connected Blackwell GPUs.

A comparative reference point: The INT4 variant outperforms the NVFP4 variant by 34% on single-stream throughput, establishing that for this hardware configuration, INT4 quantization is the superior choice for the Kimi-K2.5 architecture.

The Thinking Process Visible in the Message

The message itself is terse — it's a status confirmation rather than an analytical exposition. But the thinking process is visible in what's not said. The assistant doesn't ask "should I deploy this?" or "is this the right model?" — those questions were already resolved through the extensive benchmarking in the preceding messages. The assistant doesn't explore alternative deployment strategies (Docker, Kubernetes, supervisor) — systemd was chosen because the prior three deployments (GLM-5, NVFP4 Kimi, MiniMax) all used systemd, establishing it as the team's convention.

The parenthetical "(~30 min for this model)" reveals an awareness of the user's time expectations. The assistant knows that a 30-minute load time is significant and preemptively communicates it so the user isn't left wondering whether the service is stuck. The use of head -10 on the systemctl output shows deliberate filtering — the full status output would include the ExecStartPre loop iterations and potentially confuse the reader.

Conclusion

Message [msg 2392] is the quiet culmination of a massive engineering effort. It represents the moment when a 1-trillion-parameter model, after surviving driver installations, CUDA toolkit conflicts, flash-attn build failures, GGUF dequantization bugs, Triton attention backend defects, NCCL tuning dead ends, and FP8 KV cache incompatibilities, finally becomes a production service. The systemd unit file is the formal contract between the model and the infrastructure — a promise that this 547GB neural network will be available when the server boots, will restart if it crashes, and will deliver 82 tokens per second to every user who asks. In the world of large-scale ML inference, that's the definition of done.