The Final Deployment: Systemd Service Registration for a 1T-Parameter Model

The Message

scp /home/theuser/glm-kimi-sm120-rtx6000bw/vllm-kimi-k25.service root@10.1.230.174:/etc/systemd/system/vllm-kimi-k25.service && ssh root@10.1.230.174 'systemctl daemon-reload && systemctl enable vllm-kimi-k25 && systemctl start vllm-kimi-k25 && echo "Service started"'

The output confirms the first step succeeded:

Created symlink /etc/systemd/system/multi-user.target.wants/vllm-kimi-k25.service -> /etc/systemd/system/vllm-kimi-k25.service.

Then the bash tool timed out after 30 seconds, leaving the service start potentially still in progress on the remote machine.

This single message, message 2150 in a long coding session spanning dozens of rounds and multiple model deployments, represents the culminating moment of an extraordinary engineering effort: transitioning a manually-launched vLLM inference server for a 1-trillion-parameter mixture-of-experts model into a properly managed, production-grade systemd service. To understand why this seemingly mundane command matters, one must appreciate the journey that led here.

The Road to Deployment

The context preceding this message is a saga of hardware-software co-evolution at the frontier of large language model deployment. The assistant had been working on a machine equipped with 8 NVIDIA RTX PRO 6000 Blackwell GPUs (compute capability SM120) — powerful workstation-class hardware that nonetheless lacks certain data-center features. The original goal was to deploy the GLM-5 model using a custom GGUF quantization, but after extensive patching of vLLM's gguf_loader.py, building custom Triton kernels, and resolving tensor parallelism sharding mismatches, the assistant pivoted to a different model entirely: nvidia/Kimi-K2.5-NVFP4, a 1-trillion-parameter MoE model based on the DeepSeek V3 architecture, quantized by NVIDIA using their NVFP4 format.

This pivot was not trivial. The old GLM-5 GGUF weights (402 GB) had to be removed. The new model, spanning 540 GB across 119 safetensor shards, had to be downloaded. A critical blocker emerged immediately: the NVFP4 checkpoint shipped with FP8 KV cache configuration baked into its hf_quant_config.json and config.json files, but no MLA attention backend on SM120 supports FP8 KV cache. The TRITON_MLA backend, the only viable option for Blackwell workstation GPUs, hardcoded a NotImplementedError for FP8. The assistant resolved this by surgically removing kv_cache_quant_algo and kv_cache_scheme from the configuration files, forcing a fallback to fp16 KV cache — a pragmatic decision that traded theoretical memory efficiency for functional correctness.

After resolving the KV cache issue, the model loaded successfully (70.8 GiB per GPU, taking 523 seconds), and the assistant benchmarked it at approximately 60 tok/s for single-request decode throughput — a respectable figure for a 1T MoE model running on workstation GPUs. Coherence testing across factual, multi-step, multi-turn, and creative prompts confirmed the model produced correct, coherent output with proper reasoning traces.

Why This Message Was Written

The motivation for message 2150 is rooted in operational maturity. Up to this point, the vLLM server had been launched manually via nohup from an SSH session — a fragile arrangement vulnerable to terminal disconnection, accidental process termination, and requiring manual restart after any system reboot. The assistant recognized that a model of this scale, serving as what appears to be a persistent API endpoint, needed proper lifecycle management.

The decision to create a systemd service reflects several engineering considerations:

Persistence and Reliability. A systemd service survives SSH session termination, can be configured to restart automatically on failure, and starts on boot. For a model that takes 523 seconds to load, every unplanned restart is costly — making automatic recovery essential.

Operational Hygiene. The manual process had already been killed in message 2148 (pkill -9 -f "python3.*vllm"), and shared memory segments cleaned up (rm -f /dev/shm/psm_* /dev/shm/sem.mp-*). The systemd service provides a clean interface for future management: systemctl start, systemctl stop, systemctl status, and journalctl -u vllm-kimi-k25 for logs.

Configuration Encapsulation. The service file (written in message 2149) encapsulates all the tuning parameters discovered through hours of debugging: the NCCL_PROTO=LL and NCCL_P2P_LEVEL=SYS environment variables that mitigate the PCIe allreduce bottleneck, the --max-model-len 131072 that fits within the 96 GB per-GPU memory budget, the --tool-call-parser kimi_k2 and --reasoning-parser kimi_k2 flags that enable the model's reasoning capabilities, and the --gpu-memory-utilization 0.95 setting.

The Technical Decisions in the Command

The command itself is a carefully constructed pipeline with distinct phases. First, scp copies the service file from the local development environment to the remote server's systemd directory. The path /etc/systemd/system/ is significant — this is the directory for system-provided services (as opposed to /lib/systemd/system/ for vendor-supplied packages or /etc/systemd/user/ for user services). Placing the file here signals that this is a system-level service.

The second phase, executed via SSH, chains four systemctl commands with && operators, ensuring each step succeeds before the next begins. daemon-reload instructs systemd to rescan the unit files, picking up the newly added service. enable creates the symlink in multi-user.target.wants, ensuring the service starts automatically at boot. start launches the service immediately. The final echo "Service started" provides a simple confirmation.

The output confirms that the symlink was created successfully — the service is now enabled for automatic startup. However, the bash tool timed out after 30 seconds, meaning the systemctl start command may still have been executing when the tool was terminated. This is expected: vLLM model loading takes several minutes (523 seconds in the previous run), and systemctl start blocks until the service reports readiness or fails. The service was likely still in its loading phase when the timeout occurred.

Assumptions and Their Validity

The message makes several assumptions, most of which are well-grounded. It assumes the service file was correctly written in message 2149 — a reasonable assumption given that the write tool reported success. It assumes the remote server runs systemd, which is true for Ubuntu 24.04. It assumes the previous manual vLLM process was successfully killed and GPU memory freed, which was verified in message 2148. It assumes the service file's ExecStart path (/root/ml-env/bin/python3) and working directory are correct — assumptions validated by the earlier successful manual launch.

One subtle assumption is that the service will eventually start successfully. The timeout prevents immediate verification, but the assistant likely planned to check the service status in a subsequent message (which indeed happens in the following messages of the session).

Input Knowledge Required

To fully understand this message, one needs knowledge of Linux system administration (systemd unit files, the systemctl command family, the unit file directory hierarchy), remote file transfer with scp, and the architecture of vLLM as an inference server. One must also understand the broader context: that this is a 1T-parameter MoE model using NVFP4 quantization, that it runs on 8 Blackwell GPUs with SM120 compute capability, that the PCIe interconnect is a bottleneck mitigated by NCCL protocol tuning, and that the model's KV cache configuration had to be surgically modified to work around hardware limitations.

Output Knowledge Created

This message creates a persistent, managed service for the Kimi-K2.5-NVFP4 model. The immediate output is a symlink in multi-user.target.wants, ensuring the service starts automatically on every boot. The broader output is a production-grade deployment that can be monitored, restarted, and maintained using standard systemd tools. The service encapsulates hours of debugging and optimization into a repeatable, documented configuration.

Significance

This message represents the transition from experimentation to production. The earlier messages in this segment were about discovery — resolving incompatibilities, patching code, benchmarking performance. This message is about commitment — locking in the configuration, making the deployment persistent, and accepting the current state as good enough for service. The 60 tok/s throughput, constrained by the PCIe allreduce bottleneck, is now the operating baseline rather than a benchmark to optimize further. The systemd service file becomes the authoritative record of how this model should be served, encoding all the hard-won knowledge about NCCL protocols, memory utilization, and model configuration into a form that can be reviewed, version-controlled, and reproduced.

In the broader narrative of the coding session, this message is the capstone of the Kimi-K2.5 deployment — the moment when a complex, multi-day engineering effort crystallizes into a single file and a single command.