The Final Act: Deploying a 1T-Parameter Model as a Systemd Service

The Message

scp /home/theuser/glm-kimi-sm120-rtx6000bw/vllm-kimi-k25-int4.service root@10.1.230.174:/etc/systemd/system/vllm-kimi-k25-int4.service && ssh root@10.1.230.174 "systemctl daemon-reload && systemctl enable vllm-kimi-k25-int4 && systemctl start vllm-kimi-k25-int4 && echo 'Service started and enabled'"

Output:

Created symlink /etc/systemd/system/multi-user.target.wants/vllm-kimi-k25-int4.service -> /etc/systemd/system/vllm-kimi-k25-int4.service.
Service started and enabled

At first glance, this message appears to be a routine system administration task — copying a file and running a few systemctl commands. But in the context of the opencode session it belongs to, this single bash invocation represents the culmination of an extraordinary engineering journey spanning multiple days, dozens of model deployments, countless debugging sessions, and a systematic exploration of the performance boundaries of eight NVIDIA RTX PRO 6000 Blackwell GPUs connected over PCIe.

The Journey That Led Here

To understand why this message was written, one must appreciate the path that preceded it. The session had been a whirlwind tour of deploying and benchmarking some of the largest open-weight language models available — models with one trillion parameters. The assistant had started with the GLM-5-NVFP4 model, wrestling with custom GGUF quantization formats, patching vLLM's source code to support novel architectures like glm_moe_dsa, and debugging incoherent output caused by tensor parallelism sharding mismatches in fused projections. It had then pivoted to the NVFP4 variant of Kimi-K2.5, only to discover that the FP8 KV cache was incompatible with the Blackwell SM120 architecture, requiring surgical removal of the kv_cache_quant_algo parameter. Later came MiniMax-M2.5, a 230B-parameter FP8 model using Grouped-Query Attention (GQA), which demonstrated dramatically better throughput — nearly 4,000 tok/s at high concurrency — precisely because its smaller active parameter count and GQA architecture reduced the allreduce burden across the PCIe bus.

But the final pivot was back to Kimi-K2.5, this time in its native INT4 quantization format. The INT4 variant, hosted at moonshotai/Kimi-K2.5 on Hugging Face, weighed in at 547GB across 119 safetensor shards and took 36 minutes to load onto the GPUs. Yet it delivered a single-stream throughput of 82 tok/s — a 34% improvement over the NVFP4 variant's 61 tok/s, and well above the user's target of 40–50 tok/s. Extensive NCCL tuning experiments (varying algorithms between Ring and Tree, adjusting channel counts, thread counts, and buffer sizes) confirmed that the bottleneck was not software-configurable but physical: the PCIe bandwidth available for allreduce operations across eight GPUs for the 61-layer Multi-Head Latent Attention (MLA) architecture.

The Trigger: A Simple Question

The immediate trigger for this message was the user's question in [msg 2387]: "Is the deployment in systemd already?" This was a natural question from someone concerned with production readiness. Up to that point, the Kimi-K2.5 INT4 model was running as a nohup background process inside an SSH session — fragile, ephemeral, and dependent on the terminal session remaining open. A server reboot, an accidental Ctrl+C, or a network interruption would bring down the entire inference service.

The assistant's response in [msg 2388] acknowledged the gap: "No, it's running as a background nohup process right now. Let me fix that." It then checked the existing systemd services on the machine, revealing three prior deployments (vllm-glm5.service, vllm-kimi-k25.service, vllm-minimax-m25.service), all disabled but present — artifacts of earlier iterations in the session. In [msg 2389], the assistant wrote a new service file vllm-kimi-k25-int4.service to the local workspace. And in [msg 2390], it killed the running nohup process and verified that GPU memory was fully freed (all eight GPUs reporting 0 MiB used), creating a clean slate for the systemd-managed launch.

The Message Itself: Anatomy of a Deployment Command

The message in [msg 2391] executes a carefully orchestrated sequence of operations, chained together with && so that any failure aborts the entire pipeline. The command has two logical phases:

Phase 1 — File Transfer: scp copies the service unit file from the assistant's local workspace to the server's /etc/systemd/system/ directory. This is the standard location for systemd unit files on Linux. The choice of scp over alternatives like rsync or direct file write via SSH reflects a pragmatic decision: the file is small, the transfer is one-time, and scp is universally available with no additional dependencies.

Phase 2 — Service Registration and Launch: The SSH command runs four systemctl operations in sequence:

  1. daemon-reload — Instructs systemd to reload its unit file configuration, picking up the newly copied service file. This is mandatory after adding or modifying unit files; without it, systemd would not know about vllm-kimi-k25-int4.service.
  2. enable — Creates the necessary symlinks in /etc/systemd/system/multi-user.target.wants/ to ensure the service starts automatically on boot. The output confirms this: "Created symlink..."
  3. start — Launches the service immediately. Systemd forks the process, monitors its PID, and will restart it if configured to do so.
  4. echo 'Service started and enabled' — A simple confirmation message that serves as a positive signal that all preceding commands succeeded.

The Reasoning Behind the Decisions

Several design decisions are embedded in this seemingly simple command:

Atomicity through chaining: By using && rather than ; or running separate commands, the assistant ensures that the service is never left in a partially configured state. If the scp fails (network issue, disk full), the SSH command never runs. If daemon-reload fails (syntax error in unit file), the service is not enabled or started. This is production-oriented thinking.

No health check verification: Notably, the assistant does not verify that the service actually started successfully. It trusts the systemctl start exit code and the echo confirmation. In a more rigorous deployment, one might add a polling loop (similar to the loading-progress loops used earlier in the session) to wait for the health endpoint to return HTTP 200. The omission suggests either confidence in the service file's correctness (it was modeled on three previous working deployments) or a pragmatic decision that the user can verify manually.

No rollback strategy: The command is a one-way operation. If the service file contains an error that causes the model to fail loading, there is no automated rollback. The assistant would need to debug interactively, as it had done many times before in this session.

Assumption of prior cleanup: The command assumes that the previous nohup process was fully killed and GPU memory released (verified in [msg 2390]). If any residual process were holding GPU memory, the new vLLM instance might fail with an OOM error during model loading. The assistant's earlier verification that all eight GPUs showed 0 MiB used mitigates this risk.

Input Knowledge Required

To understand this message fully, one needs knowledge of:

Output Knowledge Created

This message produces several lasting artifacts:

  1. A persistent service: The Kimi-K2.5 INT4 model is now managed by systemd, meaning it will survive reboots, can be monitored with systemctl status, and can be restarted with systemctl restart. This is the difference between an experimental prototype and a production deployment.
  2. A named unit: The service vllm-kimi-k25-int4.service joins the roster of three prior services on the machine. Its naming convention (vllm-{model}-{variant}.service) is consistent with the established pattern.
  3. A symlink in multi-user.target.wants: This ensures the service starts automatically at boot time, making the deployment self-healing after power cycles or kernel updates.
  4. A clean handoff: The assistant has transferred operational responsibility from its interactive session to the operating system's service manager. The model is now "installed" rather than "running in a terminal."

What This Message Reveals About the Thinking Process

The assistant's reasoning is visible in the structure of the command itself. The use of && chaining reveals a risk-averse mindset — each step depends on the previous one succeeding. The decision to scp the file rather than write it remotely via SSH (e.g., using tee or a heredoc) suggests a preference for having a local copy of the service file as a record of the deployment configuration. The inclusion of daemon-reload shows awareness that systemd caches unit files and requires explicit reloading.

The absence of a verification step (checking systemctl is-active or polling the HTTP health endpoint) is the most telling omission. Throughout the session, the assistant had used elaborate polling loops to track model loading progress, waiting minutes for models to finish loading. Here, it simply trusts the systemd machinery. This could reflect either confidence born from repeated successful deployments or a subtle blind spot — the assumption that because the service file was syntactically correct and the model had loaded successfully before under nohup, it would work identically under systemd. In practice, systemd launches processes with a different environment, different working directory, and different signal handling than a shell session, and subtle issues can arise (e.g., missing PATH entries, different ulimit settings).

Broader Significance

This message is the final step in a journey that began with raw hardware — two freshly installed RTX PRO 6000 Blackwell GPUs on Ubuntu 24.04 — and ended with a production-grade inference service for one of the most capable open-weight language models in existence. Along the way, the assistant had:

Conclusion

Message [msg 2391] is a masterclass in the principle that the final step of any engineering effort should be the simplest. After hours of complex debugging, architectural decisions, and performance optimization, the deployment itself is reduced to a single scp and four systemctl commands. The complexity is not in the message — it is in the knowledge that the message presupposes. The correct NCCL environment variables, the right model path, the appropriate Python environment, the proper service file syntax — all of these were established through the hard work of preceding messages. This message simply reaps the reward.

The assistant's decision to deploy as a systemd service, triggered by the user's question, reflects an understanding that production readiness is not an afterthought but an integral part of the engineering process. A model that runs only in an SSH session is a demo. A model that runs as a systemd service is a product. This message is the boundary between those two states.