From Experiment to Production: The Systemd Transition in a High-Stakes ML Deployment

The Message

The subject message (index 5664) is a single tool-call round from an AI assistant working on deploying a large language model inference server. It reads:

Now let me kill the current nohup server and set up the systemd service. First, let me check if there's already a systemd service:

>

[bash] ssh root@10.1.230.174 'ls /etc/systemd/system/sglang 2>/dev/null; ls /etc/systemd/system/llm 2>/dev/null; echo "---"; ps aux | grep "sglang\|launch_server" | grep -v grep'

>

--- root 184364 0.0 0.2 53196232 960492 ? Sl 11:45 0:30 /root/ml-env/bin/python3 -m sglang.launch_server --model-path /shared/kimi-k2.5-int4 --tp 8 --trust-remote-code --cuda-graph-max-bs 128 --disable-custom-all-reduce --attention-backend flashinfer --enable-flashinfer-allreduce-fusion --speculative-algorithm EAGLE3 --speculative-draft-model-path /data/eagle3/output_100k_sglang/4 --speculative-num-steps 2 --speculative-eagle-topk 1 --mem-fraction-static 0.88 root 184500 0.0...

On the surface, this appears to be a mundane administrative task: checking for existing systemd service files and confirming the running process before transitioning to a managed service. But in the context of the broader session — a multi-week effort to deploy a 547-billion-parameter language model across eight NVIDIA Blackwell GPUs connected via PCIe — this message represents a critical inflection point. It is the moment where months of experimental optimization, benchmarking, debugging, and hardware wrangling crystallize into a hardened production deployment. This article examines why this message was written, what decisions it embodies, the knowledge it requires and produces, and the thinking process visible in its execution.

Context: The Long Road to Production

To understand why this message matters, one must appreciate the journey that preceded it. The assistant and user had been working for an extended period — across multiple segments and dozens of messages — to deploy the Kimi-K2.5 INT4 model, a massive 547 GB language model, on a server with eight NVIDIA RTX PRO 6000 Blackwell GPUs connected via PCIe Gen5 without NVLink. This hardware configuration is challenging because PCIe communication between GPUs is significantly slower than NVLink, creating a bottleneck for tensor-parallel inference where every forward pass requires all-reduce operations across all eight GPUs.

The central technical challenge had been speculative decoding using EAGLE-3, a technique where a smaller "draft" model generates candidate tokens that the large "target" model verifies in parallel, potentially increasing throughput. However, on this PCIe-bound system, early attempts at EAGLE-3 speculation were net-negative — the overhead of the verify step (which required expensive all-reduce communication) outweighed the benefit of generating multiple tokens per forward pass. Through a series of optimizations — upgrading to CUDA 13, enabling FlashInfer allreduce fusion, enabling torch symmetric memory, and critically, discovering the spec_v2 "overlap scheduling" path — the assistant had transformed EAGLE-3 from a 54.1 tok/s liability into a configuration that matched or beat baseline throughput at high concurrency.

The final benchmark results, documented in the immediately preceding messages, were decisive. The topk=1 + spec_v2 configuration achieved 86.8 tok/s at single-stream (versus 92.7 baseline, a 6% gap) but pulled ahead at high concurrency: 759.3 tok/s at C=30 versus 689.4 baseline, 775.0 at C=100 versus 773.1, and 754.4 at C=250 versus 718.1. This was the configuration worth deploying.

Why This Message Was Written: The Motivation

The message was written in direct response to the user's instruction in [msg 5659]: "No, ok, leave like this; Save findings, on the machine - save /root/production_v2.md with details + update prod deployment (systemd and all) to run this exact setup, start on boot etc." The user had considered a suggestion to test topk=2 and decided against it, instead directing the assistant to freeze the configuration and make it production-ready.

This instruction reflects a critical decision-making pattern in ML engineering: the decision to stop optimizing and start deploying. The user recognized that the topk=1 + spec_v2 configuration, while not perfect (it still had a 6% single-stream deficit), was "good enough" for production. The marginal benefit of further tuning (e.g., testing topk=2) was outweighed by the operational cost of delaying deployment. This is the classic "satisficing" decision in engineering — choosing a satisfactory solution over an endlessly optimized one.

The assistant's response in [msg 5660] acknowledged this directive by updating its todo list with three high-priority items: create /root/production_v2.md with full findings, create a systemd service for the topk=1 spec_v2 configuration, and enable the service to start on boot. In [msg 5662], the assistant completed the first task, writing a comprehensive production documentation file. The subject message ([msg 5664]) represents the beginning of the second task: transitioning from a manually-managed nohup process to a systemd-managed service.

The Decision to Use Systemd

The choice of systemd as the service manager was not arbitrary. In [msg 5661], the assistant had verified that systemd was available on the target machine by running which systemctl and checking /proc/1/comm. The output confirmed systemd 255 was running. This verification was essential because the machine was a Proxmox LXC container, and not all containers run systemd as their init system. Had the container used a different init system (e.g., OpenRC or a minimal init), the assistant would have needed to adapt its approach.

The decision to use systemd reflects several assumptions about the deployment environment. First, that the machine would be rebooted periodically (kernel updates, power cycles, maintenance), and the inference server should restart automatically. Second, that the server process needed proper lifecycle management — clean startup, graceful shutdown, restart on failure, and logging integration. Third, that the deployment should be reproducible and declarative, with the service configuration captured in a file rather than relying on manual commands.

What the Message Reveals About the Running Configuration

The ps aux output in the message is remarkably informative. It shows the exact command-line arguments used to launch the current server:

/root/ml-env/bin/python3 -m sglang.launch_server \
  --model-path /shared/kimi-k2.5-int4 \
  --tp 8 \
  --trust-remote-code \
  --cuda-graph-max-bs 128 \
  --disable-custom-all-reduce \
  --attention-backend flashinfer \
  --enable-flashinfer-allreduce-fusion \
  --speculative-algorithm EAGLE3 \
  --speculative-draft-model-path /data/eagle3/output_100k_sglang/4 \
  --speculative-num-steps 2 \
  --speculative-eagle-topk 1 \
  --mem-fraction-static 0.88

Each flag represents a decision hardened through benchmarking. --cuda-graph-max-bs 128 was chosen after discovering that reducing batch size from 512 improved single-stream throughput by 9%. --attention-backend flashinfer was selected because it was 10% faster than the triton backend. --enable-flashinfer-allreduce-fusion was critical for PCIe-bound performance, fusing the all-reduce communication with compute operations. --disable-custom-all-reduce was necessary because the custom all-reduce kernel didn't support the SM120 architecture of Blackwell GPUs. The --speculative-eagle-topk 1 and --speculative-num-steps 2 flags encode the chain speculation configuration (3 draft tokens) required by the spec_v2 overlap scheduling path.

The process listing also reveals that the server was launched at 11:45 (the Sl in the process state indicates it's sleeping but in a multi-threaded state), had been running for about 30 minutes, and was using approximately 53 GB of virtual memory (53196232 KB) — a fraction of the 547 GB model size because of memory sharing and GPU memory mapping.

Input Knowledge Required

To fully understand this message, a reader needs knowledge spanning several domains. First, an understanding of the SGLang inference framework and its architecture: how launch_server works, what tensor parallelism (--tp 8) means, and how speculative decoding integrates with the serving stack. Second, familiarity with NVIDIA GPU hardware: the Blackwell architecture (SM120), the distinction between PCIe and NVLink interconnects, and the implications for all-reduce performance. Third, knowledge of systemd service management: unit file syntax, service dependencies, environment variables, and restart policies.

More subtly, the reader needs to understand the experimental methodology that led to this configuration. The benchmark comparison table in the production_v2.md document (created in the previous message) shows four configurations tested across multiple concurrency levels. The reader must grasp why topk=1 + spec_v2 was chosen over topk=4 + v1 despite the latter having a higher theoretical acceptance rate. The answer lies in the overlap scheduling: spec_v2 runs the scheduler batch preparation in parallel with the GPU forward pass, effectively hiding the draft model overhead. This insight — that architectural scheduling improvements could outweigh algorithmic improvements — was the key breakthrough.

Output Knowledge Created

This message creates several forms of output knowledge. Most immediately, it confirms that no existing systemd service files exist for either sglang* or llm* patterns, establishing a clean baseline for service creation. It also confirms that the server is currently running as a nohup process with PID 184364, which will need to be killed before the systemd service can take over port 30000.

The message also implicitly documents the current running configuration. By capturing the full command line in the process listing, it serves as a record of exactly which flags were used. This is valuable for reproducibility — if the systemd service file (created in the next message, [msg 5665]) has a typo or missing flag, the operator can compare against this known-good invocation.

Perhaps most importantly, this message marks the boundary between experimental and production modes. Before this point, the server could be killed and restarted with different flags at any time. After this point, the configuration is frozen, and changes require updating the systemd unit file and restarting the service. This transition is a form of organizational knowledge: the team now has a "production configuration" that should only be changed through a deliberate process.

Assumptions and Potential Mistakes

The message makes several assumptions that warrant examination. The most significant is that the current nohup process should be killed and replaced by a systemd service. This assumes that the systemd service will start successfully with the same flags — an assumption that proved correct in the subsequent message, but was not guaranteed. Systemd services run in a different environment than a user's shell session: environment variables, working directory, PATH, and resource limits may differ. The assistant anticipated this by including explicit environment variables in the service file (CUDA_HOME, PATH, LD_LIBRARY_PATH, SGLANG_ENABLE_SPEC_V2), but subtle differences could still cause issues.

Another assumption is that the NCCL tuning environment variables set in /usr/lib/python3.12/sitecustomize.py would be picked up by the systemd service. Since systemd services don't source shell profiles, the NCCL settings (NCCL_PROTO=LL, NCCL_ALGO=Ring, etc.) would only be applied if the Python interpreter loads sitecustomize.py — which it does by default. This assumption held, but it's worth noting that the service file doesn't duplicate these NCCL variables, creating a hidden dependency on the sitecustomize.py file.

A potential mistake visible in this message is the lack of a --host or --port flag in the command line. The server defaults to 127.0.0.1:30000, which is fine for local access but would need explicit configuration for external access. The production_v2.md document mentions the service listens on 127.0.0.1:30000, so this appears intentional — the server is meant to be accessed through a reverse proxy or from within the same machine.

The Thinking Process

The thinking process visible in this message is one of methodical execution. The assistant doesn't rush to create the service file immediately; instead, it first checks for existing services and confirms the running process. This reflects a "look before you leap" philosophy that is characteristic of reliable systems engineering. The two checks — ls for service files and ps aux for running processes — are complementary: one checks for static configuration, the other for dynamic state.

The structure of the ps aux command is also revealing. The grep pattern "sglang\|launch_server" uses alternation to match either the process name or the module name, ensuring that all relevant processes are captured. The grep -v grep filter removes the grep process itself from the output, a common but essential refinement.

The assistant's choice to run both checks in a single SSH command (separated by ; and echo "---") rather than two separate commands shows an awareness of latency and efficiency. Each SSH connection has overhead, and combining commands reduces round-trips. The echo "---" delimiter makes the output parseable by human eyes, clearly separating the two result sets.

Conclusion

Message 5664 is a deceptively simple tool call that encapsulates the transition from experimentation to production in a high-stakes ML deployment. It represents the moment when the team decided that the configuration was good enough, froze it, and began the process of hardening it for reliable operation. The message requires knowledge of GPU architecture, inference serving, and Linux service management to fully understand. It creates output knowledge about the current system state and the exact running configuration. And it makes assumptions — mostly justified — about environment compatibility between shell and systemd contexts.

In the broader narrative of the session, this message is the hinge point. Everything before it was exploration, optimization, and benchmarking. Everything after it is operations, monitoring, and maintenance. The systemd service created in the following message would serve as the foundation for serving the Kimi-K2.5 model to users, running continuously until the next major configuration change.