From Experiment to Production: Hardening an LLM Server with Systemd

Introduction

In the lifecycle of any machine learning deployment, there comes a critical inflection point: the moment when experimental benchmarks and ad-hoc server processes must give way to hardened, production-grade infrastructure. Message 5665 in this opencode session captures that exact transition. After dozens of rounds spent diagnosing performance regressions, fine-tuning speculative decoding parameters, upgrading CUDA toolkits, and benchmarking configuration variants, the assistant pivots decisively from exploration to deployment. The vehicle for this transition is a systemd service file — a seemingly mundane piece of Linux infrastructure that encodes weeks of optimization work into a single, repeatable, boot-persistent unit.

This article examines that message in depth: why it was written, what decisions it encodes, the assumptions it rests on, and the knowledge it both consumes and produces. The message is remarkable not for its technical complexity — writing a systemd unit is straightforward — but for what it represents: the crystallization of an extensive experimental journey into operational practice.

The Message

The assistant writes:

No existing systemd service. Let me create one. First kill the nohup process, then create the service:

[bash] cat << 'SVCEOF' | ssh root@10.1.230.174 'cat > /etc/systemd/system/sglang-kimi.service'
[Unit]
Description=SGLang Kimi-K2.5 INT4 with EAGLE-3 (topk=1, spec_v2 overlap)
After=network.target
# Wait for NVIDIA driver to be ready
After=nvidia-persistenced.service

[Service]
Type=simple
User=root
WorkingDirectory=/root/sglang

# CUDA and NCCL environment (also set in sitecustomize.py, but explicit here for clarity)
Environment=CUDA_HOME=/usr/local/cuda-13.0
Environment=PATH=/root/ml-env/bin:/usr/local/cuda-13.0/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
Environment=LD_LIBRARY_PATH=/usr/local/cuda-13.0/lib64

# Enable spec_v2 overlap scheduling
Environment=SGLANG_ENABLE_SPEC_V2=True

ExecStart=/root/ml-env/bin/python3 -m sglang.launch_server \
  --model-path /shared/kimi-k2.5-int4 \
  --tp 8 \
  --trust-remote-code \
  --cuda-graph-max-bs 128 \
  --disable-custom-all-reduce \
  --attention-backend flashinfer \
  --enable-flashinfer-allreduce-fusion \
  --speculative-algorithm EAGLE3 \
  --speculative-draft-model-path /data/eagle3/output_100k_sglang/4 \
  --speculative-num-steps 2 \
  --speculative-eagle-topk 1 \
  --mem-fraction-static 0.88

# Restart policy
Restart=on-failure
RestartSec=30

# Give it time to load model (547 GB)
TimeoutStartSec=900
TimeoutStopSec=60

# Kill all child processes on stop
KillMode=control-group
KillSignal=SIGTERM

# Logging
StandardOutput=journal
StandardError=journal
SyslogIdentifier=sglang-kimi

# Resource limits
LimitNOFILE=65536
LimitMEMLOCK=infinity

[Install]
WantedBy=multi-user.target
SVCEOF
echo "Service file written"
Service file written

At first glance, this is a straightforward systemd unit. But every line in this file is the product of a long and arduous optimization journey. To understand why each element appears, we must reconstruct the context that produced it.## Why This Message Was Written: The Motivation and Context

The immediate trigger for message 5665 was the user's directive in [msg 5659]: "Save findings, on the machine - save /root/production_v2.md with details + update prod deployment (systemd and all) to run this exact setup, start on boot etc." This instruction came after the assistant had just presented compelling benchmark results showing that the topk=1 + spec_v2 configuration matched or beat baseline throughput at high concurrency levels — a result that had been elusive through dozens of prior optimization attempts.

The deeper motivation, however, stretches back much further. The session's history reveals a long struggle to make EAGLE-3 speculative decoding viable on an 8× RTX PRO 6000 Blackwell system connected via PCIe Gen5 with no NVLink. Earlier attempts had produced net-negative speculation: the overhead of the verify step, dominated by NCCL all-reduce communication across PCIe, meant that running with speculation was actually slower than running without it. The assistant had explored FlashInfer allreduce fusion, Torch symmetric memory, NCCL tuning parameters, and even a CUDA toolkit upgrade from version 12.8 to 13.0 — all in pursuit of making the verify step fast enough that speculation could break even.

The breakthrough came with the spec_v2 overlap path, which runs scheduler batch preparation in parallel with GPU forward passes, effectively hiding the draft model's overhead under load. Combined with topk=1 (chain speculation instead of tree speculation), this configuration achieved 759.3 tok/s at concurrency 30 versus 689.4 tok/s for baseline — a 10% improvement where previously speculation had been a net loss.

Message 5665 is the moment this hard-won knowledge is frozen into a deployable artifact. The systemd service is not just a convenience — it is a declaration that the experimental phase is over and the production phase has begun.

Decisions Embedded in the Service File

Every parameter in the ExecStart line represents a deliberate choice, many of them the product of A/B testing and profiling:

--tp 8: Tensor parallelism across all 8 GPUs. This was never in question — the Kimi-K2.5 INT4 model at 547 GB cannot fit on a single GPU, and the hardware provides exactly 8 Blackwell RTX PRO 6000 cards. But the decision to use TP rather than pipeline parallelism or expert parallelism reflects the architecture of the model and the constraints of PCIe communication.

--cuda-graph-max-bs 128: This was the result of a specific optimization discovered earlier in the session (<msg id=5653 context>). The assistant had found that reducing the CUDA graph batch size from 512 to 128 improved baseline throughput by roughly 9%. This counterintuitive result — smaller batches yielding higher throughput — likely stems from the PCIe-bound nature of the system, where larger graphs increase communication pressure without proportional compute benefit.

--disable-custom-all-reduce: The custom all-reduce kernel does not yet support Blackwell's SM120 architecture. This flag forces the use of NCCL's all-reduce instead, which, while slower in absolute terms, is the only functional option on this hardware. The NCCL tuning parameters (set in sitecustomize.py, not in the service file) compensate by selecting optimal protocols and algorithms for the PCIe topology.

--attention-backend flashinfer: Benchmarks showed FlashInfer's attention kernel was approximately 10% faster than the Triton-based default on this hardware. This is a model-specific optimization — different architectures might favor different backends.

--enable-flashinfer-allreduce-fusion: This flag fuses the all-reduce communication with preceding compute operations, reducing the number of kernel launches and improving GPU utilization. It was one of the key optimizations that made speculative decoding viable (<msg id=5653 context>).

--speculative-algorithm EAGLE3 and --speculative-draft-model-path ...: These select the EAGLE-3 speculative decoding algorithm and point to the fine-tuned drafter model. The drafter was trained from scratch on 37K coding and agentic samples over multiple epochs, with the final epoch (epoch 4) selected for production.

--speculative-num-steps 2 and --speculative-eagle-topk 1: These parameters define the speculation strategy. With topk=1 and num_steps=2, the drafter produces a chain of 3 draft tokens (steps+1). This is a deliberately conservative choice compared to the topk=4 tree (which produces 16 draft tokens). The trade-off is lower acceptance rate per step but dramatically lower overhead — and crucially, topk=1 is required for the spec_v2 overlap path.

--mem-fraction-static 0.88: The fraction of GPU memory reserved for KV cache. The assistant notes that baseline auto-detects approximately 0.78, so this is an explicit increase to accommodate the additional memory pressure from speculative decoding's draft model and verification buffers.

Assumptions and Their Implications

The service file encodes several assumptions that deserve scrutiny:

Assumption 1: The hardware configuration is stable. The service assumes 8 GPUs, all at /dev/nvidia[0-7], all functioning correctly. It assumes PCIe Gen5 connectivity with the specific NCCL tuning parameters that were discovered through trial and error. If a GPU fails or the PCIe topology changes (e.g., moving to a different server), the NCCL tuning may become suboptimal or even counterproductive.

Assumption 2: The model paths are persistent. The service hard-codes /shared/kimi-k2.5-int4 for the base model and /data/eagle3/output_100k_sglang/4 for the drafter. If storage is remounted or the drafter is retrained, the service file must be updated. The assistant partially mitigates this by documenting the paths in production_v2.md ([msg 5662]), but the service itself has no fallback or validation logic.

Assumption 3: The spec_v2 overlap path is unconditionally beneficial. The benchmark data shows that topk=1 + spec_v2 beats baseline at high concurrency but is approximately 6% slower at single-stream (C=1). The service does not implement dynamic speculation disable — a feature the assistant had attempted to add but abandoned due to state coupling issues in the EAGLE worker (<msg id=5653 context>). This means single-user interactive sessions will be slightly slower than they could be without speculation.

Assumption 4: The environment variables in sitecustomize.py will be loaded. The service explicitly sets CUDA_HOME, PATH, LD_LIBRARY_PATH, and SGLANG_ENABLE_SPEC_V2, but it does not duplicate the NCCL tuning variables (NCCL_PROTO=LL, NCCL_ALGO=Ring, etc.). These are set in /usr/lib/python3.12/sitecustomize.py, which is loaded automatically by Python. The assistant's comment "also set in sitecustomize.py, but explicit here for clarity" shows awareness of this dependency. If someone runs the service with a different Python interpreter or a modified sitecustomize.py, the NCCL tuning would be lost and performance would degrade significantly — the assistant had previously observed baseline throughput dropping from ~89 to ~63 tok/s without these settings.

Assumption 5: Systemd is the appropriate service manager. The container runs Ubuntu 24.04 with systemd 255, so this is safe. But the service depends on nvidia-persistenced.service being available and functional. If the NVIDIA driver stack is updated or reconfigured, the service ordering could break.

Input Knowledge Required

To understand message 5665 fully, one needs knowledge spanning multiple domains:

System administration: Understanding of systemd unit files, service dependencies (After=), restart policies, environment variables, resource limits (LimitNOFILE, LimitMEMLOCK), and the boot process (WantedBy=multi-user.target).

GPU computing: Knowledge of CUDA toolkit paths, NVIDIA driver architecture (nvidia-persistenced.service), GPU memory management, tensor parallelism, and the distinction between custom all-reduce and NCCL.

LLM inference: Understanding of speculative decoding, EAGLE-3 architecture, draft models, acceptance rates, KV cache management, and the relationship between batch size, concurrency, and throughput.

Networking and topology: Awareness of how PCIe Gen5 communication differs from NVLink, why NCCL tuning matters for PCIe-bound systems, and how all-reduce fusion can mitigate communication overhead.

The specific optimization history: Knowledge that --cuda-graph-max-bs 128 was chosen after benchmarking showed it outperformed 512, that --mem-fraction-static 0.88 was increased from the auto-detected 0.78, and that --speculative-eagle-topk 1 is required by spec_v2 — these are not documented in any upstream SGLang documentation but were discovered empirically during this session.

Output Knowledge Created

Message 5665 produces several forms of knowledge:

Explicit knowledge: The service file itself, stored at /etc/systemd/system/sglang-kimi.service on the target machine. This is a repeatable, boot-persistent configuration that can be version-controlled, audited, and replicated.

Operational knowledge: The assistant demonstrates a workflow for transitioning from ad-hoc server processes (run via nohup in a terminal) to managed services. This includes killing the old process, verifying GPU availability with fuser, reloading systemd, enabling the service for auto-start, and starting it. These steps are documented in the following messages ([msg 5666] through [msg 5670]).

Institutional knowledge: The service file, combined with production_v2.md created in [msg 5662], forms a complete deployment record. Anyone with access to these files can understand not just what is running but why each parameter was chosen — the document includes benchmark comparisons, NCCL tuning rationale, and a summary of applied patches.

Negative knowledge: The service file implicitly documents what was not chosen. There is no --speculative-eagle-topk 4, no --enable-torch-symmetric-memory (which was tested but not ultimately needed for the overlap path), no dynamic speculation disable (attempted but abandoned). A reader familiar with the optimization history can infer the rejected alternatives from the absence of their corresponding flags.

The Thinking Process Visible in the Message

The assistant's reasoning is visible in several aspects of the message:

The ordering of operations: The message begins with "No existing systemd service. Let me create one. First kill the nohup process, then create the service." This reveals a mental model of the deployment as a state machine: the old process must be terminated before the new service can take over, and the service file must exist before it can be started. The assistant then executes these steps in the following messages, checking GPU availability between kill and start.

The choice of what to include in the service file vs. what to leave in sitecustomize.py: The assistant explicitly duplicates CUDA_HOME, PATH, and LD_LIBRARY_PATH in the service file even though they are also set in sitecustomize.py. The NCCL tuning variables, however, are left in sitecustomize.py. This reveals a judgment about what is "infrastructure" (CUDA paths, which might change independently of the service) versus what is "application configuration" (NCCL tuning, which is tightly coupled to the model and hardware). The comment "also set in sitecustomize.py, but explicit here for clarity" shows the assistant weighing the trade-off between redundancy and robustness.

The resource limit choices: LimitNOFILE=65536 and LimitMEMLOCK=infinity are not arbitrary. The high file descriptor limit accommodates the many socket connections and file handles that an LLM inference server opens under load. The unlimited memlock prevents the GPU driver from failing when it needs to pin memory for CUDA operations. These choices reflect an understanding of how SGLang interacts with the Linux kernel's resource management.

The timeout values: TimeoutStartSec=900 (15 minutes) for a 547 GB model that must be loaded into GPU memory across 8 devices. TimeoutStopSec=60 for clean shutdown. These values show the assistant reasoning about the physical constraints of the hardware — loading half a terabyte of model weights takes time, and killing the process prematurely could leave GPUs in an inconsistent state.

Mistakes and Potential Pitfalls

While the service file is well-constructed, several potential issues deserve mention:

No health check or readiness notification: The service uses Type=simple, meaning systemd considers it "started" as soon as the ExecStart process forks. The model loading takes 15 minutes (as reflected in TimeoutStartSec=900), but systemd has no way to know when the server is actually ready to accept requests. A production improvement would be to use Type=notify with sd_notify() calls in the server code, or to add an ExecStartPost script that polls the health endpoint.

No logging to disk: The service sends all output to the journal (StandardOutput=journal, StandardError=journal). While journald is reliable, logs are lost if the journal is corrupted or if the system crashes before flushing. For a production service, adding StandardOutput=append:/var/log/sglang-kimi.log would provide a secondary log destination.

Hard-coded paths: The model path /shared/kimi-k2.5-int4 and drafter path /data/eagle3/output_100k_sglang/4 are hard-coded. If the drafter is retrained (e.g., epoch 5 performs better), the service file must be manually updated. An environment variable or configuration file would be more maintainable.

No resource isolation: The service runs as root with no CPU or memory limits. Under heavy load, the SGLang server could consume all available system resources, potentially starving other services. For a dedicated inference node this may be acceptable, but in a multi-tenant environment, cgroup-based limits would be advisable.

The NCCL tuning dependency: As noted above, the NCCL tuning variables are not in the service file. If sitecustomize.py is ever modified or bypassed, performance will silently degrade. A more robust approach would be to set these variables directly in the service's Environment directives, even at the cost of some redundancy.

Conclusion

Message 5665 is a deceptively simple artifact that encapsulates weeks of intensive optimization work. A systemd service file, at its core, is just a list of directives and commands. But this particular file is the product of a journey that spanned CUDA toolkit upgrades, NCCL tuning, FlashInfer fusion experiments, speculative decoding topology optimization, and the hard-won discovery that overlap scheduling could transform a net-negative speculation scheme into a net-positive one.

The message represents the moment when experimental knowledge crystallizes into operational practice. Every flag in the ExecStart line, every environment variable, every timeout value, every resource limit tells a story of a benchmark run, a profiling session, or a debugging marathon. The service file is not just infrastructure — it is the final report of an optimization campaign, encoded in the language of system configuration.

For anyone deploying large language models in production, this message offers a template not just for a systemd unit, but for the mindset required to transition from experimentation to deployment. The key insight is that production hardening is not a separate phase that begins after optimization is complete — it is the act of preserving optimization results in a form that survives reboots, survives terminal closures, and survives the engineer walking away from the keyboard. The systemd service file is the vessel that carries hard-won knowledge from the ephemeral realm of SSH sessions into the permanent realm of boot-persistent infrastructure.