The Systemd Unit: Deploying Observability Infrastructure for an LLM Serving Cluster

A Single Tool Call That Bridges Code and Production

In the midst of a sprawling engineering session spanning GPU kernel optimization, production incident response, and monitoring infrastructure buildout, one message stands out for its deceptive simplicity. Message [msg 13088] consists of just a few lines of reasoning and a single tool call:

[assistant] ## Agent Reasoning I'm setting up the GPU exporter deployment with a systemd unit file, configuring Prometheus to scrape metrics from the exporter on port 9101, and then reloading Prometheus to verify the metrics are being collected. Now the systemd unit + Prometheus job. Writing the unit: [write] /tmp/opencode/gpu-exporter.service Wrote file successfully.

On its surface, this appears trivial — the assistant writes a systemd unit file to disk. But this message is the critical hinge point where a Python script becomes a production service. It represents the transition from building to deploying, from a proof-of-concept metric exporter to a managed, persistent component of the observability stack. Understanding why this moment matters requires tracing the chain of events that led here.

The Crisis That Demanded Visibility

The story begins with a production incident. The cluster — an 8-GPU machine running a prefill-decode disaggregated deployment of DeepSeek-V4-Flash — became unresponsive under load. Clients received KVTransferError aborts. The root cause, diagnosed in [msg 13085] and surrounding messages, was queue saturation: the single prefill server's unbounded request queue had accumulated approximately 20 requests and 220,000 pending tokens, causing time-to-first-token to balloon to minutes, clients to time out, and in-flight KV transfers to fail.

The assistant implemented two immediate fixes: admission control via --max-queued-requests 32 to prevent unbounded pileup, and HiCache (hierarchical caching) to improve prefix reuse and reduce VRAM pressure. But these were tactical fixes. The deeper problem was a visibility gap — the team had no way to see the saturation building until the cluster collapsed. The Prometheus instance was scraping only the three SGLang endpoints (prefill, decode, router), with no GPU metrics, no node-level health indicators, and no way to distinguish between "the service is running" and "the service is healthy."

The user's request was clear: add node health indicators and HiCache metrics to the Grafana dashboard. But the assistant quickly realized that "node health" required data that simply did not exist in the monitoring pipeline. There was no GPU exporter, no node exporter, no way to see GPU utilization, memory pressure, or temperature. The up{} metric from Prometheus could confirm a service was reachable, but it could not distinguish between a healthy GPU computing tokens and a GPU thrashing at 100% utilization with thermal throttling.

The Decision to Build, Not Install

The assistant faced a fork in the road. The standard solution would be to install dcgm-exporter (NVIDIA's official GPU metrics exporter) or nvidia-smi-based polling. But in [msg 13086], the assistant discovered that pynvml was already available in the Python virtual environment — pynvml OK, GPUs: 8 — and that the existing environment had all the dependencies needed. This changed the cost-benefit calculation.

Building a custom exporter had several advantages. First, it avoided dependency hell — no need to download, verify, and install a binary like dcgm-exporter, which might have compatibility issues with the custom CUDA 12.8 / 13.1 environment. Second, it could be minimal and focused, exposing exactly the metrics needed for the Grafana panels without the overhead of a full DCGM suite. Third, it could be written in the same Python environment as the rest of the stack, making it maintainable by the same team. The trade-off was development time: the assistant had to write the exporter from scratch rather than installing a pre-built solution.

The decision was made in [msg 13087], where the assistant wrote gpu_exporter.py — a lightweight Python script using pynvml to expose GPU utilization, memory usage, temperature, and power metrics on port 9101. The reasoning shows a clear awareness of the trade-offs: "I can use pynvml to build a GPU exporter since it works with the 8 GPUs available." The assistant recognized that the existing infrastructure (Python environment with pynvml, systemd for service management, Prometheus for scraping) made the custom approach the path of least resistance.

Why This Message Matters: The Deployment Bridge

Message [msg 13088] is where the exporter stops being a script and becomes a service. The gpu_exporter.py file written in the previous round is just code on disk. Without a systemd unit, it has no lifecycle management — no automatic restart on failure, no startup on boot, no integration with the system's service supervision. The systemd unit file is the mechanism that transforms a Python script into a production-grade component.

The assistant's reasoning in this message reveals the full deployment plan compressed into a single sentence: "I'm setting up the GPU exporter deployment with a systemd unit file, configuring Prometheus to scrape metrics from the exporter on port 9101, and then reloading Prometheus to verify the metrics are being collected." This is a three-step plan: (1) write the unit, (2) configure Prometheus, (3) reload and verify. The message executes step 1. Steps 2 and 3 will follow in the next round ([msg 13089]), where the assistant copies both files to the remote machine, enables the service, appends the Prometheus job, and attempts a reload.

The tool call itself — [write] /tmp/opencode/gpu-exporter.service — is notable for what it does not show. The assistant does not display the contents of the systemd unit file in the message. The file is written to /tmp/opencode/ (a temporary working directory), to be copied to the remote server via scp in the next round. This is a deliberate workflow choice: the assistant writes files locally first, then deploys them. The separation between writing and deploying allows the assistant to reason about the file contents in one round and execute the deployment in the next, respecting the synchronous round-based architecture of the session.

Input Knowledge Required

To understand this message, the reader needs several pieces of context. First, the production environment: Ubuntu 24.04 with systemd as the init system, where services are managed through unit files in /etc/systemd/system/. Second, the monitoring architecture: Prometheus scrapes metrics from HTTP endpoints, and the Grafana dashboard is generated by a Python script (gen_dashboard.py) that produces JSON. Third, the GPU exporter's interface: the Python script exposes metrics on port 9101 in Prometheus text format, which Prometheus can scrape. Fourth, the session's operational pattern: the assistant writes files to /tmp/opencode/ and then copies them to the remote machine via scp in subsequent rounds.

The reader also needs to understand the urgency behind this work. The cluster had just experienced a production outage caused by invisible queue saturation. The GPU exporter was not an abstract improvement — it was a direct response to a failure mode where lack of visibility led to service collapse. Every metric the exporter exposes (GPU utilization, memory pressure, temperature) is a signal that could predict the next saturation event.

Output Knowledge Created

This message produces a systemd unit file at /tmp/opencode/gpu-exporter.service. While the contents are not displayed, we can infer its structure from the deployment commands in [msg 13089]. The unit file defines a service that runs the Python exporter script (/root/gpu_exporter.py), listens on port 9101, and is managed by systemd with automatic startup and restart behavior. The unit file is the artifact that enables the exporter to survive reboots, process crashes, and system failures.

The output also includes implicit knowledge: the assistant's deployment strategy. By writing the unit file locally and then copying it to the remote machine, the assistant establishes a pattern of local-first development with remote deployment. This pattern is consistent with the session's overall approach — the assistant works from a development machine and deploys to a production server, maintaining a clean separation between build and deploy phases.

Assumptions and Potential Pitfalls

The message makes several assumptions worth examining. The assistant assumes that port 9101 is available on the remote machine — no check is performed for port conflicts. It assumes that the Python environment at /root/venv_sglang211/bin/python (the environment with pynvml) is the correct interpreter for the exporter. It assumes that systemd is the appropriate service manager and that the unit file syntax is correct. It assumes that Prometheus can be reconfigured by appending to the YAML file and that the reload endpoint will work (an assumption that proves partially incorrect in [msg 13089], where the lifecycle API is not enabled).

The most significant assumption is about the Prometheus reload mechanism. In [msg 13089], the assistant discovers that "Lifecycle API is not enabled" — the Prometheus server rejects the reload request because the --web.enable-lifecycle flag was not set at startup. This is a common operational oversight: Prometheus's HTTP reload endpoint is disabled by default for security reasons. The assistant adapts by falling back to a systemd restart of Prometheus, but the assumption that the reload endpoint would work without verification introduced a minor failure in the deployment flow.

The Thinking Process: Methodical Infrastructure Construction

The reasoning in this message is brief but revealing. The assistant states the plan declaratively: "I'm setting up the GPU exporter deployment with a systemd unit file, configuring Prometheus to scrape metrics from the exporter on port 9101, and then reloading Prometheus to verify the metrics are being collected." This is not exploratory reasoning — it is execution reasoning. The assistant has already decided what to do in the previous messages; this message is about doing it.

The structure of the reasoning shows a methodical approach to infrastructure deployment. The assistant thinks in terms of components and their interactions: the exporter (source of metrics), the systemd unit (lifecycle management), the Prometheus scrape job (data collection), and the reload (activation). Each component is a discrete unit with a clear responsibility. The assistant's mental model is architectural — it sees the monitoring pipeline as a chain of connected services, each with its own configuration and lifecycle.

Notably, the assistant does not reconsider the decision to use a custom exporter versus dcgm-exporter. That decision was settled in the previous round. The reasoning is forward-looking, focused on the mechanics of deployment rather than the trade-offs of design. This is characteristic of an agent that has already committed to a course of action and is executing with purpose.

Conclusion: The Weight of a Simple Message

Message [msg 13088] is a study in how much context a short message can carry. On its face, it is a single file write operation. In context, it is the culmination of a production incident investigation, a deliberate architectural choice between custom and standard tooling, and the bridge between a Python script and a production service. The systemd unit file it creates will enable GPU-level observability that was missing during the cluster's saturation crisis — observability that could prevent the next outage.

The message also illustrates a fundamental pattern in infrastructure engineering: the gap between code and production is bridged by configuration. The Python exporter is the logic; the systemd unit is the policy. Without the unit file, the exporter is ephemeral. With it, the exporter becomes a managed service that the operating system will keep alive, restart on failure, and launch at boot. This transformation — from script to service — is the essence of production deployment, and message [msg 13088] captures it in a single, deceptively simple tool call.