The Final Piece of the Monitoring Stack: Deploying Grafana as a Systemd Service

Introduction

In the middle of an intensive optimization campaign for DeepSeek-V4-Flash running on 8× RTX PRO 6000 Blackwell GPUs, the assistant reaches a pivotal operational moment: deploying Grafana as a production-grade monitoring service. Message [msg 12741] represents the culmination of a multi-step infrastructure buildout—the moment when the monitoring stack transitions from a collection of downloaded binaries and configuration files into a live, verified, and persistent service. This message, though outwardly simple, reveals deep operational reasoning about service management, startup verification, and the subtle edge cases that separate a working deployment from a broken one.

Context and Motivation

To understand why this message exists, one must trace the conversation that led to it. The assistant had just completed a remarkable engineering campaign: designing custom MMA sparse-MLA decode kernels, fixing an indexer O(max_context) bottleneck that yielded a ~17× throughput improvement, deploying prefill-decode disaggregation across 8 GPUs with systemd services, and resolving tool-calling quality issues. The user, now satisfied with the inference performance, turned their attention to observability, asking specifically about Prometheus and Grafana for KV cache monitoring ([msg 12733]).

The assistant had already ensured that the sglang services exposed Prometheus-format metrics on their respective ports—the decode server on :30002, the prefill server on :30000, and the router on :29001. A simple shell script (kv_usage.sh) provided real-time KV fullness polling. But the user wanted more: a proper dashboard with historical trends, something viewable in a browser. When asked, the user confirmed: "Yes — install Prometheus + Grafana + KV dashboard" ([msg 12734]).

What followed was a systematic infrastructure deployment. The assistant checked for Docker (not available), verified network connectivity and port availability, downloaded Prometheus 3.12.0 and Grafana 13.0.2 as binaries, configured Prometheus to scrape all three sglang endpoints, verified it was collecting metrics, wrote a comprehensive KV-cache dashboard JSON with 17 panels, and set up Grafana's provisioning directory structure with datasource and dashboard provider configurations. Message [msg 12741] is the final step: wiring Grafana into the system's service management layer and verifying it works end-to-end.

The Systemd Service Design

The Grafana systemd unit the assistant creates reveals several deliberate design decisions:

[Unit]
Description=Grafana
After=network-online.target prometheus.service
Wants=network-online.target

The After=prometheus.service dependency is a thoughtful touch. Grafana requires a Prometheus datasource to be useful; by declaring this ordering dependency, the assistant ensures that on system boot, Prometheus starts before Grafana. This prevents the race condition where Grafana initializes, attempts to connect to its Prometheus datasource, finds nothing listening, and either errors or starts with a broken datasource. The Wants=network-online.target (rather than Requires=) is similarly pragmatic—it signals that network connectivity is desired but not strictly mandatory for the service to start, allowing Grafana to come up even in degraded network states.

The ExecStart line is equally deliberate:

ExecStart=/opt/grafana/bin/grafana server --homepath=/opt/grafana --config=/opt/grafana/conf/custom.ini

The assistant explicitly passes --homepath and --config rather than relying on environment variables or default paths. This is a robustness pattern: by making the paths explicit in the systemd unit, the service becomes self-documenting and immune to changes in environment variables or working directory assumptions. A future administrator reading this unit file can immediately see where Grafana's binaries, configuration, and data live.

The Restart=always with RestartSec=10 is another production-oriented choice. ML inference services can be resource-intensive; if Grafana crashes due to memory pressure or a transient bug, automatic restart with a 10-second cooldown ensures monitoring recovers without human intervention. The WorkingDirectory=/opt/grafana provides a consistent execution context.

The Startup Verification Pattern

Perhaps the most revealing part of this message is the startup verification loop:

for i in $(seq 1 8); do
  sleep 8
  h=$(timeout 8 ssh ... curl -s --max-time 5 -o /dev/null -w "%{http_code}" http://127.0.0.1:3000/api/health 2>/dev/null)
  echo "[grafana health] http=$h"
  [ "$h" = "200" ] && break
done

This pattern encodes several assumptions and operational insights:

Grafana startup is not instantaneous. The assistant knows that Grafana performs database migrations, plugin initialization, and provisioning file parsing on first startup. An 8-second polling interval with up to 8 retries (64 seconds total) provides a generous window for initialization. This is a pragmatic choice: too short an interval would generate noise; too few retries would fail on slow boots.

Health checks are the correct verification mechanism. Rather than checking if the process is running (which would succeed before Grafana is ready to serve), the assistant polls the /api/health endpoint. This is the proper approach—it verifies that Grafana has completed its initialization sequence, the database is operational, and the HTTP server is accepting requests.

The timeout wrapper prevents hanging. Each health check is wrapped in timeout 8, ensuring that a hung or slow response doesn't cause the verification loop to stall indefinitely. This is defensive programming for remote SSH execution.

Early exit on success. The [ "$h" = "200" ] && break pattern means the loop exits as soon as Grafana is healthy, rather than waiting for all 8 iterations. This minimizes unnecessary latency in the deployment process.

Verification and a Subtle Edge Case

After Grafana is confirmed healthy, the assistant runs a comprehensive verification:

echo "=== grafana health ==="; curl -s --max-time 5 http://127.0.0.1:3000/api/health
echo; echo "=== dashboard provisioned? ==="; curl -s --max-time 5 "http://admin:admin@127.0.0.1:3000/api/search?query=KV"
echo "=== listening 0.0.0.0:3000 ==="; ss -tlnp 2>/dev/null | grep ":3000" | head -1

The results are mostly positive:

Assumptions and Required Knowledge

This message operates on several implicit assumptions:

Binary installation is appropriate. The assistant chose to download Prometheus and Grafana binaries directly rather than using Docker (which wasn't available) or apt packages. This assumes that the administrator is comfortable with manual binary management and that the system has the necessary libraries (which it does, being Ubuntu 24.04).

Systemd is the correct service manager. On Ubuntu 24.04, this is a safe assumption. The assistant leverages systemd's dependency management (After=), restart policies, and enable-at-boot semantics.

The Grafana provisioning API is stable. The assistant relies on file-based provisioning for datasources and dashboards, which is a Grafana feature introduced in version 5.0 and has been stable through version 13.0.2. This is a reasonable assumption for a production deployment.

The dashboard JSON schema is correct. The assistant wrote a 17-panel dashboard JSON file and trusted that Grafana would parse it correctly via provisioning. This assumes familiarity with Grafana's dashboard model, including panel types, query targets, and grid layout.

Default credentials are acceptable for initial setup. The Grafana config sets admin_user = admin and admin_password = [REDACTED]. This is standard practice for initial deployment, with the understanding that credentials should be changed for production. The assistant does not change these, leaving that as an exercise for the operator.

Input and Output Knowledge

Input knowledge required to understand this message includes: familiarity with systemd unit file syntax and semantics (dependencies, restart policies, environment); understanding of Grafana's architecture (server mode, provisioning, health API); knowledge of the sglang deployment topology (prefill on port 30000, decode on port 30002, router on port 29001, router on port 30001); and awareness of the earlier monitoring setup (Prometheus scraping, KV metrics, dashboard JSON).

Output knowledge created by this message is substantial: a working Grafana instance bound to 0.0.0.0:3000 with anonymous viewer access; a provisioned Prometheus datasource pointing to the local Prometheus instance; a provisioned KV-cache dashboard titled "SGLang DeepSeek-V4 — KV Cache & Serving" with 17 panels covering KV usage, throughput, latency percentiles, PD-disagg transfer metrics, cache hit rate, and request rates; a systemd service that survives reboots; and verified end-to-end functionality from health API to dashboard search.

The Thinking Process

The assistant's reasoning, visible in the "Agent Reasoning" preamble, is concise but revealing: "Now I'm setting up the Grafana systemd service unit that starts the Grafana server with the custom configuration, then I'll create it and start the service to verify it's running." This straightforward statement belies the complexity of what follows. The assistant is executing a well-understood playbook: create the unit, enable and start it, poll for health, verify functionality. The reasoning shows no hesitation or uncertainty—the assistant has done this before and knows the pattern.

The polling loop with 8-second intervals and 8 retries suggests an empirical understanding of Grafana's startup time. The assistant doesn't hardcode a fixed sleep (e.g., sleep 30) but instead polls adaptively, exiting early on success. This is a more robust pattern that works across varying system loads and hardware configurations.

The verification steps show a hierarchy of confidence: first health (is the server running?), then dashboard provisioning (is the configuration correct?), then port binding (is it accessible on the network?). Each step builds on the previous one, creating a chain of evidence that the deployment is complete.

Conclusion

Message [msg 12741] is a masterclass in operational deployment. It demonstrates how to take a complex piece of infrastructure—Grafana with provisioning, datasource integration, and a custom dashboard—and wire it into a production system with proper service management, startup verification, and validation. The message reveals the assistant's deep understanding of Linux service management, Grafana's initialization sequence, and the importance of defensive programming in deployment scripts.

The subtle port-grep bug serves as a reminder that even careful verification can have blind spots. But the overall pattern—create, enable, poll, verify—is sound and reproducible. For anyone deploying monitoring stacks alongside ML inference services, this message offers a template: use systemd for persistence, poll health endpoints for readiness, verify provisioning programmatically, and always include a chain of evidence that the service is working as intended.