Deploying GPU Monitoring Infrastructure: A Case Study in Production ML Observability

Introduction

In the high-stakes world of production machine learning deployments, observability is not a luxury—it is a necessity. When a cluster serving a 671-billion-parameter language model across eight Blackwell GPUs becomes unresponsive under load, the ability to quickly distinguish between a queue saturation problem, a GPU memory exhaustion issue, or a kernel compilation deadlock is what separates a five-minute fix from a five-hour firefight. Message [msg 13089] captures a pivotal moment in the operational lifecycle of a DeepSeek-V4-Flash deployment: the deliberate construction of a GPU monitoring pipeline from scratch, using a custom Python exporter, systemd integration, and Prometheus configuration. This seemingly straightforward bash command embodies a wealth of engineering judgment, trade-off analysis, and practical wisdom about running large language models in production.

The Message

The subject message consists of a single bash command that orchestrates the deployment of a GPU metrics exporter across three distinct phases: file transfer, service installation and verification, and Prometheus integration. Here is the message in full:

scp -q /tmp/opencode/gpu_exporter.py root@10.1.230.171:/root/gpu_exporter.py
scp -q /tmp/opencode/gpu-exporter.service root@10.1.230.171:/etc/systemd/system/gpu-exporter.service
ssh root@10.1.230.171 'set +e
systemctl daemon-reload
systemctl enable --now gpu-exporter 2>&1 | tail -1
sleep 2
echo "gpu-exporter: $(systemctl is-active gpu-exporter)"
echo "=== sample GPU metrics ==="; curl -s http://127.0.0.1:9101/ 2>/dev/null | grep -E "gpu_util_percent|gpu_mem_used_percent" | head -4
echo "=== add prometheus job (if absent) ==="
grep -q "job_name: gpu" /etc/prometheus/prometheus.yml || python3 - <<"PY"
p="/etc/prometheus/prometheus.yml"; s=open(p).read()
job="\n  - job_name: gpu\n    static_configs:\n      - targets: [\"127.0.0.1:9101\"]\n"
# append under scrape_configs (assume it ends file); safe append
open(p,"a").write(job)
print("appended gpu job")
PY
echo "--- reload prometheus ---"
curl -s -X POST http://127.0.0.1:9090/-/reload 2>/dev/null; sleep 3
curl -s "http://127.0.0.1:9090/api/v1/query?query=up{job=\"gpu\"}" 2>/dev/null | python3 -c "import sys,json;d=json.load(sys.stdin);print(\"gpu target up:\",[r[\"value\"][1] for r in d[\"data\"][\"result\"]])" 2>&1' 2>&1

The output reveals a mixed result: the service was created and activated successfully, GPU metrics began flowing (gpu_util_percent, gpu_mem_used_percent), the Prometheus job was appended to the configuration file, but the lifecycle reload endpoint returned an error—Prometheus was started without --web.enable-lifecycle, so the configuration change would not take effect until a manual restart.

Why This Message Was Written: The Observability Gap

To understand the motivation behind this message, we must trace back through the preceding conversation. The cluster had suffered a production incident where it became unresponsive under load, returning KVTransferError aborts to clients ([msg 13085] context). The assistant diagnosed the root cause as a single prefill server with an unbounded queue that accumulated approximately 20 requests and 220,000 pending tokens, causing time-to-first-token to balloon to minutes. Clients aborted, in-flight KV transfers failed, and the entire system entered a degraded state.

The critical observation was that no GPU-level metrics were available to aid this diagnosis. The Prometheus instance was scraping only the three SGLang service endpoints (prefill, decode, router). There was no node exporter, no dcgm-exporter, and no mechanism to track GPU utilization, memory pressure, or temperature. The assistant had to reason about the failure purely through application-level metrics—queue depths, request counts, and KV cache usage. While sufficient for the immediate diagnosis, this gap represented a significant blind spot for future incidents.

When the user subsequently requested Grafana dashboards with "node health indicators" ([msg 13085]), the assistant recognized an opportunity to close this observability gap. The decision to build a custom GPU exporter rather than deploy an existing solution like dcgm-exporter or nvidia-smi's built-in Prometheus endpoint was itself a significant engineering judgment call, reflecting the specific constraints of this deployment environment.## The Decision to Build a Custom Exporter

The assistant's reasoning process, visible in the preceding messages, reveals a careful weighing of alternatives. The standard approach for NVIDIA GPU monitoring in Prometheus environments is dcgm-exporter, a production-grade tool from NVIDIA that exposes a rich set of GPU metrics via a Prometheus-compatible endpoint. However, deploying dcgm-exporter requires downloading a binary (or installing via a package manager), setting up a systemd service, and ensuring compatibility with the driver version—in this case, NVIDIA driver 590.48.01 on Ubuntu 24.04. The assistant's check in [msg 13086] confirmed that pynvml was available in the Python virtual environment, and that nvidia-smi was present. Rather than introduce a new binary dependency, the assistant chose to write a lightweight Python exporter using pynvml—a library already present and working.

This decision embodies several implicit assumptions:

The pynvml library is sufficient for the required metrics. The assistant assumed that GPU utilization percentage and memory usage percentage—the two metrics confirmed in the output—would be adequate for node health monitoring. More exotic metrics like GPU temperature, power draw, PCIe throughput, or memory bandwidth were deemed unnecessary for the immediate goal of detecting queue saturation versus GPU exhaustion.
A custom exporter is maintainable in this context. The assistant was operating in a research/deployment environment where the same engineer who writes the exporter also manages the infrastructure. In a larger team with formal SRE practices, a custom exporter might be frowned upon in favor of standard tooling. But here, the trade-off favored speed and simplicity.
The Prometheus lifecycle API would work. This assumption proved incorrect—the Prometheus server was started without --web.enable-lifecycle, so the /-/reload endpoint returned an error. The assistant's error handling was minimal: the set +e at the top of the remote script allowed execution to continue past the failure, and the subsequent query for up{job="gpu"} would naturally return empty until the configuration was reloaded manually. This is a minor mistake, but an instructive one: it reveals the tension between automation and the reality of pre-existing infrastructure configurations.

The Engineering Structure: A Three-Phase Deployment

The message orchestrates a deployment in three distinct phases, each with its own failure modes and verification steps.

Phase 1: File Transfer

The two scp commands transfer the exporter script and the systemd unit file to the remote host. The -q flag suppresses progress output, keeping the result clean for parsing. The choice of /root/gpu_exporter.py as the destination reflects a practical judgment: the exporter runs as root (via systemd), so placing it in root's home directory avoids permission issues. The systemd unit file goes to /etc/systemd/system/, the standard location for system-wide services.

Phase 2: Service Installation and Verification

The remote script begins with systemctl daemon-reload to register the new unit file, then systemctl enable --now gpu-exporter to start the service and configure it to start on boot. The 2>&1 | tail -1 captures only the last line of output, typically the success message ("Created symlink..."). A two-second sleep follows—a heuristic wait for the service to initialize and bind to its port.

The verification step is elegant: it queries the exporter's HTTP endpoint directly (curl -s http://127.0.0.1:9101/) and filters for the two key metrics. The output confirms both gpu_util_percent and gpu_mem_used_percent are being served, with GPU 0 showing 0% utilization and 80.90% memory usage. This single data point already provides actionable information: the GPU is idle but nearly full on memory, consistent with a model-loaded-but-not-serving state.

Phase 3: Prometheus Integration

The Prometheus integration uses an idempotent pattern: grep -q "job_name: gpu" /etc/prometheus/prometheus.yml || python3 - <<"PY". This checks whether the job already exists in the configuration file, and only appends it if absent. The inline Python script reads the existing configuration, appends a new job targeting 127.0.0.1:9101, and writes the result back. The comment "safe append" reveals the assistant's awareness that this approach is fragile—it assumes the file ends cleanly and that appending is sufficient to add a new scrape job under the existing scrape_configs block. In a properly structured YAML file, this would work; in a malformed or unusually structured file, it could create duplicate or nested entries.

The final verification step queries Prometheus for up{job="gpu"}—a metric that Prometheus generates natively for each scrape target. The fact that this query returned empty (implied by the error message about the lifecycle API) confirms that the configuration change was not applied.

What Went Wrong and What Was Learned

The most visible mistake in this message is the failed Prometheus reload. The curl -s -X POST http://127.0.0.1:9090/-/reload returned an error because the Prometheus server was not started with the --web.enable-lifecycle flag. This is a common configuration oversight—the lifecycle API is disabled by default for security reasons, and must be explicitly enabled. The assistant's error handling (set +e) allowed the script to continue, but the subsequent verification step naturally failed because the new scrape job was never loaded.

This failure mode reveals an important assumption: the assistant assumed that the Prometheus server was configured with standard defaults that include the lifecycle API. In practice, many production Prometheus deployments disable this API and rely on file watches or SIGHUP signals for configuration reloads. The assistant could have used kill -HUP $(pidof prometheus) as an alternative, or checked for the lifecycle flag before attempting the reload.

A secondary issue is the fragility of the YAML append approach. While it worked in this case (the output confirms "appended gpu job"), it would break if the Prometheus configuration file had a different structure—for example, if scrape_configs was not the final block, or if the file used indentation that the simple append didn't match. A more robust approach would use a YAML-aware tool like yq or a Python script that parses and modifies the configuration structurally.

The Broader Context: Building Production Observability

This message is part of a larger arc in the conversation: the systematic construction of a production monitoring stack for a complex ML deployment. Earlier messages in this segment show the assistant diagnosing a production incident, implementing admission control to prevent queue saturation, enabling HiCache hierarchical caching for prefix reuse, and now building GPU-level monitoring. Each step layers additional observability on top of the previous work.

The Grafana dashboard that the assistant was simultaneously building ([msg 13085]) would eventually incorporate these GPU metrics alongside service health indicators, prefill queue depth, KV cache usage, and HiCache statistics. The GPU exporter is the final piece that transforms the dashboard from an application-monitoring view into a true node-health view, capable of distinguishing between "the model is serving slowly because the GPU is compute-bound" and "the model is serving slowly because the queue is backed up."

Conclusion

Message [msg 13089] is a masterclass in practical production engineering for machine learning deployments. It demonstrates the ability to identify an observability gap, design a lightweight solution using available tools, deploy it with appropriate automation, and verify its operation—all while managing the inevitable failures that arise when theory meets practice. The failed Prometheus reload is not a mark of poor engineering; it is a normal part of operating complex systems. What matters is that the assistant recognized the gap, acted to close it, and left the system in a state where a simple Prometheus restart would complete the integration.

The knowledge created by this message is twofold: first, a working GPU monitoring pipeline that feeds metrics into Prometheus and ultimately into Grafana dashboards; second, a documented pattern for deploying lightweight Python-based exporters in environments where standard tooling is not available or appropriate. For anyone operating large language models in production, this pattern—identify the metric gap, build the simplest possible bridge, iterate—is far more valuable than any single tool.