Building a Custom GPU Exporter: A Decision Point in Production Observability
Introduction
In the middle of a high-stakes production debugging session for a DeepSeek V4 inference cluster running on 8 Blackwell GPUs, a single message from the AI assistant marks a pivotal shift from reactive firefighting to proactive infrastructure building. The message at index 13087 is not about fixing a crash or diagnosing a throughput regression—it is about choosing to build a custom GPU metrics exporter from scratch rather than relying on off-the-shelf solutions. This decision, captured in a few paragraphs of reasoning and a single file write, embodies the trade-offs that define production AI engineering: speed versus completeness, custom versus standard, and the constant tension between fixing the immediate problem and preventing the next one.
The Message
The assistant's message reads in full:
## Agent Reasoning I can use pynvml to build a GPU exporter since it works with the 8 GPUs available. The hicache metrics already have the engine_type="prefill" label I need, and since the Grafana provisioning directories are empty, the dashboard was pushed via the HTTP API—likely through a curl request to the Grafana endpoint with admin credentials. I need to find where that upload mechanism is implemented.
>
My plan is straightforward: write a minimal GPU exporter in Python using pynvml, run it as a systemd service on port 9101, add a Prometheus scrape job, then update the dashboard generator to include node health metrics (service status, prefill queue depth, GPU utilization and memory) and HiCache metrics (host usage, token counts, cache hit rates). After regenerating the dashboard JSON, I'll locate the upload script and credentials to push it to Grafana.
>
Let me search the workspace for how the dashboard gets uploaded.
>
Writing GPU exporter... Writing the exporter... pynvml works (8 GPUs), HiCache metrics carry engine_type="prefill", and Grafana provisioning dirs are empty (dashboard pushed via API). Let me complete real node health by standing up a tiny GPU exporter, then build the dashboard. Writing the exporter: [write] /tmp/opencode/gpu_exporter.py Wrote file successfully.
This message is deceptively simple. On its surface, it announces the creation of a GPU exporter. But embedded within it is a chain of reasoning that draws on system administration knowledge, production debugging experience, and an understanding of the trade-offs between speed and completeness in incident response.
Why This Message Was Written: The Motivation and Context
To understand why this message exists, we must trace back through the preceding messages. The cluster had suffered a production incident where it became unresponsive under load, returning KVTransferError aborts. The root cause was a single prefill server whose unbounded queue had accumulated ~20 requests and ~220K pending tokens, causing time-to-first-token to balloon to minutes, clients to abort, and in-flight KV transfers to fail. The assistant had already implemented admission control (--max-queued-requests 32) and enabled HiCache hierarchical caching for prefix reuse, but these were fixes for the symptoms of a blind spot. The cluster had no way to signal that it was becoming saturated before it collapsed.
The assistant's reasoning in the preceding message ([msg 13086]) reveals this awareness explicitly: "For the node-health dashboard row, I'm prioritizing the service status indicators, prefill queue depth as the main stall predictor with color thresholds, decode KV saturation percentage, retracted and paused request counts to show backpressure, and router request rates with error status if available." The assistant recognized that the real failure was not a software bug but a monitoring gap. The cluster had been flying blind.
When the assistant checked for GPU monitoring infrastructure, it found none: "No GPU/node exporter is running — Prometheus only scrapes the 3 sglang endpoints" ([msg 13085]). The Prometheus instance was collecting metrics from the SGLang services themselves (prefill, decode, router), but there was no way to see GPU utilization, memory pressure, or temperature—the very metrics that would have predicted the saturation failure. The assistant weighed whether to install dcgm-exporter (NVIDIA's official GPU metrics exporter) or build something custom, and the pynvml check in [msg 13086] was the decisive moment: "pynvml OK, GPUs: 8."
How Decisions Were Made
The decision to build a custom exporter rather than install dcgm-exporter reveals the assistant's prioritization framework. The reasoning, visible across the two messages, weighs several factors:
Speed of deployment. The assistant could have downloaded the dcgm-exporter binary, configured it, and set it up as a systemd service—but that would have required finding the right package, verifying compatibility with Ubuntu 24.04 and the NVIDIA driver version (590.48.01), and potentially resolving dependency issues. Building a 50-line Python script using pynvml, which was already installed and working in the SGLang virtual environment, was faster. The assistant explicitly notes this trade-off in [msg 13085]: "I'll do a quick check for dcgm-exporter, but if it's not trivial to set up, I'll skip GPU for now and note it as a follow-up option."
Minimality. The assistant did not need the full dcgm-exporter feature set. It needed GPU utilization percentage and memory usage—two metrics that pynvml exposes with a single API call each. A custom exporter that exposes exactly these metrics, with no configuration overhead, is simpler to deploy and debug than a general-purpose exporter with dozens of metrics and a complex configuration surface.
Integration with existing infrastructure. The assistant already had a Prometheus server scraping the SGLang endpoints. Adding a new scrape target for a custom exporter on port 9101 was a single YAML addition. The assistant also knew that the Grafana dashboard was being pushed via the HTTP API (since provisioning directories were empty), and it had a dashboard generator script (gen_dashboard.py) that it could extend. Building the exporter was the missing link that completed the observability chain: exporter → Prometheus → Grafana.
The "just works" factor. The pynvml check in [msg 13086] confirmed not just that the library was importable, but that it could initialize the NVML library and enumerate all 8 GPUs. This eliminated the risk of spending time on an exporter that would fail at runtime due to driver or library incompatibilities.
Assumptions Made by the Assistant
Several assumptions underpin this message, and understanding them is crucial to evaluating the decision:
Assumption 1: The pynvml-based exporter will be stable enough for production. The assistant is writing a script that runs as a systemd service, continuously scraping NVML metrics and serving them over HTTP. This is a reasonable assumption—pynvml is a stable, well-maintained library—but it is an assumption nonetheless. A crash in the exporter would silently remove GPU metrics from Grafana without alerting anyone.
Assumption 2: The Prometheus server can be restarted safely. The assistant later discovers that the Prometheus lifecycle API is disabled ([msg 13089]), requiring a full service restart to pick up the new scrape target. This is a non-trivial operation in a production environment—a restart means a brief gap in metric collection. The assistant handles this by verifying the YAML syntax before restarting ([msg 13090]), but the assumption that a restart is safe is implicit.
Assumption 3: The Grafana upload mechanism will be found in the workspace. The assistant says "Let me search the workspace for how the dashboard gets uploaded." This assumes that the upload script or credentials are stored locally, which may or may not be true. The Grafana credentials could be in a secure vault or environment variable that the assistant cannot access.
Assumption 4: Port 9101 is available. The assistant chooses port 9101 for the GPU exporter without checking whether it is already in use. This is a common Prometheus exporter convention (node_exporter uses 9100, so 9101 is a natural increment), but it is not guaranteed to be free.
Mistakes or Incorrect Assumptions
The most notable incorrect assumption is the Prometheus lifecycle API availability. In [msg 13089], the assistant appends the GPU job to prometheus.yml and attempts a hot reload via curl -s -X POST http://127.0.0.1:9090/-/reload, only to discover "Lifecycle API is not enabled." This forces a full Prometheus restart, which the assistant handles in [msg 13090] by finding the correct service name and restarting it. The mistake is minor—the assistant recovers gracefully—but it highlights the danger of assuming standard configurations in production environments.
Another potential oversight is the lack of error handling in the exporter script. The assistant writes the exporter in [msg 13087] but does not show the content in the message. The subsequent deployment in [msg 13089] reveals that the exporter is serving metrics successfully, but we do not know whether the script handles NVML initialization failures, GPU hot-plug events, or HTTP server crashes gracefully. In a production context, these edge cases matter.
Input Knowledge Required to Understand This Message
To fully grasp the significance of this message, a reader needs knowledge in several domains:
Prometheus and Grafana observability stack. The assistant references Prometheus scrape jobs, Grafana provisioning directories, the HTTP API upload mechanism, and dashboard generator scripts. Understanding why the empty provisioning directories imply API-based upload requires familiarity with Grafana's two dashboard management methods.
NVML and GPU monitoring. The assistant uses pynvml, the Python binding for NVIDIA's Management Library (NVML). Knowing that NVML provides low-level GPU metrics (utilization, memory, temperature, power) through a C API, and that pynvml wraps this API in Python, explains why the assistant can build a custom exporter with just a few lines of code.
Systemd service management. The assistant plans to run the exporter as a systemd service, which requires writing a unit file, enabling the service, and managing its lifecycle. This is standard Linux systems administration knowledge.
SGLang architecture. The assistant references the prefill-decode disaggregated architecture, where separate servers handle prompt processing (prefill) and token generation (decode). Understanding why the prefill queue depth is the critical saturation indicator—because the single prefill server is the bottleneck that backs up under load—requires knowledge of this architecture.
The production incident history. The assistant is building this exporter in direct response to a production outage where queue saturation caused cascading failures. Without this context, the exporter might seem like premature optimization rather than a targeted fix for a known blind spot.
Output Knowledge Created by This Message
This message produces several concrete outputs:
The GPU exporter script (/tmp/opencode/gpu_exporter.py). While the content is not shown in the message, the subsequent deployment reveals that it exposes at least gpu_util_percent and gpu_mem_used_percent metrics per GPU, with a gauge metric type and a gpu label for GPU index.
A deployment plan. The assistant outlines a clear, multi-step plan: write the exporter, create a systemd unit, deploy via SCP, enable the service, add a Prometheus scrape job, reload Prometheus, update the Grafana dashboard generator, regenerate the dashboard JSON, and upload it via the API. This plan is executed in the following messages ([msg 13088], [msg 13089], [msg 13090]).
A decision record. The message documents the reasoning behind choosing a custom exporter over dcgm-exporter: pynvml availability, speed of deployment, minimality, and integration with existing infrastructure. This is valuable for future operators who might wonder why a non-standard exporter is running on the cluster.
A knowledge boundary. The assistant identifies a gap in its knowledge: "I need to find where that upload mechanism is implemented." This acknowledgment of uncertainty is important—it prevents the assistant from making assumptions about credentials and API endpoints that could lead to errors.
The Thinking Process: A Window into Production Engineering
The assistant's reasoning in this message reveals a sophisticated production engineering mindset. The thought process moves through several stages:
1. Opportunity identification. The assistant recognizes that pynvml is available and working ("pynvml OK, GPUs: 8"). This is not just a technical check—it is the identification of an opportunity to close the observability gap with minimal effort.
2. Plan formulation. The assistant lays out a clear sequence: write exporter → systemd service → Prometheus job → dashboard update → upload. This is not a vague intention but an executable plan with specific ports (9101), specific files (gpu_exporter.py), and specific integration points (Prometheus scrape config, Grafana API).
3. Dependency discovery. The assistant notes that the Grafana provisioning directories are empty, implying API-based upload. This is a critical discovery—it means the dashboard cannot be updated by dropping a JSON file into a directory; it requires finding the upload script and credentials.
4. Trade-off evaluation. The assistant implicitly evaluates the custom-vs-standard trade-off and concludes that a custom exporter is the right choice. This evaluation is not spelled out in detail, but the speed of the decision (the exporter is written in the same message) suggests that the assistant has internalized these trade-offs and can make them quickly.
5. Execution. The assistant writes the exporter file immediately, without waiting for further confirmation. This is characteristic of effective incident response: when the path is clear, execute without delay.
Conclusion
Message 13087 is a turning point in the production debugging session. Before it, the assistant was reacting to failures—crash loops, queue saturation, KV transfer errors. After it, the assistant is building infrastructure to prevent those failures from happening again. The custom GPU exporter is small—probably fewer than 100 lines of Python—but it represents a shift from reactive to proactive operations. It closes the observability gap that allowed the cluster to saturate silently, and it does so with a speed and minimality that off-the-shelf solutions could not match.
The message also illustrates a deeper truth about production AI engineering: the most important decisions are often not about algorithms or model architectures, but about infrastructure. Knowing when to build custom versus when to install standard, when to fix the symptom versus when to close the blind spot, and when to execute versus when to investigate—these are the judgments that separate stable deployments from fragile ones. In this message, the assistant makes those judgments correctly, and the result is a cluster that will be measurably more observable than it was an hour before.