The Question That Built a Monitoring Stack: "do we have any metrcis / prometheus / grafana that we can use to see how full kv-cache is?"

A Single Question, A Pivot in Focus

In the middle of a grueling kernel optimization campaign that had spanned custom MMA attention kernels, indexer bottlenecks, prefill-decode disaggregation, and parser fixes, the user asked a question that at first glance seems almost mundane:

"do we have any metrcis / prometheus / grafana that we can use to see how full kv-cache is?"

This message, indexed as <msg id=12724> in the conversation, is deceptively simple. It contains a typo ("metrcis" for "metrics"), it asks about a specific operational concern (KV cache utilization), and it mentions two specific tools (Prometheus and Grafana). But understanding why this message was written, what it reveals about the state of the deployment, and what it set in motion, requires unpacking the entire arc of the conversation up to this point.

The Context: A System Freshly Brought to Life

The user's question did not come from a blank slate. It came at the end of an extraordinary engineering push that had transformed a struggling inference deployment into a high-performance production service. The assistant had just completed a multi-phase optimization campaign on an 8× RTX PRO 6000 Blackwell (sm_120) system running DeepSeek-V4-Flash-NVFP4.

The journey had been remarkable. The assistant had discovered and fixed an indexer bottleneck that was computing scores over the full ~1M-token max context every decode step, even when actual context was only ~512 tokens. This single fix delivered a 17.9× throughput improvement at concurrency 64 (from 29.7 to 531.7 tok/s). Custom MMA sparse-MLA decode kernels had been written in Triton. Prefill-decode disaggregation had been deployed across 8 GPUs with systemd services. The chat template and reasoning/tool-calling parsers had been fixed and validated through the router.

But all of this work had been focused on making the system work — getting throughput up, getting quality right, getting the architecture stable. What had not yet been addressed was observability. The system was running, but nobody could see inside it.

What the User Was Really Asking

The question "how full kv-cache is" is not just about a single number. It reveals several layers of unspoken need:

First, operational awareness. The KV cache is the most precious resource in an LLM inference server. It represents the accumulated conversational context of all active requests. If it fills up, new requests are rejected or preempted. The user needed to know: are we close to the edge? Is there headroom? The decode server was configured with --mem-fraction-static 0.85 and a context length of 524,288 tokens, giving a total KV capacity of 2,581,504 tokens. But capacity on paper is meaningless without utilization data.

Second, capacity planning. The user was implicitly asking about growth. If the service is handling N concurrent requests today, each consuming K tokens of KV cache, how many more can it handle before hitting the ceiling? This is the kind of question that drives infrastructure decisions: do we need more GPUs? Can we increase the memory fraction? Should we implement KV cache eviction policies?

Third, debugging and anomaly detection. A KV cache that fills unexpectedly can indicate a leak (requests that never release their cache slots), a misconfiguration (context length set too high), or a load spike. Without monitoring, these conditions are invisible until they cause failures.

Fourth, the user was thinking like an operator. This is a subtle but important shift. Earlier in the conversation, the user had been focused on raw performance — throughput, latency, kernel efficiency. Now, with the system deployed and serving requests through a router on port 30001, the user's mindset had shifted to running the system. Monitoring is the first step toward reliability engineering.

The Assumptions Embedded in the Question

The user's question makes several implicit assumptions worth examining:

Assumption 1: That KV cache fullness is a measurable metric. This is correct — SGLang exposes sglang:token_usage (a 0–1 fraction), sglang:kv_used_tokens, sglang:kv_available_tokens, and sglang:max_total_num_tokens as Prometheus gauges on the decode server. But these metrics are only available when --enable-metrics is passed at server launch, which it was not at the time of the question.

Assumption 2: That Prometheus and Grafana are the right tools. The user specifically named these two tools, suggesting familiarity with the Prometheus + Grafana ecosystem. This is the industry-standard monitoring stack for Kubernetes and cloud-native deployments, and it was a reasonable choice. However, neither was installed on the machine. The assistant's investigation would later confirm that Docker was also not available, meaning a binary installation from scratch would be required.

Assumption 3: That the metrics endpoint exists and is accessible. At the time of the question, the decode server was running on 127.0.0.1:30002 and the prefill server on 127.0.0.1:30000, but neither had --enable-metrics in their launch arguments. The router had a Prometheus endpoint on port 29001, but it exposed only request-level metrics (latency, throughput, worker health) — not KV cache utilization. The user may have assumed that because the router had metrics, the servers did too.

Assumption 4: That "how full" is a simple scalar. In reality, KV cache fullness is a per-tensor-parallel-rank metric. The decode server runs with TP=4 across GPUs 4–7, so there are four copies of each gauge, one per TP rank. The user's question implicitly asked for a single "how full" answer, but the underlying data is distributed.

What Knowledge Was Required to Understand This Message

To fully grasp the user's question, one would need to understand:

  1. What the KV cache is — the key-value tensor that stores the computed attention states for each token in the conversation, enabling the model to attend to previous tokens without recomputing them. In the DeepSeek-V4-Flash architecture with Multi-Head Latent Attention (MLA), the KV cache is particularly large because it stores compressed latent representations.
  2. Why KV cache fullness matters — it's the primary memory bottleneck in long-context inference. With 512K context and 2.58M total token capacity, the decode server can hold roughly 5 concurrent 512K-token conversations, or many more shorter conversations. Running out of KV cache slots causes request rejection or preemption.
  3. The monitoring landscape — Prometheus is a time-series database and monitoring system that scrapes metrics endpoints at configurable intervals. Grafana is a visualization layer that queries Prometheus and renders dashboards. Together they form the standard open-source monitoring stack.
  4. The deployment architecture — that there are three services (prefill on GPUs 0–3, decode on GPUs 4–7, router on port 30001), and that KV cache lives on the decode server because it handles the autoregressive generation loop where the cache is written and read.
  5. The SGLang metrics system — that SGLang exposes Prometheus-format metrics at /metrics on the server's HTTP port when --enable-metrics is set, and that the relevant gauge names follow the pattern sglang:<metric_name>{labels}.

The Knowledge Created by This Message

The user's question set in motion a chain of discoveries and infrastructure work that created substantial new knowledge:

Immediate discovery: Metrics were not enabled. The assistant's first investigation ([msg 12725]) revealed that the decode server's /metrics endpoint returned "Not Found." The router's metrics on port 29001 exposed only smg_* metrics (HTTP request counts, worker health, circuit breaker status) — no KV cache data. The assistant had to search the SGLang source code to find the actual metric names, eventually locating them in python/sglang/srt/observability/metrics_collector.py.

Discovery of the gauge names. Through source code inspection ([msg 12729]), the assistant found the exact metric names: sglang:full_token_usage, sglang:kv_used_tokens, sglang:kv_available_tokens, sglang:max_total_num_tokens, plus sglang:cache_hit_rate, sglang:num_running_reqs, and sglang:num_queue_reqs. These names were not documented anywhere in the conversation before this point.

Discovery of the Prometheus format. When the assistant first queried the metrics after enabling them ([msg 12730]), the grep returned empty. The issue was that Prometheus metrics include label sets in curly braces (e.g., sglang:full_token_usage{engine_type="decode",tp_rank="0"}), so the grep pattern needed to match the metric name without requiring a trailing space.

Discovery of the label structure. The metrics include labels for engine_type (decode/prefill), model_name, tp_rank, pp_rank, and moe_ep_rank. This means there are multiple time series per metric — one per tensor-parallel rank. For a TP=4 decode server, there are four sglang:kv_used_tokens gauges, each reporting the local rank's view.

The full monitoring stack installation. The assistant went on to install Prometheus 3.12.0 and Grafana 13.0.2 from binaries (no Docker available), configure Prometheus to scrape all three sglang endpoints, set up Grafana with a provisioned Prometheus datasource, and create a 17-panel KV cache dashboard. This was not a trivial undertaking — it involved downloading tarballs, writing systemd units, configuring provisioning YAML files, and crafting a complex dashboard JSON.

The operational pattern. The assistant also created a simple shell script (/root/kv_usage.sh) that queries the decode server's /metrics endpoint directly and prints a human-readable summary: KV-cache: 12,034 / 2,581,504 tokens (0.47% full) | running=3 queue=0 cache_hit=18.2%. This gave the user an immediate way to check KV fullness without waiting for the full Grafana stack.

The Thinking Process Visible in the Assistant's Response

The assistant's reasoning in the response to this question ([msg 12725]) reveals a structured investigative approach:

The assistant first recognized the gap: "Right now metrics are mostly off: the servers were launched with enable_metrics=False (default), so they don't expose /metrics." This shows an understanding of the SGLang configuration surface.

The assistant then reasoned about where KV cache metrics live: "KV-cache utilization is a per-server gauge (it lives on the decode server, which holds the KV pool)." This reflects architectural knowledge — the KV cache is owned by the decode server in a PD-disaggregated setup, so that's where the metrics must be enabled.

The assistant also considered the router's metrics: "Only the router has a Prometheus endpoint (:29001), but that's request/latency/load — not KV fullness." This shows awareness of the separation of concerns between the router (request routing) and the servers (model inference).

The assistant then formulated a plan: enable --enable-metrics on the servers, find the exact metric names in the SGLang source, and verify they populate correctly. This plan was executed over the next several messages, leading to the full monitoring stack deployment.

Mistakes and Incorrect Assumptions

The assistant made one notable mistake in the initial investigation: the grep pattern used to extract KV metrics from the /metrics endpoint was too strict. The pattern grep -E "^sglang:(full_token_usage|kv_used_tokens) " required a space after the metric name, but Prometheus format places label sets immediately after the name (e.g., sglang:full_token_usage{...}). This caused the initial query to return empty results, leading to a brief moment of confusion before the assistant realized the issue and adjusted the pattern.

This is a classic "format mismatch" error — the assistant assumed a simple key-value format but encountered a labeled Prometheus format. It's a reminder that even when you know the metric names, you still need to understand the serialization format to extract them correctly.

The Broader Significance

The user's question at <msg id=12724> marks a transition point in the conversation. Before this message, the focus was on building — kernel optimization, throughput improvements, deployment architecture. After this message, the focus shifts to operating — monitoring, dashboards, reliability. This is the natural progression of any production deployment: first you make it work, then you make it observable, then you make it reliable.

The question itself is a model of concise, effective communication. In just 16 words (with one typo), the user conveyed: