The KV-Cache Gauge: Building Observability Into a Production Inference Stack

Introduction

In the high-stakes world of large language model serving, observability is not a luxury—it is a prerequisite for reliable operation. When a model like DeepSeek-V4-Flash-NVFP4 is deployed across eight RTX PRO 6000 Blackwell GPUs with prefill-decode disaggregation, serving thousands of tokens per second, the operator needs to know one thing above all others: how full is the KV cache? The answer determines whether the next request will be accepted or rejected, whether the system is approaching a memory-pressure event, and whether capacity planning is on track. This article examines a single message in an opencode coding session—message 12732—where an AI assistant, after a multi-step investigation, delivers exactly that answer by exposing KV-cache metrics and building a reusable monitoring script.

The message sits at the intersection of infrastructure engineering, observability design, and practical tool-building. It is not a flashy kernel optimization or a breakthrough in speculative decoding; it is the quieter, equally essential work of making a production system legible to its operators. To understand why this message matters, we must trace the chain of reasoning that produced it, examine the decisions embedded in its tool calls, and appreciate the assumptions—both correct and mistaken—that shaped its outcome.

The Context: A Production System Without a Dashboard

The conversation leading to message 12732 is set in the final stages of a massive engineering campaign. Over the preceding segments, the assistant had optimized DeepSeek-V4-Flash on Blackwell GPUs through custom MMA attention kernels, discovered and fixed an O(max_context) indexer bottleneck that yielded a ~17× throughput gain, deployed prefill-decode disaggregation across eight GPUs with systemd services, and resolved tool-calling quality issues. The system was running: prefill on GPUs 0–3 (NUMA0), decode on GPUs 4–7 (NUMA1), with a router on port 30001 providing the OpenAI-compatible endpoint. KV capacity was set to 2.58 million tokens at a 512K context window.

But observability was an afterthought. The router exposed Prometheus metrics on port 29001—HTTP request counts, latency buckets, worker health—but those were router-level metrics, not server-level. The decode server, which actually held the KV cache pool, had metrics disabled by default. The prefill server was similarly dark. The user's question in message 12724—"do we have any metrics / prometheus / grafana that we can use to see how full kv-cache is?"—exposed a critical gap. The answer was effectively "no."

This launched a rapid investigation spanning messages 12725 through 12731. The assistant first checked the router's metrics endpoint, finding only smg_* router metrics. It then searched the SGLang source tree for KV-cache gauge names, discovering sglang:full_token_usage, sglang:kv_used_tokens, sglang:kv_available_tokens, and sglang:max_total_num_tokens in the observability module. It added --enable-metrics to both server launch scripts, restarted the systemd services, waited through a 80-second restart cycle, and queried the metrics endpoint. The first query returned empty—a grep pattern too strict for Prometheus's label-enriched format. A corrected query, preceded by a warmup request to populate the gauges, finally succeeded, revealing the KV metrics in full Prometheus exposition format with labels for engine_type, model_name, moe_ep_rank, pp_rank, and tp_rank.

Message 12732 is the culmination of that investigation: the moment the assistant reports success and delivers a practical tool.

The Message: Reporting and Tool-Building

The message opens with an Agent Reasoning block that reveals the assistant's thought process:

I'm looking at the KV cache metrics exposed on the decode server, which show the total capacity at about 2.58 million tokens with all of it currently available since the system is idle, and zero tokens in use right now. The user wants a way to monitor KV fullness over time, and while the metrics are now exposed in Prometheus format on the decode server, setting up a full Prometheus + Grafana stack would be a bigger undertaking since neither is installed yet. I should offer both options: querying the metrics directly for immediate visibility, or helping set up Prometheus and Grafana for a proper monitoring dashboard. I should give them a practical way to watch KV fullness in real-time with a simple command that polls the metrics endpoint, then ask if they want me to set up the full Prometheus and Grafana stack for persistent monitoring and dashboards. Writing the helper script...

This reasoning reveals a deliberate two-track strategy. The assistant recognizes that the user asked about Prometheus and Grafana specifically, but also understands the engineering principle of "make it work, then make it pretty." Rather than diving into a full Prometheus + Grafana installation—which would require downloading binaries, configuring scrapers, provisioning dashboards—the assistant first delivers immediate value by exposing the raw metrics and building a lightweight query tool. The full stack is offered as a follow-up, not deferred indefinitely.

The message then presents a clean table of the key metrics and their idle values:

| metric | value | meaning | |---|---|---| | sglang:max_total_num_tokens | 2,581,504 | total KV capacity | | sglang:kv_used_tokens | 0 | tokens in use | | sglang:kv_available_tokens | 2,581,504 | free | | sglang:token_usage | 0.0 | fraction full (0-1) | | sglang:num_running_reqs / num_queue_reqs | 0 / 0 | load |

This table is notable for its clarity. The assistant has selected five metrics from what is likely a much larger Prometheus exposition (including per-TP-rank variants) and distilled them into a human-readable summary. The token_usage metric is highlighted as the "fraction full (0-1)"—the single number that answers the user's original question most directly.

The bash tool call then creates the helper script:

ssh -o StrictHostKeyChecking=no root@10.1.230.171 'cat > /root/kv_usage.sh << "EOF"
#!/bin/bash
# Live KV-cache fullness from the decode server metrics.
/root/venv_sglang211/bin/python - << "PY"
import urllib.request
m = urllib.request.urlopen("http://127.0.0.1:30002/metrics", timeout=5).read().decode()
def g(name):
    for line in m.splitlines():
        if line.startswith(f"sglang:{name}{{") and "tp_rank=\"0\"" in line:
            return float(line.rsplit(" ",1)[1])
    return 0.0
used,mx,use=g("kv_used_tokens"),g("max_total_num_tokens"),g("token_usage")
run,q,hit=g("num_running_reqs"),g("num_queue_reqs"),g("cache_hit_rate")
print(f"KV-cache: {used:,.0f} / {mx:,.0f} tokens  ({use*100:5.2f}% full) | running={run:.0f} queue={q:.0f} cache_hit={hit*100:.1f}%")
PY
EOF
chmod +x /root/kv_usage.sh; echo "--- sample output ---"; bash /root/kv_usage.sh'

The script is a small but elegant piece of engineering. It uses Python's urllib.request to fetch the Prometheus metrics text, then parses lines with a helper function g(name) that matches lines starting with sglang:{name}{ and containing tp_rank=&#34;0&#34;. The tp_rank=&#34;0&#34; filter is a pragmatic choice: since all TP ranks share the same KV pool (they are shards of the same model), any one rank's gauge is representative. The script formats the output with thousands separators and percentage formatting, producing a single-line status report.

The sample output confirms the system is idle:

KV-cache: 0 / 2,581,504 tokens  ( 0.00% full) | running=0 queue=0 cache_hit=0.0%

Why This Message Was Written

The immediate trigger is the user's question about KV-cache monitoring. But the deeper motivation is the assistant's recognition that a production inference system without KV-cache observability is flying blind. The KV cache is the most constrained resource in a LLM serving stack—it determines maximum concurrency, maximum context length, and ultimately whether requests succeed or fail with out-of-memory errors. The assistant had just configured the decode server with --mem-fraction-static 0.85 and verified a KV capacity of 2.58M tokens, but that configuration was invisible to the operator. The user had no way to know, moment to moment, whether the cache was 10% full or 90% full.

The assistant's reasoning also reveals a sensitivity to the user's workflow. Rather than dumping raw Prometheus text or requiring the user to remember a complex curl command, the assistant builds a dedicated script and places it at /root/kv_usage.sh—a predictable, memorable location. The script is self-contained (it embeds the Python code via heredoc) and requires no external dependencies beyond the Python standard library. It can be run repeatedly, integrated into shell prompts, or wrapped in a watch command for continuous monitoring.

How Decisions Were Made

Several design decisions are embedded in this message, each reflecting a trade-off the assistant navigated:

Python over shell parsing. The Prometheus metrics format includes labels in curly braces after the metric name, e.g., sglang:kv_used_tokens{engine_type=&#34;decode&#34;,model_name=&#34;...&#34;,tp_rank=&#34;0&#34;} 0. Parsing this with grep and awk is fragile—label order can vary, values can contain special characters, and multi-line values are possible. Python's string methods provide more robust line-by-line parsing with substring matching. The assistant chose Python specifically for this reliability.

Filtering by tp_rank=&#34;0&#34;. The metrics are emitted once per TP rank (four ranks in this TP4 deployment), so a naive query would return four values for each metric. The assistant's g(name) function selects only the line containing tp_rank=&#34;0&#34;, deduplicating the output. This is correct because all TP ranks share the same KV cache pool—the kv_used_tokens gauge is identical across ranks. An alternative would be to average across ranks, but that would add complexity without information gain.

Embedding the script via heredoc. Rather than creating the file locally and copying it with scp, or writing it in multiple SSH commands, the assistant uses a single cat &gt; file &lt;&lt; &#34;EOF&#34; heredoc. This is atomic—if the SSH connection drops mid-stream, the file is either complete or absent. It also avoids temporary files on the local machine.

Offering the full stack as a follow-up. The assistant explicitly notes in its reasoning that Prometheus and Grafana "are not installed yet" and that setting them up "would be a bigger undertaking." By delivering the lightweight script first and positioning the full stack as optional, the assistant respects the user's time and lets them decide how much observability infrastructure they want.

Assumptions and Mistakes

The message rests on several assumptions, some explicit and some implicit:

The metrics endpoint is stable. The script hardcodes http://127.0.0.1:30002/metrics as the source. This assumes the decode server's metrics endpoint will remain on that port and path, which is reasonable given it's configured via systemd. However, if the server is restarted with different arguments or if the port changes, the script breaks silently.

tp_rank=&#34;0&#34; is always representative. This is true for the current TP4 configuration, but if the model were deployed with a different parallelism strategy (e.g., EP-only without TP), the KV cache might not be identical across ranks. The assumption is valid for this deployment but not universal.

The earlier mistake—the too-strict grep pattern in message 12730—was corrected before this message. In that earlier attempt, the assistant used grep -E &#34;^sglang:full_token_usage &#34; (with a trailing space), which failed to match because Prometheus metrics have {labels} immediately after the name. The assistant diagnosed this correctly in message 12731: "Prometheus format includes labels in curly braces after the metric name, so it's sglang:full_token_usage{model=..., tp_rank=...} 0.0, not just sglang:full_token_usage with a space." The Python script in message 12732 avoids this entirely by using startswith(f&#34;sglang:{name}{{&#34;), which correctly handles the label prefix.

The assumption that metrics would be populated immediately after server restart was also corrected. In message 12731, the assistant realized "the metrics might only appear after the first request comes through, since the scheduler reports them on decode batches." It sent a warmup request before querying, and the metrics appeared. This is a subtle but important point: Prometheus gauges in SGLang are not initialized to zero at server start; they are only set when the scheduler first reports them during a decode batch.

Input Knowledge Required

To understand this message fully, a reader needs knowledge spanning several domains:

SGLang's metrics architecture. The assistant knows that KV-cache metrics are per-server gauges exposed on the decode server's /metrics endpoint, not on the router. This reflects an understanding of SGLang's internal architecture: the router handles request routing and load balancing, while the decode server owns the KV cache pool and scheduler.

Prometheus exposition format. The assistant correctly handles the Prometheus text format, where each metric line has the structure metric_name{label1=&#34;val1&#34;,label2=&#34;val2&#34;} value. The Python parser uses startswith to match the metric name and label-opening brace, then splits on the last space to extract the numeric value.

The deployment topology. The script queries 127.0.0.1:30002, which is the decode server's internal endpoint. The assistant knows this because it configured the deployment: prefill on :30000, decode on :30002, router on :30001. The script is designed to run on the host itself (via SSH), not remotely.

Python standard library. The script uses urllib.request (not requests, which would require an external package) and basic string parsing. This is a deliberate choice to minimize dependencies—the venv at /root/venv_sglang211/bin/python is guaranteed to have the standard library.

Output Knowledge Created

This message produces several lasting artifacts:

The script /root/kv_usage.sh. A reusable monitoring tool that any operator can run to check KV cache fullness. It transforms a raw Prometheus metrics dump into a single, readable line of output. The script is self-documenting (the heredoc includes a comment) and follows Unix conventions (exit code on failure, stdout for output).

Confirmation of the metrics pipeline. The message validates that the entire chain works: --enable-metrics flag → Prometheus gauge registration → HTTP /metrics endpoint → text exposition → parsing → human-readable output. This confirmation is valuable for debugging future issues—if KV monitoring ever breaks, the operator knows where to look.

Documentation of KV capacity. The output reveals the exact capacity (2,581,504 tokens) and current usage (0 tokens) at idle. This establishes a baseline for future monitoring. If the operator later sees kv_used_tokens at 2 million, they know the system is near capacity.

A pattern for extending observability. The g(name) helper function is generic—it can be reused to query any metric from the decode server, not just KV-cache ones. The operator could easily extend the script to report cache_hit_rate, num_running_reqs, or any other gauge.

The Thinking Process

The Agent Reasoning block in this message is unusually revealing. It shows the assistant weighing options in real time:

"The user wants a way to monitor KV fullness over time, and while the metrics are now exposed in Prometheus format on the decode server, setting up a full Prometheus + Grafana stack would be a bigger undertaking since neither is installed yet."

This is a cost-benefit analysis. The assistant recognizes that the user's question has two possible answers: a quick one (here's how to query the raw metrics) and a thorough one (here's a full monitoring stack). The assistant chooses to deliver both, but in sequence—quick first, thorough as a follow-up offer.

"I should offer both options: querying the metrics directly for immediate visibility, or helping set up Prometheus and Grafana for a proper monitoring dashboard."

The phrase "I should offer both options" reveals a user-centric design philosophy. The assistant is not making a unilateral decision; it's presenting a menu and letting the user choose. This is particularly appropriate given that the user explicitly mentioned Prometheus and Grafana—the assistant respects that preference while also providing an immediate alternative.

"I should give them a practical way to watch KV fullness in real-time with a simple command that polls the metrics endpoint, then ask if they want me to set up the full Prometheus and Grafana stack for persistent monitoring and dashboards."

The word "practical" is key. The assistant is optimizing for time-to-value. A full Prometheus + Grafana setup might take 30 minutes; the script takes 30 seconds to create and deploy. By delivering the script first, the assistant ensures the user can monitor KV fullness now, while the option of a full dashboard remains open.

Broader Significance

This message, while small in scope, illustrates a pattern that recurs throughout infrastructure engineering: the gap between "it works" and "we can see it working." The assistant had already deployed a highly optimized inference system with custom kernels, PD disaggregation, and systemd orchestration. But without observability, the system was opaque. The user's question exposed that opacity, and the assistant's response closed the gap with a lightweight, practical tool.

The message also demonstrates a mature approach to tool-building. The script is not over-engineered—it doesn't use a configuration file, doesn't support multiple output formats, doesn't handle every edge case. It solves one problem (how full is the KV cache?) and solves it well. The assistant correctly judges that a simple script is better than no script, and that a full monitoring stack can wait.

In the broader arc of the conversation, this message marks a transition from optimization to operations. The kernel work was done, the deployment was stable, and now the focus shifts to keeping it running. The KV-cache script is the first piece of that operational tooling—a small but essential window into the system's internal state.