The Edit That Fixed a Phantom: Diagnosing Prefix Cache Poisoning in SGLang Benchmarking

On its surface, message [msg 12162] is almost invisible — a single line confirming that a Python file was edited:

[assistant] [edit] /home/theuser/glm-kimi-sm120-rtx6000bw/kdtree-engine/python/bench_context_decode.py Edit applied successfully.

Barely a sentence. Yet this message is the culmination of a sophisticated diagnostic chain that reveals how deeply an AI assistant must understand the runtime behavior of a complex inference system to produce trustworthy measurements. The edit itself is trivial — a few lines added to a benchmark script. The reasoning that necessitated that edit is the real story.

The Benchmark That Broke

The assistant had just completed a major deployment milestone: extending the SGLang inference server running Kimi K2.6 on CT200 to support 200k-token context lengths ([msg 12153]). The user's next directive was simple: "Benchmark long context perf" ([msg 12155]). The goal was to characterize how prefill throughput, time-to-first-token (TTFT), and decode throughput scale as context grows from 1k to 192k tokens — critical data for understanding whether the deployment was practically usable at extreme lengths.

The assistant reached for an existing tool: bench_context_decode.py, a script that uses the classic "two-point method" to isolate decode throughput. The idea is elegant: send two requests with the same prompt but different max_tokens values (say, 1 and 17). The first request measures prefill + 1 decode step; the second measures prefill + 17 decode steps. Subtract the times, divide by the token difference, and you get pure decode throughput, with prefill overhead canceled out.

The assistant extended this script to also report TTFT and prefill tok/s ([msg 12158]), deployed it to CT200 ([msg 12159]), and ran the first sweep at context lengths 1024, 8192, and 32768 ([msg 12160]). The results were baffling:

 ctx_req prompt_tok   TTFT_s prefill_tok/s decode_tok/s s/out_tok     t_a     t_b
    1024       1407     0.46           nan          nan       nan    0.46    0.20
    8192      11382     3.86           nan          nan       nan    3.86    1.57
   32768      45507    23.72           nan          nan       nan   23.72    4.26

Every decode measurement was nan — not a number. And the raw times told the story: t_b (the supposedly longer request) was consistently smaller than t_a (the supposedly shorter one). At 1024 tokens, t_a was 0.46s but t_b was only 0.20s. At 32768 tokens, t_a was 23.72s but t_b was 4.26s. The second request was finishing in a fraction of the time.

The Diagnostic Chain

In message [msg 12161], the assistant works through the problem with remarkable clarity. The root cause is SGLang's radix/prefix cache: when the second request arrives with an identical prompt, the server recognizes the prefix from the first request's KV cache and skips the entire prefill phase. The second request essentially starts decoding from a fully cached state, making it appear that decode throughput is infinite (or negative, in the subtraction).

The assistant systematically evaluates three solutions:

  1. Disable prefix caching entirely — rejected because it would require restarting the server with --disable-radix-cache, changing the very configuration that was just tuned for 200k context.
  2. Switch to streaming — appealing because a single streaming request yields both TTFT and decode rate from one prefill, halving the prefill cost (important at 180k tokens where prefill takes minutes). But streaming introduces granularity issues with speculative decoding: DDTree may emit multiple tokens per step, making inter-token timing ambiguous.
  3. Flush the cache between calls — SGLang exposes a /flush_cache endpoint. The assistant can call req(a) → flush → req(b), forcing both requests to pay full prefill cost. This preserves the two-point method's accuracy at the cost of doubling prefill time. The assistant initially leans toward streaming, then circles back to the flush approach, reasoning that the double prefill cost is "steep for long sequences but worth it for accuracy." This is a classic engineering tradeoff: measurement fidelity versus measurement cost. The assistant also adds a per-context UUID nonce to the prompt to prevent any accidental cache hits from earlier runs.

What the Edit Actually Does

The edit applied in message [msg 12162] modifies bench_context_decode.py to:

  1. Call /flush_cache between the two requests for each context length, ensuring both incur the full prefill cost.
  2. Prepend a random UUID nonce to each prompt, preventing SGLang's radix cache from matching any prefix across different benchmark runs.
  3. Recompute prefill throughput and decode rate from the two measurements using the corrected timings. These are small changes — perhaps 10–15 lines of Python — but they transform the script from producing nan to producing meaningful measurements. The assistant verifies the fix in the next message ([msg 12163]) by testing the /flush_cache endpoint directly, confirming it responds with "Cache flushed."

The Broader Lesson

This episode reveals something important about benchmarking in complex AI systems: the measurement instrument itself must be understood as part of the system under test. The two-point method is a textbook technique, but it assumes each request is independent. SGLang's radix cache violates that assumption silently — not with an error, but by making the second request faster. The result is not a crash or a warning, but plausible-looking numbers that happen to be nan.

The assistant's thinking process in [msg 12161] shows a pattern that recurs throughout the session: form a hypothesis, test it, observe unexpected results, and reason about the discrepancy. The initial benchmark run produced nan — a signal that something was wrong. Rather than dismissing it or re-running, the assistant examined the raw timing data (t_a vs t_b), recognized the pattern of t_b < t_a, and connected it to the server's caching behavior. This is the same diagnostic discipline visible in the earlier kernel optimization work ([chunk seg=0 chunk=1]), where the assistant used profiling data to disprove its own assumptions about CPU orchestration overhead.

The edit in message [msg 12162] is a small fix, but it represents a larger commitment: measurements must be trustworthy, and trustworthiness requires understanding the system deeply enough to anticipate how it might distort the measurement. In the next message, the assistant deploys the corrected script and begins the real benchmark sweep — now armed with a methodology that accounts for the infrastructure it runs on.