The NaN That Revealed a Flaw: Benchmarking Long-Context Decode in SGLang

 ctx_req prompt_tok   TTFT_s prefill_tok/s decode_tok/s s/out_tok     t_a     t_b
    1024       1407     0.46           nan          nan       nan    0.46    0.20
    8192      11382     3.86           nan          nan       nan    3.86    1.57
   32768      45507    23.72           nan          nan       nan   23.72    4.26

This table, produced by a single bash command in message 12160 of a long-running opencode session, appears at first glance to be a failed benchmark. Every cell in the prefill_tok/s, decode_tok/s, and s/out_tok columns reads nan—not a number. The t_b column (time for the second request) is consistently smaller than t_a (time for the first request), a physical impossibility under the assumptions of the measurement methodology. What looks like a bug is actually a profound discovery about the system under test: SGLang's radix-based prefix caching was silently invalidating the benchmark's core premise.

Why This Message Was Written

The message was written in direct response to the user's command at <msg id=12155>: "Benchmark long context perf." The user had just finished a major deployment milestone—extending the Kimi K2.6 model's context length from 32K to 200K tokens on a server running SGLang with speculative decoding via DDTree. The assistant had verified that a 60,010-token prompt could be processed and answered correctly, but the user wanted quantitative performance data: how does prefill throughput and decode latency scale as context grows?

The assistant's reasoning at <msg id=12156> shows careful planning. It identified an existing benchmark script (bench_context_decode.py) that used a "two-point method": send two requests with the same prompt but different max_tokens values (A and B), then compute decode throughput as (B - A) / (t_B - t_A). The subtraction cancels prefill time, isolating pure decode performance. The assistant extended this script to also report TTFT (time-to-first-token) and prefill tok/s, then deployed it to the remote server via scp at <msg id=12159> and ran the first chunk of the sweep—context lengths 1024, 8192, and 32768 tokens.

The message exists because the assistant needed to see whether its measurement methodology worked before committing to the full sweep up to 180K tokens, where each prefill takes minutes. It was a sanity-check step, and it paid off dramatically.

The Assumption That Broke Everything

The two-point method rests on a critical assumption: that both requests incur the same prefill cost. The assistant's reasoning at <msg id=12158> shows it was aware of potential pitfalls: "For the two-point measurement with a=1 and b=17, I can isolate prefill time by subtracting the single decode step from the first measurement." But it missed the most important confound: SGLang's radix attention prefix cache.

The assumption was that sending two identical prompts would cause the server to compute the full attention from scratch both times. In reality, SGLang's prefix caching engine recognized the repeated prompt on the second request and reused the KV cache computed during the first request. The second request thus skipped the prefill phase entirely, completing in just the decode time for 16 tokens (B - A = 16). Since t_b (0.20s at 1K context) was less than t_a (0.46s), the denominator t_b - t_a became negative, producing NaN in the derived metrics.

This is a beautiful example of a measurement artifact that reveals a genuine system property. The NaN values are not noise—they are SGLang's prefix caching announcing its presence. The assistant's methodology was correct in theory but failed in practice because it didn't account for the very feature that makes SGLang performant for repeated prompts.

Input Knowledge Required

To understand this message, one needs to know several things:

  1. The two-point measurement method: A standard technique for isolating decode latency from prefill latency by sending two requests with the same prompt but different output token counts. The difference in wall-clock time divided by the difference in output tokens gives decode throughput.
  2. SGLang's radix attention prefix cache: SGLang uses a radix tree to cache KV tensors from previously computed prefixes. When a new request shares a prefix with a cached request, the server reuses the cached KV entries rather than recomputing them. This is a key performance optimization for serving workloads with shared prompt prefixes.
  3. The DDTree speculative decoding setup: The server is running with a custom DDTree drafter that uses tree-based speculative decoding to accelerate generation. The benchmark is measuring the combined performance of the base model plus the drafter.
  4. The hardware context: The server runs on RTX PRO 6000 Blackwell GPUs (sm_120 architecture) with 8 GPUs, tensor-parallelism of 8, and a 200K context-length configuration that required careful memory tuning.
  5. The prompt structure: The benchmark uses a synthetic repeated-sentence prompt ("The quick brown fox jumps over the lazy dog. " repeated thousands of times), which creates a highly predictable prefix that the cache can exploit.

Output Knowledge Created

Despite the NaN values, this message produced valuable knowledge:

  1. TTFT and prefill scaling data: The TTFT_s column shows time-to-first-token growing from 0.46s at 1K tokens to 23.72s at 32K tokens. The t_a column (which captures prefill + 1 decode step) shows the same trend. These numbers, while not the primary target, reveal that prefill time scales superlinearly with context length—consistent with the O(n²) complexity of full attention over the sequence.
  2. The existence and impact of prefix caching: The fact that t_b < t_a is itself a finding. It proves that SGLang's prefix cache is active and effective, reducing the second request's latency below the first. This is useful operational knowledge: repeated prompts to the same model will see dramatically lower latency.
  3. A methodological failure mode: The message documents a failure mode of the two-point method that future benchmarks must account for. Any system with prefix caching, KV caching, or request deduplication will invalidate the assumption that both requests pay the same prefill cost.
  4. The need for cache flushing: The assistant's subsequent reasoning at <msg id=12161> identifies the fix: flush the cache between the two requests using SGLang's /flush_cache endpoint, or switch to a single streaming request that measures TTFT and decode latency from inter-token intervals.

The Thinking Process Revealed

The assistant's reasoning before this message (visible in <msg id=12156> and <msg id=12158>) shows a methodical approach to benchmarking. It considered multiple options: streaming requests, cache flushing, unique nonces, and the two-point method. It chose the two-point method because it seemed clean and the existing script already implemented it. The reasoning reveals a tension between thoroughness and efficiency: the assistant wanted to minimize the number of long prefills (each taking minutes at 180K context) while still getting accurate decode measurements.

The critical oversight was not considering that SGLang's default behavior includes aggressive prefix caching. The assistant's mental model of the server treated each request as an independent computation, but SGLang is designed to avoid redundant computation. This is a common blind spot when moving from single-request testing to multi-request benchmarking: the system's optimizations for production serving interact with measurement methodology in unexpected ways.

Mistakes and Corrective Action

The mistake was not in the code but in the experimental design. The assistant assumed that two requests with identical prompts would both pay full prefill cost. The correction, implemented in the following messages, was twofold:

  1. Cache flushing: Insert a call to /flush_cache between the two requests, ensuring the second request re-computes the prefix from scratch. This preserves the two-point method's validity.
  2. Per-request nonces: Prepend a unique identifier to each prompt to prevent cross-request cache hits even without explicit flushing. The assistant's response to the NaN output was immediate and insightful. Rather than dismissing the results as a bug, it recognized that t_b < t_a was the signature of prefix cache reuse—a correct interpretation that led to a targeted fix. This diagnostic skill—reading the pattern of failure to infer the system's behavior—is the hallmark of experienced systems engineering.

Broader Significance

This message is a case study in the challenges of benchmarking modern ML serving systems. These systems are not simple request-response processors; they are stateful, caching, optimizing engines that actively work to minimize redundant computation. A benchmark methodology that works on a stateless server fails on a stateful one. The NaN values in message 12160 are not a failed measurement—they are a successful measurement of the wrong thing, which is often more informative than a successful measurement of the obvious thing.

The message also illustrates the importance of incremental benchmarking. Rather than launching a full 6-point sweep that would have taken 20+ minutes and produced systematically wrong results, the assistant ran a small 3-point sanity check first. The failure was caught early, the methodology was corrected, and the subsequent runs (visible in later messages) produced clean, interpretable data. This is a best practice that every ML engineer should internalize: benchmark iteratively, verify your methodology on small data before scaling up, and treat unexpected results as discoveries rather than failures.