The Prefill Ramp: When Benchmarking Numbers Lie

"Are you sure we're measuring thinking tokens correctly? Why is early output slower vs later?"

This single question, posed by the user at message index 12529 in the middle of an intensive DeepSeek-V4-Flash benchmarking session, cuts to the heart of a problem every ML engineer eventually faces: the moment when benchmark numbers look wrong, and you have to decide whether the measurement is broken or your understanding is.

The question appears deceptively simple. The assistant had just completed a concurrency scaling sweep on an 8× RTX PRO 6000 Blackwell (sm_120) server running DeepSeek-V4-Flash with NVFP4 quantization. After recovering from an OOM crash at C=64 caused by insufficient CUDA graph batch size, the server was restarted with --cuda-graph-max-bs 64 and --mem-fraction-static 0.60, and the sweep completed. The user, watching the live polling output, saw something that triggered their suspicion: at C=64, the first throughput reading was 5.00 tok/s, and just 30 seconds later it had climbed to 32.85 tok/s — a 6.5× increase in half a minute.

The Thinking-Token Hypothesis

The user's first instinct was to question the measurement methodology. DeepSeek models are known for producing "thinking tokens" — internal reasoning tokens generated before the visible response. These tokens are produced by a separate process within the model (the "thinking" or "reasoning" phase) and are typically not counted as output tokens in standard benchmarks. If the benchmark were accidentally counting thinking tokens as output tokens, the throughput numbers would be misleading: early in the generation, when the model is producing its internal reasoning, the apparent "throughput" would be inflated by tokens that shouldn't be counted. Conversely, if thinking tokens were being counted inconsistently — sometimes included, sometimes excluded — the early-slow pattern could be an artifact of the model transitioning from a thinking-heavy phase to a generation-heavy phase.

This is a sophisticated concern. It reveals that the user understands the architecture of reasoning models like DeepSeek and knows that "token counting" is not a trivial matter when the model produces multiple categories of tokens with different generation dynamics. The question also reveals an assumption: that the benchmark tool (sglang.bench_serving) might not be handling the chat template and reasoning parser correctly, potentially leaking thinking tokens into the output count.

The Actual Answer: Prefill Starvation

The assistant's response — detailed across the following messages — reveals that the thinking-token hypothesis was incorrect, but for reasons that illuminate the deeper dynamics of LLM serving. The benchmark was configured with --dataset-name random, which generates synthetic prompts and completions without any chat template or reasoning structure. There were no thinking tokens to mis-count. The early-slow pattern was a prefill scheduling artifact.

When 96 prompts are submitted at concurrency 64, every single one of them must first be prefilled — the model must process the 256 input tokens for each request, computing KV cache entries before any decoding can begin. Prefill is a compute-intensive operation that serializes on the GPU. During this ramp-up phase, the decode phase is starved: the GPU is busy computing KV caches for new requests instead of generating output tokens for requests already in the decode loop. The first throughput reading of 5.00 tok/s captures this starvation period. Once all 64 requests have been prefilled and are simultaneously in the decode phase, throughput stabilizes at the true decode ceiling of 32.85 tok/s.

The median Time-To-First-Token (TTFT) of 13.9 seconds at C=64 confirms this diagnosis. That 14-second delay is almost entirely prefill queue time — the time each request spends waiting for the GPU to finish prefilling the requests ahead of it. After that queue clears, the steady-state decode throughput of ~33 tok/s represents the true serving capacity.

What the Question Unlocks

The user's question, even though it was based on an incorrect hypothesis, triggered one of the most productive investigative threads in the session. The assistant responded by:

Validating the measurement: Confirming that the benchmark correctly counts only completion tokens, with no thinking-token contamination, by virtue of the random dataset configuration and ignore_eos behavior.
Quantifying the prefill ramp: Extracting steady-state throughput from the server's #running-req logs to separate ramp-up from true decode performance, producing a corrected scaling table: | Concurrency | Steady gen tok/s | Step time | Marginal cost | |---|---|---|---| | C=1 | 12.2 | 82 ms | — | | C=16 | ~29 | ~550 ms | ~31 ms/req | | C=64 | 32.85 | ~1950 ms | ~29 ms/req | This revealed a clean linear model: step_time ≈ 52 + 30·N ms, with a hard asymptote of ~33 tok/s. Each concurrent request adds ~30 ms of pure GPU kernel time — a fixed serial cost that limits throughput regardless of how many requests are in flight.
Confirming the bottleneck is not communication: The user's follow-up observation that PCIe bandwidth was only ~200 MB/s at C=64 (negligible compared to PCIe5 x16's 64 GB/s capacity) confirmed the GPUs were not communication-bound. The NCCL all-reduce share was just 1–2% of GPU time.
Diagnosing the kernel-level bottleneck: The user's cufall observation — SMs pegged at near-100% active cycles but only 340W power draw on a 600W TDP card — provided the crucial clue. SMs are resident and active, but power is low, meaning they are stalled on memory latency or running SIMT fallback kernels rather than feeding the tensor or FMA pipes. This is the signature of CUDA-core fallback kernels: the model is ~30× off the memory-bandwidth roofline, indicating a fixable software bottleneck rather than a hardware limit.

The Deeper Significance

This message exemplifies a pattern that recurs throughout high-performance ML engineering: the most valuable questions are often the ones based on incorrect premises. The user's thinking-token hypothesis was wrong, but asking the question forced a rigorous examination of the benchmark methodology, separated transient ramp-up effects from steady-state performance, and led directly to a kernel-level profiling campaign that identified the true bottleneck.

The question also reveals the user's mental model. They are not passively consuming benchmark results — they are actively monitoring the live output, noticing anomalies, and forming hypotheses. The reference to "thinking tokens" shows familiarity with DeepSeek's architecture. The follow-up message about PCIe bandwidth shows they are already reasoning about the communication vs. compute tradeoff. This is an engineer who understands the system deeply enough to ask the right wrong question — the question that, once answered, unlocks the next level of understanding.

For the assistant, this question served as a forcing function. The assistant had already gathered good benchmark numbers, but the user's skepticism demanded a deeper analysis: not just "what are the numbers" but "why are the numbers what they are." The resulting investigation — the steady-state extraction, the linear model fitting, the PCIe bandwidth confirmation, the cufall power analysis, and the subsequent kernel profiling — transformed a simple benchmark report into a genuine diagnostic investigation that identified the root cause of the ~33 tok/s ceiling.

Input and Output Knowledge

To understand this message, a reader needs to know: the distinction between prefill and decode in LLM serving; the concept of thinking tokens in reasoning models like DeepSeek; how concurrency affects batch serving benchmarks; the role of CUDA graphs in reducing kernel launch overhead; and the relationship between GPU power draw, SM activity, and memory bandwidth utilization.

The message creates new knowledge by: establishing that the benchmark measurement is clean (no thinking-token contamination); quantifying the prefill ramp effect and its impact on aggregate throughput numbers; confirming the linear marginal-cost model for decode scaling; and providing the evidence chain (low PCIe bandwidth + high SM activity + low power draw) that points to CUDA-core fallback kernels as the bottleneck. This diagnosis directly informed the next phase of work: custom kernel development to replace the fallback kernels with tensor-core operations, which ultimately delivered a ~17× throughput improvement.

Conclusion

The user's four-word question — "Are you sure we're measuring thinking tokens correctly?" — is a masterclass in productive skepticism. It challenges the measurement without assuming the answer, it draws on domain knowledge about the model architecture, and it opens the door to a deeper investigation that transforms the team's understanding of their system. In the high-stakes world of ML inference optimization, where a 2× throughput improvement can save months of GPU time, the willingness to question benchmark methodology is not pedantry — it's the first step toward genuine breakthrough.