The Metrics That Tell the Truth: Inter-Token Latency Analysis in vLLM Performance Tuning

In the middle of an intense performance tuning session for a large language model deployment, a single message can serve as the fulcrum between blind experimentation and data-driven optimization. Message 124 of this opencode conversation is precisely such a moment. It is a short but analytically dense message where the assistant pivots from exhaustive NCCL parameter sweeps to a deeper, metrics-based investigation of where time is actually being spent during token generation. This message, quoted in full below, represents the transition from "what configuration is fastest" to "what is actually happening inside the engine."

All 1623 tokens fall in the 10-25ms bucket (0.01 < ITL ≤ 0.025). So inter-token latency is 10-25ms, consistent with 57.5 tok/s (= ~17.4ms per token). [bash] ssh -o StrictHostKeyChecking=no root@10.1.230.174 'curl -s http://localhost:8000/metrics | grep -i "ttft\|time_to_first"' # HELP vllm:time_to_first_token_seconds Histogram of time to first token in seconds. # TYPE vllm:time_to_first_token_seconds histogram vllm:time_to_first_token_seconds_bucket{engine="0",le="0.001",model_name="/shared/glm5-gguf/GLM-5-UD-Q4_K_XL.gguf"} 0.0 vllm:time_to_first_token_seconds_bucket{engine="0",le="0.005",model_name="/shared/glm5-gguf/GLM-5-UD-Q4_K_XL.gguf"} 0.0 vllm:time_to_first_token_seconds_bucket{engine="0",le="0.01",model_name="/shared/glm5-gguf/GLM-5-UD-Q4_K_XL.gguf"} 0.0 vllm:t...

The Context: A Tuning Journey Hits a Plateau

To understand why this message was written, one must appreciate the journey that led to it. The assistant had been systematically optimizing the inference throughput of a GLM-5 model (quantized to GGUF Q4_K_XL format, ~402 GB total) running on an 8-GPU RTX PRO 6000 Blackwell system with tensor parallelism. The hardware configuration is unusual: eight high-end GPUs connected over PCIe rather than NVLink, which makes NCCL communication topology a critical tuning parameter.

The preceding messages ([msg 99] through [msg 123]) document an exhaustive NCCL tuning campaign. The assistant tested the default protocol (~47.3 tok/s), then NCCL_PROTO=LL (~57.6 tok/s, a 22% improvement), then NCCL_PROTO=LL128 (~45.5 tok/s, worse than default), then various shared memory optimizations like NCCL_SHM_USE_CUDA_MEMCPY and NCCL_SHM_MEMCPY_ASYNC (~57.5 tok/s, no improvement over LL alone). The conclusion was clear: NCCL tuning had plateaued at approximately 57.5 tokens per second, and no further protocol tweaking would move the needle.

At this point, the assistant faced a fork in the road. It could continue trying obscure NCCL environment variables in the hope of squeezing out a few more percent, or it could step back and ask a more fundamental question: what is actually consuming the 17.4 milliseconds per token? Message 124 is the moment this question is asked — and answered — using the most reliable source of timing data available: vLLM's own Prometheus metrics endpoint.

Why the Metrics Endpoint Was the Right Tool

The assistant's decision to query the metrics endpoint was not arbitrary — it was a direct response to a failed experiment. In the immediately preceding message ([msg 122]), the assistant had attempted to measure Time to First Token (TTFT) using the streaming API. The result was comically wrong: "Decode tok/s: 5443564.8." This absurd number arose because vLLM's streaming implementation batches all generated tokens into a single response chunk for short generations, making it impossible to distinguish TTFT from total generation time using client-side timing alone.

The metrics endpoint, by contrast, provides server-side histogram data that is immune to client-side measurement artifacts. vLLM exports Prometheus-style metrics including vllm:inter_token_latency_seconds and vllm:time_to_first_token_seconds, which are populated by the engine itself as it processes each token. These histograms use cumulative buckets (le = "less than or equal to"), so the assistant can read off exactly how many tokens fell into each latency range.

This was a savvy methodological choice. Rather than continuing to bang on the NCCL tuning drum, the assistant recognized that it needed a more precise diagnostic instrument. The metrics endpoint is that instrument.

What the Data Revealed

The inter-token latency (ITL) histogram told a remarkably clean story. Of 1623 tokens sampled, every single one fell into the bucket 0.01 < ITL ≤ 0.025 seconds — that is, between 10 and 25 milliseconds per token. No tokens were faster than 10ms, and none were slower than 25ms. This is an unusually tight distribution, indicating that the engine's per-token execution time is highly consistent and free of outliers.

The assistant correctly computed the implied average: at 57.5 tok/s, each token takes approximately 17.4 milliseconds. This number is squarely in the middle of the 10-25ms bucket, which is exactly what one would expect if the distribution is centered around the mean. The fact that no tokens fell into the 0-10ms bucket means there is no "fast path" — every token requires the same full sequence of operations. And the fact that none exceeded 25ms means there are no scheduling hiccups, memory contention events, or NCCL stragglers causing tail latency.

The TTFT data, which the assistant began collecting in this message (and would interpret in [msg 125]), showed a similarly clean picture: most TTFT values fell in the 20-100ms range, averaging approximately 52ms. This is small relative to the total decode time for any generation of reasonable length, confirming that prefill is not the bottleneck.

Assumptions and Their Validity

The assistant made several implicit assumptions in this message that are worth examining. First, it assumed that the 1623 tokens sampled by the metrics endpoint are representative of steady-state behavior. This is a reasonable assumption given that the metrics accumulate over the lifetime of the server process and the assistant had already run multiple benchmark queries. However, it is worth noting that the metrics include tokens from the warmup phase and any other requests that may have been issued, so the distribution could be slightly contaminated by cold-start effects.

Second, the assistant assumed that the histogram bucket boundaries are accurate and that vLLM's internal timing is reliable. This is generally true — vLLM uses torch.cuda.Event or CUDA events for GPU-side timing — but it is worth noting that the 10-25ms bucket is relatively coarse. The true distribution could be, say, 16-18ms or 10-24ms, and the histogram would look identical. The assistant acknowledged this implicitly by computing the average from the throughput rate rather than relying on the bucket midpoints.

Third, the assistant assumed that inter-token latency is the right metric to optimize. This is correct for single-request decode throughput, but it is worth noting that in a production setting with concurrent requests, other metrics like time-to-first-token under load or batch efficiency might be more important. The assistant's focus on single-request throughput was appropriate given the stated goal of optimizing decode performance.

The Thinking Process Visible in the Message

Although the message is short, it reveals a sophisticated analytical process. The assistant is performing a kind of differential diagnosis: having ruled out NCCL communication as the primary bottleneck (all protocols give ~57.5 tok/s), it is now checking whether the bottleneck lies in the per-token compute path or in the scheduling/engine overhead between tokens.

The key insight is that 17.4ms per token across 78 layers yields approximately 223 microseconds per layer. This number, which the assistant would compute explicitly in the next message ([msg 125]), is the critical constraint. Each layer must complete its entire computation — attention (MLA with compressed KV), MoE expert routing (8 out of 256 experts activated), GGUF dequantization, and two allreduce operations — in about 223 microseconds. This is an extremely tight budget that leaves little room for overhead.

The assistant's decision to query TTFT metrics in the same message shows that it is thinking holistically about the generation pipeline. TTFT captures prefill latency (the initial forward pass over the prompt), while ITL captures decode latency (each subsequent token). By measuring both, the assistant can determine whether optimization effort should focus on prefill (which matters for interactive applications) or decode (which matters for throughput).

Input Knowledge Required

To fully understand this message, one needs several pieces of background knowledge. First, familiarity with vLLM's metrics system: the assistant knows that vLLM exposes Prometheus histograms at the /metrics endpoint, that these histograms use cumulative buckets, and that the bucket labels (le="0.01", le="0.025") represent upper bounds in seconds. This is not obvious from the raw curl output — the assistant had to know what it was looking for.

Second, understanding of the relationship between inter-token latency and tokens-per-second: 1 / 0.0174 ≈ 57.5 tok/s. The assistant performed this calculation implicitly, confirming that the metrics data is consistent with the benchmark results.

Third, knowledge of the model architecture: the assistant knows that GLM-5 uses 78 layers with Mixture-of-Experts (8 active experts out of 256) and Multi-head Latent Attention (MLA). This architectural knowledge is what allows it to later decompose the 223μs per layer into attention, MoE, dequantization, and communication components.

Output Knowledge Created

This message produced several pieces of actionable knowledge. First, it confirmed that the system is running at a stable 57.5 tok/s with no tail latency issues — all 1623 tokens fall within a narrow 10-25ms window. This is valuable because it rules out many classes of problems: NCCL timeouts, GPU memory thrashing, kernel launch overhead spikes, and scheduling jitter.

Second, it established a baseline for TTFT that the assistant would interpret in the next message: approximately 52ms average, mostly in the 20-100ms range. This is small relative to decode time for any generation longer than a few tokens, confirming that prefill optimization is not the priority.

Third, and most importantly, the message set the stage for a deeper investigation. With the knowledge that each token takes exactly 17.4ms and that this time is consumed by 78 layers × 223μs each, the assistant could now ask: what fraction of that 223μs is spent on compute (attention, MoE, dequantization) versus communication (allreduce)? This question would drive the next phase of the investigation, leading to an exploration of allreduce-RMS fusion and other kernel-level optimizations.

Conclusion

Message 124 is a masterclass in diagnostic pivoting. When the NCCL tuning space was exhausted, the assistant did not continue tweaking parameters in the hope of incremental gains. Instead, it reached for a more precise measurement tool — vLLM's built-in metrics — and used it to confirm that the system was running at its expected performance with no hidden pathologies. The clean histogram (all 1623 tokens in a single bucket) was both reassuring and constraining: it meant there was no low-hanging fruit in the scheduler or communication layers, and any further gains would require deeper, kernel-level changes. This message is the quiet moment of clarity before the storm of optimization that follows.