Measuring Throughput Under Fire: A 10-Second Snapshot of Kimi-K2.5 INT4 on 8× Blackwell GPUs

In the middle of a marathon synthetic data generation run, the assistant paused to take the pulse of the system. Message [msg 2891] is deceptively simple — a single bash command that measures token throughput by sampling vLLM's Prometheus metrics endpoint over a 10-second window. But this brief measurement sits at a critical inflection point in the session, where the assistant must validate that the inference pipeline is performing adequately before committing to a multi-hour data generation run. The result — 1598 tok/s — provides the first concrete performance baseline for the Kimi-K2.5 INT4 model running on 8× NVIDIA RTX PRO 6000 Blackwell GPUs under realistic load.

The Context: A Pipeline Hung on Inference Speed

To understand why this message matters, we must trace the thread back through the preceding messages. The session's goal was ambitious: train an EAGLE-3 speculative decoding draft model for Kimi-K2.5, a ~1-trillion-parameter Mixture-of-Experts model. The training pipeline required high-quality synthetic data — actual reasoning traces from the target model itself. The assistant had just launched a 25K-sample inference run ([msg 2876]) using the 01b_generate_synthetic.py script, feeding questions from the mlabonne/open-perfectblend dataset through the vLLM inference server at a concurrency of 200, with completions up to 8192 tokens.

Twenty minutes later, the assistant checked progress ([msg 2888]) and saw: "Progress: 150/25000 (0 errors), 0.9 req/s, avg completion: 648 tokens." The response was telling: "Hmm, 0.9 req/s is slow." The assistant recognized that request rate (req/s) is a misleading metric when completion lengths vary wildly — a single 8000-token response counts as one request but represents vastly more work than a 50-token response. What mattered was token throughput (tok/s), the true measure of system performance.

This realization drove message [msg 2891]. The assistant needed a clean, real-time measurement of how many tokens the 8-GPU system was actually generating per second under full saturation (200 concurrent requests, all 8 GPUs pegged at 100% utilization as confirmed in [msg 2878]).

The Measurement: A Clever Bash One-Liner

The command issued in [msg 2891] is a study in pragmatic engineering:

ssh root@10.1.230.174 'T1=$(curl -s http://localhost:8000/metrics 2>/dev/null | grep "vllm:generation_tokens_total{" | awk "{print \$2}"); sleep 10; T2=$(curl -s http://localhost:8000/metrics 2>/dev/null | grep "vllm:generation_tokens_total{" | awk "{print \$2}"); python3 -c "print(f\"Throughput: {(${T2} - ${T1}) / 10:.0f} tok/s\")"'

This is a three-step pipeline executed as a single remote SSH command:

  1. Sample T1: Fetch the /metrics endpoint from the local vLLM server, extract the vllm:generation_tokens_total counter value using grep and awk.
  2. Wait: Sleep for 10 seconds to allow tokens to accumulate.
  3. Sample T2: Fetch the counter again, then compute the difference divided by 10 seconds using Python's f-string formatting. The switch from bc (used in the previous attempt at [msg 2890], which failed with "bash: line 1: bc: command not found") to python3 -c is a small but telling adaptation. Rather than installing bc or restructuring the command, the assistant used a tool already known to be available in the environment — Python 3. This is a pattern seen throughout the session: when a tool fails, reach for what's already installed rather than introducing new dependencies.

Why This Measurement Matters

The 1598 tok/s figure is more than a number — it's a validation point for the entire pipeline. Here's why:

At the system level, 1598 tok/s on an 8-GPU PCIe-connected system running a 1T-parameter INT4 model is a meaningful data point. Earlier in the session ([msg 2869]), the assistant had analyzed the PCIe bottleneck extensively, noting that AllReduce communication over PCIe Gen5 was the primary throughput limiter for Mixture-of-Experts models like Kimi-K2.5. The 1598 tok/s measurement provides empirical grounding for those earlier theoretical estimates.

At the pipeline level, this measurement tells the assistant how long the 25K-sample generation will take. With an average completion of ~650 tokens per sample (from [msg 2888]), 25,000 samples would generate roughly 16.25 million tokens. At 1598 tok/s, that's approximately 2.8 hours of generation time — plus prompt processing overhead. This informs the user's planning and expectations.

At the architectural level, the measurement validates that the vLLM server configuration (EP8 expert parallelism, INT4 quantization, the custom systemd service) is performing as expected under realistic load. The 200 concurrent requests saturate all 8 GPUs (confirmed at 100% utilization in [msg 2878]), making this a genuine stress test.

Assumptions Embedded in the Measurement

Every measurement carries assumptions, and this one is no exception:

The Thinking Process Visible in the Message

The assistant's reasoning is visible in the progression from [msg 2888] to [msg 2891]. In [msg 2888], the assistant sees "0.9 req/s" and immediately recognizes it as potentially misleading — the system could be generating tokens at high throughput while completing few requests because each request is long. Rather than speculating, the assistant designs a direct measurement.

The first attempt ([msg 2890]) uses bc for arithmetic, which fails. The assistant doesn't panic or restructure the entire approach — it simply substitutes python3 -c for bc, keeping the same sampling strategy. This reveals a pragmatic, adaptive mindset: the measurement methodology is sound, only the arithmetic tool needed replacement.

The choice of a 10-second window is also telling. It's long enough to average out second-scale fluctuations but short enough to not materially delay the ongoing inference run. The assistant is balancing measurement accuracy against operational overhead — a common concern when instrumenting a live production system.

What This Message Creates

The primary output is knowledge: the Kimi-K2.5 INT4 model on 8× Blackwell GPUs achieves approximately 1598 tok/s under 200-way concurrent load with an average completion length of ~650 tokens. This is a concrete, actionable number that feeds into:

Conclusion

Message [msg 2891] is a moment of empirical grounding in a session filled with complex engineering decisions. After hours of debugging flash-attn builds, patching API incompatibilities, configuring NCCL tuning, and deploying systemd services, the assistant finally measures what the system actually delivers: 1598 tokens per second. It's not a flashy result — no breakthrough, no new technique — but it's the kind of measurement that separates real engineering from theoretical planning. The number becomes a new fact in the shared understanding between user and assistant, informing every subsequent decision about pipeline timing, data volume, and training strategy.