The First Data Point: Running the Macro Benchmark on Kimi-K2.5 INT4

In any optimization campaign, there comes a moment when speculation gives way to measurement. Message [msg 2420] captures precisely that transition: the first concrete performance data from the Kimi-K2.5 INT4 model running on an 8x RTX PRO 6000 Blackwell system. The message is deceptively simple — a bash command executing a Python benchmark script, followed by its output — but it represents the culmination of careful planning, the resolution of a dependency issue, and the beginning of a data-driven optimization journey that would ultimately reshape the team's understanding of where their bottlenecks truly lay.

The Road to Measurement

To understand the significance of this message, one must appreciate the context that led to it. The team had spent days deploying and debugging large language models on a novel hardware configuration: eight NVIDIA RTX PRO 6000 Blackwell GPUs (SM120 architecture) connected only via PCIe, without NVLink. They had successfully deployed the Kimi-K2.5 INT4 model — a 1-trillion-parameter Mixture-of-Experts architecture quantized to 4-bit integers — using vLLM with tensor parallelism across all eight GPUs. The model was serving requests, but the question of how well it was performing remained unanswered.

In the preceding messages ([msg 2404] through [msg 2407]), the assistant had laid out a comprehensive three-phase profiling plan. Phase 1 would measure macro-level throughput and latency against the running vLLM server. Phase 2 would micro-benchmark individual GEMM operations at the exact dimensions Kimi-K2.5 uses. Phase 3 would capture a full torch.profiler trace to see inside the CUDAGraph replays. The user's instruction was unambiguous: "proceed with all benchmarks, write down results into k25b6000bench1.md as you gather them" ([msg 2408]).

The assistant began executing. It verified the vLLM service was running and healthy ([msg 2410], [msg 2411]). It discovered the HTTP profiler API was unavailable — the service had been started without --profiler-config ([msg 2412]). It created the results document ([msg 2414]) and wrote two benchmark scripts: bench_k25_macro.py for macro-level HTTP benchmarks and bench_k25_micro.py for GPU-level micro-benchmarks ([msg 2415], [msg 2416]). Then came the first attempt to run the macro benchmark, which failed with a ModuleNotFoundError: No module named 'aiohttp' ([msg 2417]). The assistant resolved this by installing the missing package ([msg 2418], [msg 2419]).

What the Message Contains

Message [msg 2420] is the successful execution of the macro benchmark script after that dependency was resolved. The output shows:

Target: http://10.1.230.174:8000/v1/chat/completions
Model: /shared/kimi-k2.5-int4
Time: 2026-02-21 12:33:31 UTC

======================================================================
SINGLE-STREAM LATENCY BENCHMARK
======================================================================
Warming up...
  Warmup done: 0.12s

Testing max_tokens=32...
  Trial 1: 0.407s, 32 tokens, TPOT=12.7ms, tok/s=78.7
  Trial 2: 0.414s, 32 tokens, TPOT=12.9ms, tok/s=77.3
  Trial 3: 0.398s, 32 tokens, TPOT=12.4ms,...

The output is truncated in the conversation record — the benchmark continued with additional trials and concurrency levels. The subsequent message ([msg 2421]) reveals the full picture: "Throughput plateaus at ~1536 tok/s at C=128 and doesn't improve beyond that."

These numbers are significant. A single-stream TPOT (time per output token) of ~12.7 milliseconds translates to roughly 78 tokens per second for a single user. At higher concurrency, the system achieves up to 1,536 tokens per second before plateauing. This is the first quantitative baseline for the Kimi-K2.5 INT4 deployment.

The Reasoning and Decisions Embedded in This Message

The message reveals several implicit decisions. First, the choice to benchmark via the HTTP API rather than using vLLM's Python client or lower-level interfaces reflects a pragmatic trade-off: the HTTP API is the production interface, so measuring it captures real-world performance including serialization, network overhead, and the vLLM scheduler. It also requires no changes to the running service — zero downtime.

Second, the benchmark script itself embodies assumptions about what matters. It measures single-stream latency (TPOT) and multi-concurrency throughput, but not prefill latency, time-to-first-token, or memory bandwidth utilization. These are deliberate choices: the team already knew from the earlier GLM-5 work that decode throughput was the primary bottleneck, not prefill.

Third, the warmup of 0.12 seconds is brief. This assumes the model and CUDAGraphs are already compiled and cached from previous requests — a reasonable assumption given that the service had been running for 24 minutes and the warmup request itself would trigger any lazy compilation.

Assumptions and Potential Pitfalls

Several assumptions underpin this measurement. The most critical is that the benchmark script correctly isolates decode performance from prefill. The max_tokens=32 setting means each request generates 32 tokens, but the first token includes prefill time. The script appears to measure total generation time and divide by token count, which would slightly overstate TPOT if prefill is included. For 32 tokens, the prefill contribution is small but not zero.

Another assumption is that the benchmark client running on a separate machine (the assistant's environment, not the GPU server) adds negligible latency. The network path between the benchmark client and the vLLM server involves a SSH tunnel or direct connection to 10.1.230.174:8000. Network jitter could introduce variability, though the three trials showing consistent results (12.7, 12.9, 12.4 ms) suggest this is minimal.

The script also assumes that the vLLM server has sufficient request capacity to handle concurrent requests without queueing artifacts. At higher concurrency levels (C=128), the plateau at ~1536 tok/s could indicate either GPU saturation or a software bottleneck in the vLLM scheduler or tokenizer — the macro benchmark alone cannot distinguish these.

Input Knowledge Required

To fully understand this message, one needs knowledge of: the Kimi-K2.5 model architecture (a 1T-parameter MoE with 61 layers, MLA attention, and INT4 quantization using Marlin kernels); the hardware configuration (8x RTX PRO 6000 Blackwell GPUs, PCIe-only interconnect, no NVLink); the vLLM serving stack with its HTTP API and CUDAGraph optimization; and the earlier GLM-5 profiling results that showed dtype-cast as the dominant bottleneck in the NVFP4 path. The benchmark script itself uses aiohttp for async HTTP requests, which explains the earlier dependency resolution.

Output Knowledge Created

This message creates the first quantitative baseline for Kimi-K2.5 INT4 performance on this hardware. The single-stream TPOT of ~12.7ms and peak throughput of ~1,536 tok/s become reference points against which all subsequent optimizations are measured. These numbers also serve as input to the micro-benchmarking phase: if the macro numbers suggest the model is slower than expected from GPU compute alone, the micro-benchmarks can isolate whether the bottleneck is in the GEMM kernels, the allreduce communication, or the software stack.

The message also implicitly validates that the Marlin W4A16 kernels are working correctly. The earlier GLM-5 NVFP4 deployment suffered from massive dtype-cast overhead (69% of decode time) because FlashInfer's CUTLASS kernels didn't fuse dequantization into the GEMM. The fact that Kimi-K2.5 INT4 achieves 78 tok/s single-stream without obvious dtype-cast issues suggests Marlin is doing its job — though the full torch.profiler capture would later confirm this.

The Thinking Process Visible in the Reasoning

The assistant's reasoning is visible in the sequence of actions leading to this message. It did not blindly run a benchmark; it first verified the service was healthy, checked for profiler API availability, created the results document, wrote the benchmark scripts, and only then attempted execution. When the first attempt failed due to a missing dependency, it resolved the issue and retried. This systematic approach — verify, prepare, execute, iterate — reflects an understanding that benchmarking a production service requires care: one must ensure the service is in a known state, the measurements are meaningful, and failures are handled gracefully.

The choice to run the macro benchmark before the micro benchmarks also reveals strategic thinking. Macro benchmarks are non-invasive (they use the HTTP API, requiring no GPU access) and can run while the production service is live. Micro benchmarks require stopping the service to free GPU memory. By running macro benchmarks first, the assistant gathers immediate data without disrupting the running service, and only later stops it for deeper investigation. This minimizes downtime and allows the macro results to inform the micro-benchmark design.

The Broader Significance

This message is the first domino in a chain of discoveries. The macro benchmark establishes that the system achieves ~1,536 tok/s at peak concurrency. The subsequent micro-benchmarks and torch.profiler capture would reveal that AllReduce accounts for 51.5% of decode time — a surprising result that reframes the entire optimization problem. Without this initial baseline, that discovery would have no context. The 12.7ms TPOT measured here becomes the "before" picture against which all "after" optimizations are compared.

In the end, the team would pivot to speculative decoding as the most promising software-only optimization path, constrained by the hard physical limits of PCIe-only multi-GPU inference. But that conclusion rests on the foundation laid by this message: the first real data, measured carefully, from a running production system.