The 80 tok/s Milestone: Validating Native INT4 Kimi-K2.5 on Blackwell

In the high-stakes world of deploying 1-trillion-parameter language models on 8x Blackwell RTX PRO 6000 GPUs, a single message can represent the culmination of hours of infrastructure wrestling, model pivots, and hardware-boundary exploration. Message <msg id=2366> is exactly such a moment. It is the quiet, data-driven confirmation that the moonshot has landed — the native INT4 variant of Kimi-K2.5 is not just running, but running at 80.1 tok/s, nearly double the user's stated target of 40–50 tok/s. But this message is far more than a victory lap. It reveals the assistant's methodical reasoning, exposes a subtle interpretive tension about what "single stream" throughput means in the context of reasoning models, and sets the stage for the NCCL tuning experiments that follow.

The Path to This Moment

To understand why <msg id=2366> was written, we must trace the journey that led to it. The session had been a whirlwind tour of model deployment on PCIe-bound Blackwell hardware. Earlier segments had seen the team pivot from GLM-5 NVFP4 (which suffered from incoherent output due to tensor parallelism sharding bugs) to MiniMax-M2.5 FP8 (which achieved an impressive ~4,000 tok/s at high concurrency but used only 4 GPUs with Expert Parallelism), and finally to the native INT4 Kimi-K2.5 from Moonshot AI.

The user's directive in <msg id=2359> was clear: "Run benchmarks and try to get to single stream >40~50; If not there try NCCL LL alg and other safe-ish tricks." The assistant had just completed the grueling 36-minute load of the 547GB INT4 model across 8 GPUs, and the baseline benchmark in <msg id=2364> had already returned a promising 81.4 tok/s at concurrency 1. This was the first signal that the INT4 variant might dramatically outperform its NVFP4 predecessor (which had topped out at ~61 tok/s).

What the Message Actually Says

The message opens with a triumphant arithmetic statement: "512 tokens in 6.39s = 80.1 tok/s. That's already well above the 40-50 target." The assistant is cross-validating the benchmark result from <msg id=2364> with a real curl-based measurement of a single request with 512 max_tokens. The 6.39-second wall time for a full 512-token generation confirms the benchmark's finding.

But then the assistant immediately introduces a note of caution: "But this includes reasoning tokens." This is the key insight that drives the rest of the message. The Kimi-K2.5 model, like many modern reasoning models, produces both a reasoning trace (invisible chain-of-thought) and the final visible content. The 512 completion tokens are split between these two phases, and the assistant wants to understand the breakdown.

The assistant dispatches a second curl request, this time parsing the response to extract reasoning and content character counts. The result: ~1,652 characters of reasoning and ~634 characters of content for the 512 total completion tokens. This reveals that roughly half of the generation budget goes to reasoning — a critical detail for understanding what "single stream throughput" actually means for an end user.

The Assumptions and Their Implications

The message reveals several important assumptions. First, the assistant assumes that the user's "40-50 tok/s" target refers to the raw decode rate — tokens per second as measured by the model's completion token count divided by wall time. By this metric, 80 tok/s is a resounding success. However, there's an unspoken ambiguity: does the user care about the rate at which visible tokens appear, or the total generation rate including reasoning? If the user is thinking about perceived latency — the time until the first visible token appears, or the rate at which content tokens stream back — then the effective rate is closer to 40 tok/s (since only ~50% of tokens are content). The assistant does not resolve this ambiguity in this message, but the very act of checking the reasoning/content split shows an awareness that the answer matters.

Second, the assistant assumes that the NCCL tuning mentioned by the user ("try NCCL LL alg and other safe-ish tricks") is a contingency for a failure to meet the target, not an optimization to pursue regardless. Since the target is already exceeded, the implicit reasoning is that NCCL tuning may not be necessary. This assumption is tested in the very next message (<msg id=2367>), where the assistant nonetheless proceeds with NCCL tuning — perhaps out of thoroughness, or because the user explicitly asked for it.

Third, the assistant assumes that the curl-based measurement is a fair representation of single-stream throughput. The prompt used ("Write a detailed step-by-step guide on how to build a basic web application using Python and Flask.") is reasonably representative, but it's worth noting that different prompts produce different reasoning-to-content ratios, which would affect the perceived throughput.

Input Knowledge Required

To fully understand this message, the reader needs to know:

  1. The hardware context: 8x NVIDIA RTX PRO 6000 Blackwell GPUs (96GB each) connected via PCIe, with SM120 architecture that requires special Triton MLA attention backends. The PCIe interconnect is the primary bottleneck for allreduce operations in tensor-parallel inference.
  2. The model architecture: Kimi-K2.5 uses Multi-Head Latent Attention (MLA), the same architecture as DeepSeek V3. This means every transformer layer requires an allreduce of the attention output, which is particularly expensive across 8 PCIe-connected GPUs. The INT4 quantization (via compressed-tensors, group_size=32, symmetric) only applies to the MoE routed expert weights — attention layers remain in full precision.
  3. The vLLM deployment stack: The model is served using vLLM's OpenAI-compatible API server with tensor-parallel-size=8, running on a custom Python environment with Triton MLA sparse attention backends patched for SM120 support.
  4. The benchmark methodology: The assistant uses a custom benchmark script that measures throughput at various concurrency levels (C=1 through C=1024), and supplements it with direct curl measurements for single-stream latency analysis.
  5. The prior results: The NVFP4 Kimi-K2.5 achieved ~61 tok/s single-stream, and the MiniMax-M2.5 FP8 achieved ~84 tok/s single-stream but with only 230B active parameters on 4 GPUs.

Output Knowledge Created

This message produces several critical pieces of knowledge:

  1. INT4 Kimi-K2.5 single-stream throughput is ~80 tok/s, far exceeding the NVFP4 variant's ~61 tok/s. The improvement is attributed to the lower weight bandwidth requirements of INT4 quantization for the MoE experts.
  2. The reasoning-to-content split is approximately 50:50 for a typical 512-token generation. This means that while the raw decode rate is 80 tok/s, the visible content rate is closer to 40 tok/s. This is crucial information for anyone designing user-facing applications where perceived latency matters.
  3. The model is coherent and functioning correctly — the content preview ("Here is a comprehensive step-by-step guide to building a Task Manager web application...") shows that the model is producing sensible output with proper formatting.
  4. The 40-50 tok/s target is already met and exceeded by the raw metric, which means the NCCL tuning contingency may not be needed (though it is pursued anyway in subsequent messages).

The Thinking Process Visible in Reasoning

The assistant's reasoning in this message follows a clear pattern: measure, question, refine. The first measurement (benchmark at 81.4 tok/s) is cross-validated with a direct curl measurement (80.1 tok/s). The agreement between these two methods builds confidence. But rather than stopping there, the assistant immediately questions whether the metric is meaningful: "But this includes reasoning tokens." This intellectual honesty — the willingness to complicate a good-news story — is a hallmark of rigorous engineering.

The second curl request is designed to decompose the 512 tokens into reasoning and content components. The assistant doesn't just accept the raw throughput number; it wants to understand what the number means for the user experience. The resulting data (~1,652 chars reasoning, ~634 chars content) is presented without strong commentary — the assistant lets the data speak for itself.

There's also a subtle narrative tension in the message. The assistant has just achieved a clear win (80 tok/s, target exceeded), but the user's instruction to "try NCCL LL alg and other safe-ish tricks" hangs in the background. The assistant must decide: declare victory and stop, or continue tuning as instructed? The message ends without resolving this tension — the data is presented, and the next message (<msg id=2367>) shows the assistant choosing to proceed with NCCL tuning anyway. This suggests a reasoning process that goes: "We've exceeded the target, but the user explicitly asked for tuning, and there might be further gains. Let me investigate the bottleneck first."

Mistakes and Incorrect Assumptions

The message contains one subtle but important potential misinterpretation. The assistant treats "single stream >40~50" as a throughput target measured in raw tok/s. But the user might have meant "40-50 tok/s of visible content" — i.e., the rate at which the end user sees tokens appearing. If we interpret the target this way, the effective visible throughput is ~40 tok/s (since half the tokens are reasoning), which is right at the lower bound of the target range, not comfortably above it. The assistant does not flag this ambiguity explicitly, though the reasoning/content breakdown data provides the raw material for the reader to make their own judgment.

Additionally, the assistant assumes that the NCCL tuning is a fallback for failure rather than a routine optimization. This leads to a slight hesitation in the workflow — the assistant has to mentally re-orient from "we succeeded, stop" to "we succeeded, but let's optimize anyway." This is visible in the slightly awkward transition between <msg id=2366> and <msg id=2367>.

Conclusion

Message <msg id=2366> is a masterclass in data-driven engineering communication. It reports a clear success (80 tok/s, target exceeded), immediately complicates that success with a nuance about reasoning tokens, provides the data needed to resolve the nuance, and leaves the door open for further optimization. It is the kind of message that an experienced engineer writes when they want their stakeholder to have complete information — not just the headline number, but the context needed to interpret it correctly. In the broader arc of the session, this message marks the moment when the INT4 Kimi-K2.5 deployment transitioned from "will it work?" to "how fast is it?" — and the answer turned out to be very fast indeed.