Benchmarking a 1T-Parameter MoE Model on Blackwell GPUs: The Moment of Truth

Introduction

In the long arc of deploying a 1-trillion-parameter mixture-of-experts model across eight RTX PRO 6000 Blackwell GPUs, there comes a moment when all the infrastructure work — the driver installations, the CUDA toolkit configurations, the patched quantization files, the systemd service definitions — must answer the only question that matters: does it actually work, and how fast is it? Message 2146 captures this exact moment. It is a brief but dense message in which the assistant verifies that the newly deployed nvidia/Kimi-K2.5-NVFP4 model produces coherent output, then immediately pivots to measuring its steady-state decode throughput. The message contains two curl commands, their timing output, and the assistant's analysis of what those numbers mean. On its surface, it is a simple benchmark report. But in the context of the broader session — which involved resolving FP8 KV cache incompatibilities, patching model configuration files, debugging launch failures, and waiting through a 523-second model load — this message represents the culmination of an enormous engineering effort.

The Message in Full

The assistant writes:

Working correctly. 107 tokens in 6.7s (~16 tok/s including reasoning overhead and first-request CUDAGraph warmup). Let me do another request to get steady-state speed:

``` Completion tokens: 512

real 0m8.509s ```

Then issues a bash command to run a second benchmark, which returns 512 completion tokens in 8.5 seconds.

Why This Message Was Written: The Reasoning and Motivation

This message sits at the boundary between two phases of deployment: verification and optimization. The assistant had just spent several hours (across multiple segments of the conversation) bringing the Kimi-K2.5-NVFP4 model online. The model had failed to start multiple times — first because the FP8 KV cache configuration was incompatible with the SM120 architecture of the RTX PRO 6000 GPUs, then because the default 262k context window exceeded available GPU memory, requiring a reduction to 128k. Each failure required diagnosis, a configuration patch, a process kill, a cleanup of shared memory objects, and a relaunch. By message 2146, the model had finally loaded successfully and served its first response — but that first response was truncated at 64 tokens because the assistant's test request had set max_tokens too low.

The motivation for message 2146 is twofold. First, the assistant needs to confirm that the model is genuinely working — not just loading without errors, but producing sensible output. The first test request ("What is the capital of France? Answer in one sentence.") returned a coherent reasoning trace and the correct answer ("The capital of France is Paris."), confirming functional correctness. Second, the assistant needs to establish a performance baseline. The entire deployment effort was aimed at making this model usable for inference, and without throughput numbers, there is no way to know whether the deployment is successful or whether further optimization is needed.

The message also reveals an important methodological choice: the assistant recognizes that the first request includes CUDAGraph compilation overhead (what it calls "first-request CUDAGraph warmup") and therefore does not represent steady-state performance. This is why it immediately issues a second, longer request — to measure the true inference speed after all compilation and optimization has settled.

How Decisions Were Made

The decision-making in this message is subtle but important. The assistant chooses specific benchmark parameters with care:

Prompt selection: "Count from 1 to 50." is a deliberately simple, deterministic task. It avoids reasoning overhead (the model is a reasoning model that generates chain-of-thought before its answer) and measures raw generation speed. By setting temperature=0.0, the assistant ensures greedy decoding, which is the most reproducible and commonly benchmarked configuration.
Token budget: Setting max_tokens=512 gives a long enough generation to amortize any startup overhead and get a stable throughput measurement. 512 tokens at ~60 tok/s takes about 8.5 seconds, which is long enough to be meaningful but short enough to be practical.
Measurement tool: Using the Unix time command wraps the entire curl request, including network latency to the remote server (10.1.230.174), JSON parsing, and output formatting. The assistant is measuring end-to-end latency as perceived by a client, not just model inference time.
Output parsing: The assistant pipes the curl response through a Python one-liner that extracts only the completion token count, minimizing the output displayed while preserving the key metric.

Assumptions Made by the Assistant

Several assumptions underpin this message:

Steady-state assumption: The assistant assumes that after one request, CUDAGraph warmup is complete and the second request represents steady-state performance. This is a reasonable assumption for vLLM's architecture, where the first request triggers torch.compile and CUDA graph capture, but subsequent requests reuse the compiled graphs.
Single-request measurement: The assistant measures single-request throughput, not batch throughput. This is appropriate for the initial benchmark — it establishes the best-case latency for an isolated user — but it does not reveal how the system behaves under concurrent load.
No interference from other processes: The assistant assumes that no other processes are competing for GPU memory or compute during the benchmark. This is likely true given that the previous pkill commands cleaned up any leftover vLLM processes.
The model is correctly configured: The assistant assumes that the patches applied to hf_quant_config.json and config.json (removing kv_cache_quant_algo and kv_cache_scheme) are sufficient and that the model is running with FP16 KV cache as intended. The absence of KV-cache-related errors in the log supports this assumption.
Token counting is accurate: The assistant trusts that vLLM's reported completion_tokens count (512) is correct. This is a standard metric in the OpenAI-compatible API and is generally reliable, but it counts tokens as the model's tokenizer defines them, not raw characters or words.

Mistakes or Incorrect Assumptions

The message is careful and contains no obvious mistakes. However, there are a few nuances worth examining:

The ~16 tok/s figure is misleading if taken out of context. The first request generated 107 tokens in 6.7 seconds, but the assistant correctly notes that this includes "reasoning overhead and first-request CUDAGraph warmup." The reasoning overhead is significant for this model — the Kimi-K2.5 architecture generates a chain-of-thought reasoning block before the content, and those reasoning tokens are included in the 107-token count. The actual content output was just a single sentence ("The capital of France is Paris."). The 16 tok/s figure is therefore not representative of the model's generation speed for content tokens.
The 512-token completion in 8.5 seconds yields ~60 tok/s, but this is still a single-request measurement. In production, the model would likely serve multiple concurrent requests, and throughput would be affected by batching efficiency, KV cache management, and request scheduling. The assistant does not claim this is production throughput — it is clearly labeled as a single-request benchmark.
The use of time command includes Python JSON parsing overhead. The curl output is piped through python3 -c "import sys,json;..." which adds a small but measurable latency (typically 50-200ms for Python startup). This is negligible compared to the 8.5-second total, but it means the true inference time is slightly less than measured.

Input Knowledge Required

To understand this message fully, a reader needs:

Knowledge of vLLM's architecture: Understanding what CUDAGraph warmup is, why the first request is slower, and how vLLM compiles and caches GPU kernels. Without this, the distinction between first-request and steady-state performance is meaningless.
Knowledge of reasoning models: Understanding that Kimi-K2.5 is a reasoning model that generates a chain-of-thought (the reasoning field in the API response) before producing its final answer. This explains why the first request's 107 tokens include both reasoning and content.
Knowledge of the deployment context: The earlier struggles with FP8 KV cache, the SM120 architecture limitations, the 523-second model load time, and the memory constraints that forced a 128k context window. The benchmark numbers are only meaningful against this backdrop.
Knowledge of GPU architecture: Understanding that eight RTX PRO 6000 Blackwell GPUs with 96GB each, connected via PCIe (not NVLink), have specific communication bottlenecks. The assistant's earlier discovery that NCCL allreduce dominates decode time (~65-70%) is essential context for interpreting throughput numbers.
Knowledge of the NVFP4 quantization format: Understanding that NVFP4 is NVIDIA's 4-bit floating-point quantization, which requires specific CUDA kernels (FLASHINFER_CUTLASS for GEMM operations) and that the FP8 KV cache format was incompatible with the SM120 architecture.

Output Knowledge Created

This message creates several pieces of actionable knowledge:

A verified functional deployment: The Kimi-K2.5-NVFP4 model is confirmed to produce coherent, correct output on the target hardware. This is non-trivial — the assistant had to patch the model's quantization configuration to disable FP8 KV cache, and the fact that the model still works correctly validates that approach.
A steady-state throughput baseline: ~60 tok/s for single-request decode on an 8-GPU setup with a 1T-parameter model. This is the reference point against which all future optimizations will be measured.
A warmup penalty measurement: The first request is significantly slower (~16 tok/s vs ~60 tok/s), confirming that CUDAGraph compilation adds meaningful overhead. This has operational implications — in a production deployment, one might want to send a warmup request before accepting user traffic, or use vLLM's built-in warmup features.
Confirmation that PCIe is the bottleneck: While not stated explicitly in this message, the ~60 tok/s figure, when combined with the assistant's earlier analysis that 65-70% of decode time is spent in NCCL allreduce, confirms that the primary throughput limiter is inter-GPU communication bandwidth. On a system with NVLink or NVSwitch, throughput would likely be significantly higher.

The Thinking Process Visible in the Message

The assistant's reasoning is compact but structured. The message opens with a summary judgment ("Working correctly") that synthesizes the results of the first test request. Then it presents the quantitative evidence (107 tokens, 6.7 seconds) with an immediate qualification — this includes warmup overhead and reasoning tokens. The assistant does not treat the first measurement as definitive; instead, it explicitly states the need for a second measurement to get "steady-state speed."

This reveals a disciplined experimental mindset. The assistant is not merely collecting numbers — it is interpreting them in context, identifying confounding factors (warmup, reasoning overhead), and designing a follow-up experiment to isolate the true metric. The choice of "Count from 1 to 50" as the prompt is itself a thinking artifact: it is a task that minimizes reasoning overhead (counting is deterministic and requires no chain-of-thought), uses greedy decoding (temperature=0.0), and generates a long enough sequence (512 tokens) to average out transient fluctuations.

The assistant also demonstrates an understanding of what the numbers mean for the broader deployment. The 8.5-second response time for 512 tokens is fast enough for interactive use — a user would perceive this as near-instantaneous for a single question. But the assistant knows this is just the beginning; later in the conversation, it will investigate the PCIe allreduce bottleneck and consider further optimizations.

Conclusion

Message 2146 is a small but pivotal moment in a long deployment saga. It is the point at which the assistant transitions from "making it work" to "making it fast." The benchmark numbers — 107 tokens in 6.7 seconds for the first request, 512 tokens in 8.5 seconds at steady state — are not just performance metrics. They are the payoff for hours of debugging, patching, and configuration work. They confirm that the architectural decisions (disabling FP8 KV cache, limiting context to 128k, using NCCL_PROTO=LL) were correct. They establish a baseline for future optimization. And they demonstrate a rigorous approach to measurement that distinguishes between transient warmup effects and true steady-state performance. In a single message, the assistant answers the two most important questions about any ML deployment: does it work, and how fast is it?