The Art of Realistic Benchmarking: Adjusting Concurrency for a 1T-Parameter Model

"Also adjust the concurrency levels — this is a 1T model, so very high concurrency levels like 1024 aren't realistic"

This single sentence, uttered as an aside in a much larger conversation about deploying trillion-parameter language models on eight Blackwell GPUs, encapsulates a crucial moment of hardware-aware reasoning. The message — message 2225 in a sprawling opencode session — is deceptively brief. On its surface, it is merely a comment accompanying a file edit. But beneath that surface lies a sophisticated chain of reasoning about the relationship between model architecture, hardware constraints, and meaningful benchmarking methodology.

The Immediate Context

To understand why this message was written, we must first understand what preceded it. The assistant had just finished a grueling multi-hour effort to deploy the NVFP4 variant of Kimi-K2.5, a 1-trillion-parameter Mixture-of-Experts model, on a machine equipped with eight NVIDIA RTX PRO 6000 Blackwell GPUs. After resolving a cascade of issues — a flashinfer-cubin version mismatch that caused import errors, a service crash during model loading, and a 13-minute startup cycle involving torch.compile and CUDAGraph warmup — the model was finally serving requests correctly. The assistant had verified coherent output (correctly identifying Paris as the capital of France) and confirmed the server's health endpoint was responding.

The natural next step was benchmarking. In message 2223, the assistant read the existing benchmark script at /home/theuser/glm-kimi-sm120-rtx6000bw/benchmark.py. This script had been written during the earlier GLM-5 deployment phase and contained stale references: the model path pointed to a GGUF file for GLM-5, the endpoint used was the legacy /v1/completions rather than the chat completions endpoint required by reasoning models, and the concurrency levels were configured for that earlier model.

In message 2224, the assistant made the first round of edits, updating the model path to point to the Kimi-K2.5 NVFP4 directory and switching to the chat completions endpoint. The assistant also noted that for a reasoning model, the original 128 max_tokens was too low because reasoning tokens consume from the same budget.

Then came message 2225 — the subject of this article.

The Reasoning: Why Concurrency Matters

The assistant's statement — "this is a 1T model, so very high concurrency levels like 1024 aren't realistic" — reveals a deep understanding of the operational realities of large-scale inference. The original benchmark script, written for GLM-5 (a different architecture with different memory characteristics), likely included concurrency levels reaching 1024 simultaneous requests. This might have been feasible for a smaller model, or perhaps it was aspirational. But for Kimi-K2.5 NVFP4, the assistant recognized that such levels were unrealistic.

The reasoning is multi-faceted. First, there is the raw parameter count: 1 trillion parameters, even in a MoE configuration where only a fraction (~30B) are active per token, imposes enormous memory pressure. The model weights alone consume approximately 70.8 GiB per GPU (as confirmed by the nvidia-smi output in message 2213, which showed 73.7 GiB used per GPU including overhead). This leaves only about 23 GiB per GPU for KV cache and other runtime buffers.

Second, Kimi-K2.5 uses Multi-Head Latent Attention (MLA), an architecture that, while more memory-efficient than standard multi-head attention for long contexts, still requires significant KV cache per request. With a maximum context length of 131,072 tokens (as reported by the /v1/models endpoint in message 2221), each concurrent request consumes a substantial chunk of KV cache memory. At 1024 concurrent requests, the KV cache alone would exceed the available GPU memory by orders of magnitude.

Third, the assistant was acutely aware of the PCIe allreduce bottleneck that had been identified in earlier analysis (documented in the segment summaries). With 8 GPUs connected via PCIe rather than NVLink, the communication overhead for tensor parallelism grows with batch size. At very high concurrency, the allreduce operations for the 61-layer MLA architecture would become the dominant cost, leading to diminishing returns or outright thrashing.

The Decision-Making Process

The assistant's decision to adjust concurrency levels was not made in isolation. It followed a deliberate, two-step process. First, in message 2224, the assistant addressed the most obvious issues: the model path and the endpoint. These were mechanical changes — the script literally referenced a different model and a different API surface. But the concurrency adjustment in message 2225 required judgment, not just mechanical substitution.

The assistant had to ask itself: what concurrency levels are meaningful for this model on this hardware? The answer depended on understanding the model's memory footprint, the GPU memory budget, the communication topology, and the expected throughput characteristics. The assistant implicitly performed a mental capacity analysis: with ~23 GiB of free memory per GPU after weights, and each concurrent request requiring KV cache for up to 131K tokens (even if the benchmark prompt is shorter), the practical concurrency ceiling is far below 1024.

The "Also" at the beginning of the message is telling. It signals that this is a continuation of the thought process from message 2224 — a second insight that occurred after the first round of edits. The assistant could have bundled both changes into a single edit, but it chose to separate them, suggesting that the concurrency adjustment was a distinct realization that emerged after addressing the surface-level issues.

Assumptions and Their Validity

The assistant made several assumptions in this message. It assumed that the original benchmark script indeed had concurrency levels reaching 1024 (a reasonable inference from the GLM-5 era). It assumed that such levels would be unrealistic for the 1T model — a judgment that subsequent events would partially validate. Later in the session, the NVFP4 Kimi-K2.5 achieved approximately 800 tok/s at C=32 (as shown in message 2227), and the INT4 variant reached 2,276 tok/s at high concurrency. Neither came close to saturating 1024 concurrent requests.

The assistant also assumed that the user wanted realistic, actionable benchmarks rather than theoretical maximum throughput numbers. This is a design philosophy choice: benchmarks that test unrealistically high concurrency produce numbers that look impressive on paper but have no relationship to actual deployment scenarios. By capping concurrency at realistic levels, the assistant ensured that the benchmark results would inform real operational decisions.

One potential limitation of this assumption is that very high concurrency benchmarks can reveal system behavior under extreme stress — memory fragmentation patterns, scheduler pathologies, or NCCL communication saturation. By excluding those levels, the assistant may have missed an opportunity to characterize the system's failure modes. However, given that the primary goal was to determine the model's practical throughput for production use, the conservative approach was justified.

Input and Output Knowledge

To fully understand this message, a reader needs several pieces of input knowledge. They need to know that Kimi-K2.5 is a 1-trillion-parameter MoE model with MLA attention. They need to understand the hardware configuration: 8x RTX PRO 6000 Blackwell GPUs with 96 GiB each, connected via PCIe rather than NVLink. They need to know that the benchmark script existed from a previous deployment phase and contained stale configuration. And they need to understand the relationship between concurrency, KV cache memory, and throughput in LLM serving.

The output knowledge created by this message is the updated benchmark script with appropriate concurrency levels. But more importantly, the message creates meta-knowledge: it establishes that the assistant is thinking critically about benchmarking methodology, not just mechanically executing commands. It signals to the user (and to anyone analyzing the session) that the assistant understands the physical constraints of the system and is making deliberate trade-offs.

The Thinking Process

The thinking process visible in this message is a model of pragmatic engineering judgment. The assistant is not following a script or executing predetermined steps. It is actively reasoning about the relationship between model characteristics and benchmarking methodology. The chain of thought proceeds: "We have a working deployment → we need to benchmark it → the existing benchmark script is stale → first, fix the model path and endpoint → second, realize that the concurrency levels are also wrong for this model → adjust them accordingly."

This two-step realization pattern is characteristic of expert troubleshooting. The first pass catches the obvious problems; the second pass catches the subtler ones that only become apparent after the first fixes are applied. The assistant could have stopped after message 2224, but it didn't — it continued thinking about the problem and identified a second, equally important issue.

The message also reveals the assistant's mental model of the system. By stating "this is a 1T model" as the justification for adjusting concurrency, the assistant implicitly references a whole constellation of constraints: memory pressure, communication overhead, KV cache scaling, and practical throughput limits. The single sentence encodes a rich understanding of distributed inference systems.

Conclusion

Message 2225 is a small but revealing moment in a much larger engineering effort. It demonstrates that effective system deployment requires not just technical skill but also judgment — the ability to distinguish between benchmarks that look good on paper and benchmarks that produce meaningful, actionable data. By adjusting the concurrency levels for the 1T-parameter model, the assistant ensured that the subsequent benchmarking effort (which produced the impressive results shown in messages 2226–2228) would reflect real operational conditions rather than theoretical extremes. In doing so, it turned a routine file edit into a statement about the importance of hardware-aware benchmarking methodology.