The TP8 Benchmark That Completed the Parallelism Puzzle
In the high-stakes world of large language model inference on multi-GPU systems, every microsecond counts. The message at the center of this article — <msg id=11540> — is a deceptively simple benchmark execution that represents the culmination of a systematic, multi-hour investigation into parallelism strategies for the Kimi K2.6 model running on 8× NVIDIA RTX PRO 6000 Blackwell GPUs connected via PCIe. What makes this message significant is not the benchmark script itself, but the story it tells about the tradeoffs between tensor parallelism, expert parallelism, and CUDA graph optimization on a PCIe-limited multi-GPU system.
The Message in Full
The assistant executed a Python benchmark script against a freshly deployed SGLang inference server running with tensor parallelism across 8 GPUs (TP8), CUDA graphs enabled, continuous decode steps set to 8, and a maximum of 256 concurrent requests. The script sent chat completion requests to the server at http://10.1.2.200:30001, first warming up with two trivial requests, then measuring single-request throughput with a quicksort prompt at 512 tokens, and finally sweeping concurrency levels from 32 to 256 with 2048-token outputs.
The results told a striking story:
=== TP8 tuned (cuda graphs, continuous=8, maxreq=256) ===
C=1: 97.9 tok/s
C= 32: 683.9 tok/s wall=75.7s
C= 64: 951.4 tok/s wall=105.4s
C=128: 1291.0 tok/s wall=156.5s
C=192: 1213.9 tok/s wall=250.5s
C=256: 1263.0 tok/s wall=317.5sAt first glance, these are just numbers. But in the context of the preceding hours of work, they represent a critical data point that validated — and challenged — the assistant's understanding of how parallelism should work on this hardware.
The Reasoning and Motivation: Why This Message Was Written
To understand why this message exists, one must trace back through the assistant's systematic investigation. The session had been benchmarking four parallelism strategies for the Kimi K2.6 model, a Mixture-of-Experts (MoE) architecture that presents unique challenges for distributed inference. The assistant had already tested:
- EP8 (expert parallelism across 8 GPUs): achieved 65.2 tok/s at C=1 and ~1500 tok/s peak throughput (
<msg id=11527>) - EP8 with EPLB (expert load balancing): a failed experiment that dropped performance to 862 tok/s peak (
<msg id=11531>) - EP4 (expert parallelism across 4 groups of 2 TP GPUs): achieved 69.0 tok/s at C=1 and 1530.5 tok/s peak (
<msg id=11537>) But one critical data point was missing: a fair TP8 baseline. Earlier in the session, the assistant had tested TP8 without CUDA graphs, achieving a dismal 26.3 tok/s at C=1 — but that was before the CUDA 13.0 toolkit was properly installed and before CUDA graph optimization was enabled. The assistant needed to know: how much of TP8's poor single-request performance was inherent to tensor parallelism on PCIe, and how much was due to missing optimizations? This message was written to answer that question. The assistant had just spent 570 seconds waiting for the TP8-tuned service to start (<msg id=11539>— the longest service launch of the entire session), and now it needed the benchmark results to complete the comparison matrix. The motivation was clear: without this data point, the parallelism comparison would be incomplete, and the assistant couldn't make a confident recommendation about which strategy to use for production deployment.## How Decisions Were Made: The Experimental Design The benchmark script itself reveals several deliberate design decisions. First, the assistant chose to use 2048-token outputs for the concurrency sweep rather than the 512-token prompts used for the C=1 measurement. This is a critical detail: short outputs (like the 56-token "What is 2+2?" response seen in earlier benchmarks) produce unreliable throughput measurements because the overhead of request dispatch dominates. By requesting 2048 tokens, the assistant ensured that the measurement captured sustained generation throughput rather than startup latency. Second, the concurrency sweep used a fixed set of five prompts cycled across requests. This is a pragmatic choice that balances reproducibility with diversity — using the same prompts across all concurrency levels ensures that differences in output length don't confound the comparison, while cycling through five different prompts prevents the server from caching a single response pattern. Third, the assistant usedconcurrent.futures.ThreadPoolExecutorwithconcurrent.futures.as_completedrather than waiting for all futures to complete. This design choice means the wall-clock measurement captures the time from the first request submission to the last response received, which is the correct measure for aggregate throughput. The earlier EP8 benchmark (<msg id=11527>) usedconcurrent.futures.waitwhich blocks until all futures complete — a subtle difference that the assistant standardized across later benchmarks. The decision to warm up with two "Say OK" requests is also deliberate. SGLang's CUDA graph capture happens during the first few requests, and the Triton attention backend may JIT-compile kernels on first use. Without warmup, the first measured request would include compilation overhead, artificially deflating the throughput numbers.
Assumptions Made by the Assistant
The assistant operated under several assumptions when running this benchmark. The most important was that the TP8-tuned service had actually started with CUDA graphs enabled. The service took 570 seconds to start (<msg id=11539>), which was dramatically longer than the 90–150 seconds typical for other configurations. The assistant assumed this was due to CUDA graph capture during model loading, but it could also have been caused by NCCL topology discovery or memory initialization. The assistant did not verify that CUDA graphs were actually active — it trusted that the --attention-backend triton flag and the default CUDA graph behavior in SGLang would enable them.
Another assumption was that the NCCL environment variables set in the service unit file (NCCL_IB_DISABLE=1, NCCL_P2P_LEVEL=5, NCCL_ALGO=Ring, etc.) were optimal for the PCIe-only Blackwell system. These settings disable InfiniBand (correct, since the system uses PCIe), set P2P level to 5 (NVLink detection — potentially incorrect for a PCIe-only system), and force the Ring algorithm. The assistant assumed these settings from earlier tuning work, but did not verify that they were actually beneficial for the TP8 configuration.
The assistant also assumed that max-running-requests=256 and num-continuous-decode-steps=8 were reasonable defaults based on the EP8 tuning. But TP8 has different scheduling dynamics — with tensor parallelism, all GPUs must synchronize at every layer, so the optimal continuous decode steps might differ. The assistant did not sweep these parameters for TP8.
Mistakes and Incorrect Assumptions
The most significant issue with this benchmark is that it was run without verifying that CUDA graphs were actually active. The 570-second startup time was suspiciously long — earlier TP8 benchmarks without graphs started in ~90 seconds. If CUDA graph capture was the cause, the assistant should have verified this by checking the SGLang logs for graph capture messages. Without this verification, the 97.9 tok/s result might include graph capture overhead in its measurement.
Additionally, the benchmark did not include a C=1 measurement at 2048 tokens to match the concurrency sweep. The C=1 measurement was done at 512 tokens (the quicksort prompt), which means the single-request throughput of 97.9 tok/s is not directly comparable to the concurrency sweep results at 2048 tokens. The assistant likely knew this — the 512-token measurement was a quick check, while the 2048-token concurrency sweep was the real data. But this inconsistency means the C=1 number should be treated as approximate.
The assistant also did not test TP8 with CUDA graphs at lower concurrency levels (C=4, C=8, C=16) that were included in the EP8 benchmark. This makes it harder to understand where TP8's scaling advantage begins to plateau. The jump from 97.9 tok/s at C=1 to 683.9 tok/s at C=32 is a 7× improvement, but without intermediate data points, we can't tell if this scaling is linear or if there's a knee in the curve.
Input Knowledge Required
To fully understand this message, one needs knowledge of several technical domains. The concept of tensor parallelism (TP) — splitting individual layer computations across GPUs — is essential, as is understanding that TP requires AllReduce synchronization at every layer, which becomes a bottleneck on PCIe interconnects. Expert parallelism (EP) for MoE models distributes different experts across GPUs, avoiding AllReduce on the MoE layers but requiring AllGather for the expert routing. CUDA graphs eliminate CPU-side kernel launch overhead by pre-recording a sequence of GPU operations, which is particularly beneficial for small batches where kernel launch latency dominates.
Knowledge of the Kimi K2.6 model architecture is also important — it's a Mixture-of-Experts model with approximately 260 billion parameters, where only a subset of experts is activated for each token. This makes EP particularly attractive because the expert computation can be distributed without cross-GPU communication for the MoE layers.
The hardware context — 8× NVIDIA RTX PRO 6000 Blackwell GPUs connected via PCIe (not NVLink) — is critical. On PCIe, inter-GPU bandwidth is limited to ~32 GB/s per direction (PCIe Gen5 x16), compared to ~900 GB/s for NVLink. This makes communication-heavy parallelism strategies like TP potentially bottlenecked by the interconnect.
Output Knowledge Created
This message produced the final piece of the parallelism comparison puzzle. The results showed that TP8 with CUDA graphs achieved 97.9 tok/s at C=1 — a dramatic improvement over the 26.3 tok/s without graphs (a 3.7× speedup from CUDA graphs alone). This was the best single-request throughput of any configuration tested, surpassing EP8's 65.2 tok/s and EP4's 69.0 tok/s.
However, at high concurrency, TP8 plateaued at ~1291 tok/s (C=128), while EP4 reached 1530 tok/s (C=256). The TP8 throughput actually decreased from C=128 to C=192 (1291 → 1214 tok/s), suggesting that the PCIe interconnect was saturated and additional concurrency only added overhead. This is a classic sign of a communication-bound system.
The message also implicitly validated the assistant's earlier hypothesis that expert parallelism avoids PCIe AllReduce bottlenecks. The EP configurations scaled cleanly with concurrency, while TP8 hit a wall. The 97.9 tok/s single-request result showed that CUDA graphs could make TP8 competitive for interactive serving, but for batch throughput, EP remained superior.
The Thinking Process Visible in the Reasoning
The assistant's reasoning is not explicitly visible in this message — it's a bare tool call with a Python script and its output. But the thinking is embedded in the experimental design. The assistant chose to test TP8 with CUDA graphs after having already tested EP8, EP8+EPLB, and EP4. The order of experiments reveals a hypothesis-testing approach: first establish the EP8 baseline (the most intuitive configuration for MoE), then try optimizations (EPLB), then explore alternatives (EP4), and finally fill in the missing comparison point (TP8 with graphs).
The 570-second wait for the TP8 service to start (<msg id=11539>) must have been a tense period. The assistant had already invested hours in benchmarking, and a service failure would have meant starting over. The fact that the assistant persisted through 38 polling attempts (each 15 seconds apart) shows a commitment to completing the comparison matrix.
The subsequent message (<msg id=11541>) reveals the synthesis: the assistant immediately compiled a comparison table and drew conclusions. TP8 tuned was declared the winner for "interactive serving" (single-request latency), while EP4 won for "sustained throughput." This two-pronged recommendation is the direct output of the thinking that motivated this benchmark.
Conclusion
Message <msg id=11540> is a seemingly simple benchmark execution that represents the capstone of a systematic investigation into parallelism strategies for MoE inference on PCIe-connected GPUs. It completed the comparison matrix by providing the missing TP8-with-CUDA-graphs data point, revealing that CUDA graphs could transform TP8 from the worst-performing configuration (26 tok/s) to the best single-request performer (98 tok/s), while confirming that expert parallelism remained superior for batch throughput. The message embodies the scientific method in ML engineering: form hypotheses, design experiments, gather data, and let the numbers speak. In the high-dimensional optimization space of LLM inference, this disciplined approach is what separates informed deployment decisions from guesswork.