The Benchmark That Changed Everything: Measuring SGLang's Performance on 8× Blackwell GPUs

Introduction

In the sprawling narrative of deploying and optimizing large language models on cutting-edge hardware, few moments are as pivotal as the first reliable performance measurement. Message 3167 of this opencode session captures exactly such a moment. After an extended debugging odyssey spanning multiple segments and dozens of messages—during which the assistant wrestled with CUDA toolkit incompatibilities, flash-attn build failures, NCCL deadlocks, and an SGLang server that appeared to hang but was merely taking its time—the assistant finally has a working SGLang inference server. The response is not celebration but methodical measurement. This message contains the creation and deployment of a comprehensive benchmarking script designed to establish the baseline performance of the Kimi-K2.5 INT4 model running on 8× RTX PRO 6000 Blackwell GPUs through SGLang.

The message is deceptively simple: a single-line observation followed by a bash heredoc that writes a Python benchmark script, then an scp command to transfer it. But within this seemingly routine operation lies a wealth of technical decision-making, architectural understanding, and strategic thinking. This article examines message 3167 in depth—its motivations, its assumptions, its embedded knowledge, and its role in the broader optimization campaign.

The Road to This Message

To understand why message 3167 exists, one must appreciate the journey that preceded it. The session began in Segment 0 with a bare-metal Ubuntu 24.04 setup, installing NVIDIA drivers, CUDA toolkits, and wrestling with flash-attn compilation. By Segment 19, the focus had shifted to profiling the Kimi-K2.5 INT4 model on 8× Blackwell GPUs, where AllReduce was identified as the dominant bottleneck at 51.5% of decode time. This led to an investigation of speculative decoding as a software-only optimization path, which in turn spawned a massive EAGLE-3 training pipeline across Segments 20–23.

Segment 23 ended on a particularly frustrating note: the EAGLE-3 drafter trained on Kimi-K2.5 data achieved only a ~15% acceptance rate in vLLM, yielding a net throughput regression (0.66×). The assistant pivoted to SGLang, hoping its different architecture might yield better results. But SGLang immediately deadlocked on the SM120 architecture, leading to an intense debugging session in Segment 24.

The immediate predecessor to message 3167 is a series of exchanges where the assistant believed SGLang was hung after weight loading. It ran strace, examined file descriptors, checked port bindings, and analyzed initialization code. Then came the breakthrough in message 3165: the server was never hung. It had simply taken 313 seconds to load the 547GB quantized model—dequantization on 8 GPUs is not instantaneous. The server was "fired up and ready to roll," as its own log line proclaimed.

Message 3166 confirmed the server was functional with a quick curl test, and then message 3167 takes the next logical step: systematic performance measurement.

The Message in Full

The message opens with a brief observation:

The model is responding — note it's generating thinking tokens inline (not using thinking tags in this response). Now let me run a proper single-stream benchmark to measure TPOT and throughput.

This single sentence reveals that the assistant has already performed an ad-hoc test (the curl in message 3166) and observed something noteworthy about the model's output format. The Kimi-K2.5 model appears to emit reasoning tokens inline rather than using the structured thinking tags that some models employ. This is a minor but important observation—it affects how one might parse outputs and measure token generation rates.

The assistant then creates a Python benchmark script via a bash heredoc, writing to /tmp/sglang_bench.py, and transfers it to the remote machine via scp. The script itself is the core of the message.

Anatomy of the Benchmark Script

The script, written in Python, is a well-structured benchmarking tool that measures two distinct performance dimensions: single-stream latency and multi-concurrent throughput. Let us examine its design decisions.

Dual Measurement Strategy

The script defines two request functions: single_request for streaming measurement and run_non_stream for exact token counting. This dual approach is a deliberate design choice that acknowledges a fundamental tension in LLM benchmarking: streaming gives you timing granularity (time to first token, inter-token latency) but imprecise token counts (SSE chunks may split tokens arbitrarily), while non-streaming gives exact token counts but only total latency. By using both, the assistant can cross-validate measurements.

Streaming Timing

The single_request function measures:

Non-Streaming Precision

The run_non_stream function uses the standard chat completions endpoint without streaming, extracting exact token counts from the usage field in the response. This gives precise completion_tokens and prompt_tokens counts, enabling accurate TPOT (time per output token) calculations.

Warm-Up Phase

The script includes a warm-up request ("Say hello." with 16 max tokens) before any measurements. This is a critical step for GPU inference servers, as the first request often triggers CUDA kernel compilation, memory allocation, and other one-time overheads. Without warm-up, the first measurement would be contaminated by these initialization costs.

Single-Stream Benchmark

Three prompts are used for single-stream testing:

  1. "Explain the theory of general relativity in detail."
  2. "Write a Python function to implement merge sort with detailed comments."
  3. "What are the main causes and effects of climate change?" These prompts are chosen to be diverse in topic but similar in length and complexity. Each is a knowledge-intensive question that would naturally generate a substantial response (up to 256 tokens). The assistant prints per-prompt metrics including completion tokens, prompt tokens, total time, TPOT in milliseconds, and tokens per second.

Multi-Concurrent Benchmark

The concurrency test uses a single prompt ("Explain quantum computing in detail...") and varies the number of concurrent requests: 1, 4, 8, 16, 32, 64, 128. This is a standard load-testing pattern that reveals how throughput scales with batch size.

The script uses ThreadPoolExecutor for concurrent requests and measures wall-clock time from the start of the first request to the completion of the last. Throughput is calculated as total tokens across all requests divided by wall time. This gives the aggregate throughput in tok/s, which is the key metric for server capacity.

The concurrency levels are well-chosen. They start at 1 (the single-stream baseline), ramp up through moderate concurrency (4, 8, 16) to stress-test levels (32, 64, 128). The maximum of 128 is particularly aggressive—with 8 GPUs and tensor parallelism, this represents 16 concurrent requests per GPU, which would heavily stress the memory subsystem and NCCL communication.

Technical Decisions and Assumptions

Assumption: Server Stability

The assistant assumes the SGLang server will remain stable under load. This is not a trivial assumption—the server had just been proven to work with a single request, but its behavior under concurrent load was entirely unknown. The choice to run concurrency tests up to 128 requests assumes the server has sufficient queue capacity and won't OOM or deadlock.

Assumption: ThreadPoolExecutor Adequacy

Using ThreadPoolExecutor for concurrent HTTP requests assumes that Python's threading model is sufficient for this workload. Since HTTP requests are I/O-bound (waiting for network responses), threading is appropriate—the GIL is not a bottleneck because the threads spend most of their time blocked on requests.post. However, the client-side processing of streaming responses could introduce some overhead.

Decision: Non-Streaming for Exact Counts

The choice to use non-streaming requests for the actual benchmark (rather than streaming) prioritizes measurement accuracy over timing granularity. This is the right call for establishing a baseline—exact token counts are essential for TPOT calculation. The streaming function exists but is used only for TTFT measurement in the single-stream section.

Decision: Fixed Max Tokens

All requests use max_tokens=256, which standardizes the generation length. This is important because TPOT can vary with sequence length (due to KV cache effects, attention computation scaling, etc.). By fixing the output length, the assistant ensures measurements are comparable.

Decision: Temperature 0

All requests use temperature=0, which disables sampling randomness and makes the output deterministic. This is standard practice for benchmarking because it eliminates variance from the sampling process. However, it means the benchmark doesn't measure the overhead of sampling (which is negligible for most implementations but worth noting).

Input Knowledge Required

To fully understand this message, one needs knowledge of:

  1. The SGLang server architecture: Understanding that SGLang exposes OpenAI-compatible endpoints (/v1/chat/completions) with both streaming and non-streaming modes.
  2. LLM benchmarking methodology: Knowing why TTFT, TPOT, and throughput are important metrics, why warm-up is necessary, and why concurrency scaling matters.
  3. The hardware context: The server runs on 8× RTX PRO 6000 Blackwell GPUs with 96GB each, using tensor parallelism (TP=8). This affects how concurrency scales—each request consumes GPU memory for KV cache, and the total memory across 8 GPUs limits the maximum batch size.
  4. The vLLM baseline: From earlier messages, vLLM achieved 82.5 tok/s single-stream and 1,536 tok/s peak throughput. These numbers are the implicit comparison targets for the SGLang benchmark.
  5. Python threading and HTTP: Understanding that ThreadPoolExecutor with requests.post is a reasonable pattern for concurrent HTTP benchmarking, and that streaming responses require special handling.

Output Knowledge Created

This message produces several forms of knowledge:

  1. A reusable benchmark script: The script itself is a tool that can be run repeatedly to track performance changes as the server configuration evolves.
  2. Baseline SGLang performance metrics: The actual numbers produced (63.6 tok/s single-stream, 2,370 tok/s peak at C=128) become the reference point for all subsequent optimization efforts.
  3. Comparative data: The SGLang numbers can be directly compared against vLLM's 82.5 tok/s single-stream and 1,536 tok/s peak, revealing that SGLang has higher peak throughput but worse single-stream latency.
  4. Validation of the server: Successfully running the benchmark proves the SGLang server is production-ready for the model, unblocking the EAGLE-3 testing that follows.

The Broader Significance

Message 3167 sits at a critical inflection point in the session. The previous ~20 messages were consumed by debugging—chasing deadlocks, reading strace output, examining SGLang initialization code. Message 3167 represents the first constructive, forward-moving action after that debugging phase. It transforms the server from "something that might work" to "something whose performance we can measure and optimize."

The benchmark results that follow this message (in subsequent messages not shown here) reveal that SGLang achieves 63.6 tok/s single-stream and 2,370 tok/s peak throughput. Compared to vLLM's 82.5 tok/s single-stream and 1,536 tok/s peak, the trade-off is clear: SGLang excels at batch throughput (2,370 vs 1,536 tok/s, a 54% improvement) but lags in single-stream latency (63.6 vs 82.5 tok/s, a 23% deficit). This finding directly shapes the optimization strategy that follows: the assistant pivots to tuning SGLang's single-stream performance using NCCL environment variables that vLLM had been using.

The benchmark also sets the stage for the EAGLE-3 speculative decoding tests. Without baseline numbers, it would be impossible to determine whether speculative decoding provides any benefit. The subsequent tests (AQ-MedAI drafter at ~42% acceptance, custom K2.5 drafter at ~25% acceptance) can only be evaluated against the baseline established here.

Conclusion

Message 3167 is a masterclass in the pragmatics of ML engineering. It arrives at a moment of uncertainty—the server is working, but how well? The response is not guesswork but systematic measurement. The assistant designs a benchmark that covers both latency and throughput, uses both streaming and non-streaming endpoints for cross-validation, includes proper warm-up, and tests a wide range of concurrency levels.

The script itself is a model of clarity and purpose. Every decision—from the choice of prompts to the concurrency levels to the use of ThreadPoolExecutor—reflects a deep understanding of what matters when measuring LLM inference performance. The assistant is not just running a benchmark; it is establishing the foundation for all subsequent optimization decisions.

In the broader arc of the session, message 3167 is the moment when debugging ends and engineering begins. It transforms the SGLang server from a black box that might work into a measured system with known characteristics. Those measurements will drive the next phase of the project: tuning single-stream performance, evaluating speculative decoding, and ultimately deciding whether SGLang or vLLM is the right serving platform for the Kimi-K2.5 model on Blackwell hardware.