The Benchmark That Changed Everything: SGLang's First Performance Numbers on 8x Blackwell GPUs

Introduction

In the course of a sprawling ML infrastructure session spanning dozens of messages, one seemingly routine benchmark execution — message [msg 3168] — stands as a pivotal moment. It is the moment when months of environment setup, driver installation, CUDA toolkit configuration, model deployment, and speculative decoding research collided with cold, hard numbers. The message is deceptively simple: an SSH command executing a Python benchmark script against a freshly-deployed SGLang server running the Kimi-K2.5 INT4 model on 8 NVIDIA RTX PRO 6000 Blackwell GPUs. But the numbers it produced — approximately 27 tokens per second in single-stream mode — would trigger a cascade of re-evaluation, optimization, and ultimately a fundamental shift in strategy.

The Message Itself

The message contains a single bash command executed via SSH on the remote GPU server:

ssh root@10.1.230.174 '/root/ml-env/bin/python3 /tmp/sglang_bench.py' 2>&1

The output reveals the first three single-stream benchmark runs:

Warming up...
Warm-up done.

=== Single-Stream Benchmark ===
Prompt: Explain the theory of general relativity in detail...
  Tokens: 256 completion, 18 prompt
  Time: 9.61s, TPOT: 37.5ms, TPS: 26.7 tok/s

Prompt: Write a Python function to implement merge sort wi...
  Tokens: 256 completion, 20 prompt
  Time: 9.48s, TPOT: 37.0ms, TPS: 27.0 tok/s

Prompt: What are the main causes and effects of climate ch...
  Tokens: 256 completion, 19 prompt
  Time: 9.38s, TPOT: 36.6ms, TPS: 27.3 tok/s

=== Mul...

The output is truncated at "=== Mul..." — the multi-concurrent benchmark results that followed are not visible within this message. The full picture would only emerge in the subsequent message [msg 3169], where the assistant would compare these numbers against the vLLM baseline and draw the first conclusions.

The Context: A Long Road to a Running Server

To understand why this message matters, one must appreciate the journey that preceded it. The broader session (Segment 24) began with the assistant diagnosing what appeared to be a deadlocked SGLang server. In [msg 3159], the assistant found a running SGLang process (PID 50299) with all 8 GPUs showing 76GB of memory allocated but 0% GPU utilization. The log file ended abruptly after weight loading, and port 8000 was not listening. The natural conclusion was a deadlock.

What followed was an intensive debugging session spanning [msg 3160] through [msg 3165]. The assistant used strace to examine the process, finding all 131 threads stuck on futex waits. It dispatched a subagent task to analyze the SGLang initialization sequence, tracing through launch_server.py, model_runner.py, and the NCCL initialization path. The subagent produced a comprehensive analysis of the initialization pipeline.

Then came the breakthrough in [msg 3164]: the assistant re-examined the log file and discovered it had 312 lines — far more than the 80 lines it had initially tailed. The server had actually printed "The server is fired up and ready to roll!" at 22:59:08. Port 8000 was listening. The "hang" was simply a 5+ minute model loading time — the 547GB quantized model required extensive dequantization during weight loading, and the initial tail -80 had only captured the beginning of the log.

This discovery transformed the debugging session into a benchmarking session. In [msg 3166], the assistant sent a quick curl request to confirm the server was responding. In [msg 3167], it crafted a comprehensive benchmark script (/tmp/sglang_bench.py) that would measure both single-stream latency and multi-concurrent throughput. Message [msg 3168] is the execution of that script.

Why This Message Was Written

The message was written to answer a single, critical question: How fast is SGLang on this hardware? After investing enormous effort into getting SGLang running on SM120 Blackwell GPUs — including building custom sgl-kernel binaries, resolving import errors for unsupported model architectures, and waiting through a 5-minute model load — the assistant needed a quantitative baseline.

This baseline served multiple purposes:

Comparison against vLLM: The team had already benchmarked vLLM extensively, achieving 82.5 tok/s single-stream and 1,536 tok/s peak throughput (C=128). SGLang needed to at least approach these numbers to be viable.
EAGLE-3 speculative decoding evaluation: The entire speculative decoding pipeline depended on SGLang's ability to serve the base model efficiently. If SGLang's base performance was poor, speculative decoding would never close the gap.
Optimization target: Without a baseline, there was no way to measure improvement. The numbers from this message would become the "before" in an optimization campaign.

The Thinking Process Visible in the Message

The benchmark script itself reveals the assistant's thinking process. Written in [msg 3167], the script shows careful design decisions:

Two measurement functions: single_request (streaming) and run_non_stream (non-streaming). The assistant understood that streaming gives better timing granularity (TTFT, per-token timing) but non-streaming provides exact token counts via the usage field in the response.
Warm-up phase: A single short request before benchmarking, acknowledging that the first request after server startup may include JIT compilation or cache population overhead.
Multiple prompt types: Three different prompts (general relativity, merge sort, climate change) to test whether prompt content affects throughput — a sign the assistant was thinking about KV cache effects and prompt processing overhead.
Multi-concurrent sweep: Testing concurrencies from 1 to 128, showing an understanding of how throughput scales with batch size in continuous batching systems.
Statistical rigor: Using statistics.mean() for averaging and time.perf_counter() for high-precision timing. The assistant also made an implicit architectural assumption: that the benchmark should be run from the local machine via SSH rather than on the server itself. This avoids distorting measurements with the benchmark's own resource consumption, but introduces network latency as a potential confounder — though for 256-token generations taking ~9.5 seconds, network overhead is negligible.

Assumptions and Their Consequences

The most significant assumption embedded in this message is invisible: the server was launched with --disable-cuda-graph. This flag, visible in the process listing from [msg 3159], disables SGLang's CUDA graph optimization, which is critical for reducing per-step launch overhead in single-stream scenarios. The assistant had added this flag during the debugging phase to simplify the initialization path and avoid potential CUDA graph capture deadlocks — a reasonable debugging tactic, but one that catastrophically impacted performance.

The 27 tok/s result was therefore not representative of SGLang's true capability. The assistant recognized this immediately in the next message ([msg 3169]), where it attributed the 3x single-stream deficit to the disabled CUDA graphs. This assumption — that the debugging configuration was acceptable for benchmarking — was a mistake, but an understandable one given the pressure to get any numbers at all after the prolonged debugging session.

A second assumption was that the benchmark would complete without error. The truncated output ("=== Mul...") suggests the script may have been interrupted or the SSH session timed out during the multi-concurrent phase. The assistant never saw the full multi-concurrent results in this message, though they apparently completed successfully for the next message's analysis.

Input Knowledge Required

Understanding this message requires knowledge of:

SGLang architecture: The concept of CUDA graphs, continuous batching, and the server initialization sequence
LLM serving metrics: TPOT (time per output token), TTFT (time to first token), tokens per second, and how these interact with concurrency
The hardware context: 8x Blackwell GPUs with 96GB HBM each, PCIe-interconnected, running CUDA 13.1
The model: Kimi-K2.5 INT4, a 547GB quantized Mixture-of-Experts model with MLA (Multi-head Latent Attention)
The vLLM baseline: 82.5 tok/s single-stream, 1,536 tok/s peak — the numbers SGLang needed to beat
The debugging history: The server was launched with --disable-cuda-graph during hang diagnosis, which directly explains the poor single-stream performance

Output Knowledge Created

This message created the first quantitative performance data point for SGLang on SM120 Blackwell GPUs with the Kimi-K2.5 INT4 model. The specific knowledge produced:

Single-stream throughput: ~27 tok/s (range 26.7–27.3 across three prompts)
TPOT: ~37ms per token (range 36.6–37.5ms)
Consistency: Very stable across different prompt types (general relativity, merge sort, climate change), suggesting the prompt processing overhead is minimal relative to generation time
Generation time: ~9.5 seconds for 256 tokens, implying the model is compute-bound rather than memory-bound in this configuration These numbers would immediately be contextualized in [msg 3169] against the vLLM baseline, revealing the 3x single-stream deficit and the surprising 44% peak throughput advantage. This duality — worse for single users, better for batch processing — would drive the entire subsequent optimization campaign.

The Broader Significance

Message [msg 3168] sits at the inflection point of the entire Segment 24 narrative arc. Before it: debugging, deadlock analysis, and the relief of discovering the server was actually running. After it: a systematic optimization campaign including CUDA graph re-enablement (achieving 63.6 tok/s), EAGLE-3 speculative decoding testing (finding no benefit), NCCL tuning with environment variables, and ultimately a pivot to retraining the EAGLE-3 drafter using SGLang-based hidden state extraction.

The message is also a testament to the iterative nature of ML infrastructure work. The benchmark that produced disappointing numbers was run with a known suboptimal configuration — but it had to be run to confirm the problem, to establish the baseline, and to motivate the optimization work that followed. Without the 27 tok/s result, there would be no urgency to tune NCCL settings, no data point to improve upon, and no clear signal that CUDA graphs were essential.

In this sense, message [msg 3168] is not about the numbers it contains, but about the questions it forces the reader to ask: Why is it so slow? What's different from vLLM? Can we fix it? And the answers to those questions would reshape the entire remainder of the session.