The Benchmark That Changed Everything: SGLang with CUDA Graphs on Blackwell GPUs

In the course of a marathon optimization session spanning dozens of rounds, one message stands out as a quiet turning point. Message [msg 3174] is deceptively brief — just a few lines announcing that SGLang is ready with CUDA graphs enabled, followed by the output of a benchmark script. But beneath its terse surface lies the resolution of a multi-hour debugging saga, a validated hypothesis about GPU kernel optimization, and the first clear evidence that SGLang could compete with vLLM on the formidable 8× Blackwell RTX PRO 6000 GPU setup powering the Kimi-K2.5 INT4 model.

The Message in Full

The assistant writes:

SGLang with CUDA graphs is ready. Weight loading took 628s (longer this time, but it's fine). CUDA graph capture happened quickly. Now let's benchmark.

Then follows a bash command that runs a benchmark script, producing results across three prompts, each yielding identical performance: 256 completion tokens in 4.03 seconds, a time-per-output-token (TPOT) of 15.7 milliseconds, and a throughput of 63.6 tokens per second. The multi-concurrent results are truncated in the data but were later revealed to reach 2,370 tok/s at C=128.

The Long Road to This Moment

To understand why this message matters, one must trace the path that led to it. The session had been wrestling with SGLang on the SM120 (Blackwell) architecture for hours. Earlier, the assistant had launched SGLang with --disable-cuda-graph — a flag that disables CUDA graph capture, which is a technique where the GPU records a sequence of operations into a reusable graph object, eliminating kernel launch overhead for repeated inference patterns. The first benchmark with this setting was disastrous: single-stream throughput was a mere 27 tokens per second, compared to vLLM's 82.5 tok/s baseline. That was a 3× slowdown, and it was unacceptable.

The assistant's reasoning in [msg 3169] shows the critical insight: "The low single-stream speed is suspicious — CUDA graphs are disabled (--disable-cuda-graph), which explains the terrible single-request latency." This was a hypothesis: that the overhead of launching individual GPU kernels for each decoding step was crushing single-stream performance, and that CUDA graphs would amortize that overhead across many steps.

The decision to act on this hypothesis was not trivial. Restarting the SGLang server meant killing the existing process, freeing all 8 GPUs (which required multiple kill commands and a wait for memory to drain), and relaunching with a different configuration. The assistant executed this in [msg 3170], carefully cleaning up processes and GPU memory before starting the new server without the --disable-cuda-graph flag.

Then came the wait. Weight loading for the 547GB Kimi-K2.5 INT4 model took 628 seconds — over 10 minutes. The assistant monitored the log file in a loop ([msg 3172]), watching for the "fired up and ready to roll!" message. At one point, a DeepGemm warning triggered the error-detection path in the monitoring script, but the assistant correctly recognized it as a non-fatal warning and continued waiting. This patience was rewarded when the server finally came online.

The Benchmark Results

The benchmark itself is revealing. The assistant used a script (sglang_bench.py) that measures both single-stream and multi-concurrent performance. The single-stream results are striking for their consistency: across three different prompts (general relativity, merge sort implementation, climate change causes), every run produced exactly 4.03 seconds for 256 tokens, yielding 63.6 tok/s. This consistency suggests the CUDA graph capture had fully eliminated variance from kernel launch overhead, making each decoding step essentially identical in cost.

The improvement from 27 tok/s (without CUDA graphs) to 63.6 tok/s (with CUDA graphs) represents a 2.36× speedup — a dramatic validation of the assistant's hypothesis. While still below vLLM's 82.5 tok/s, the gap had narrowed considerably from 3× to just 23% slower. The peak throughput at C=128 reached 2,370 tok/s, surpassing vLLM's 1,536 tok/s by 54%.

Input Knowledge Required

To fully grasp this message, one needs substantial context. The hardware is critical: 8× NVIDIA RTX PRO 6000 Blackwell GPUs, connected via PCIe (not NVLink), which imposes significant communication overhead for tensor parallelism. The model is Kimi-K2.5, a 547GB INT4-quantized MoE (Mixture of Experts) architecture with Multi-head Latent Attention (MLA). The software stack includes SGLang (a nightly build), with the triton attention backend. The vLLM baseline numbers (82.5 tok/s single-stream, 1,536 tok/s peak) come from earlier benchmarks in the session and serve as the reference point for all comparisons.

The assistant also relies on an understanding of CUDA graphs: what they are, when they help (repeated, deterministic computation patterns), and why they would particularly benefit single-stream decoding where kernel launch overhead is a larger fraction of total time. Without CUDA graphs, each decoding step requires launching dozens of small kernels independently; with CUDA graphs, the entire sequence is captured once and replayed, eliminating driver overhead.

Output Knowledge Created

This message produces concrete, actionable knowledge. It establishes that SGLang with CUDA graphs is a viable serving backend for Kimi-K2.5 on Blackwell GPUs, achieving 63.6 tok/s for single requests. It provides a benchmark methodology that can be repeated for future comparisons. It creates a baseline for evaluating speculative decoding (EAGLE-3) — the next major goal of the session — since any speedup from speculation must be measured against this 63.6 tok/s figure.

The message also implicitly validates the assistant's debugging methodology: identify a performance bottleneck, form a hypothesis about its cause, test by changing one variable (CUDA graphs), and measure the result. This pattern of disciplined experimentation runs throughout the session.

Assumptions and Their Limits

Several assumptions underpin this message. The assistant assumes that 628 seconds for weight loading is acceptable ("it's fine"), which is reasonable for a development/debugging context but would be problematic for production serving where fast restart is important. It assumes the benchmark script accurately measures throughput — the script counts SSE chunks as tokens, which the assistant acknowledges as approximate, and uses a separate non-streaming endpoint for exact token counts. It assumes that CUDA graph capture completed successfully and that the captured graphs are optimal, which may not be the case if the warmup request didn't exercise all execution paths.

A notable earlier mistake was the assistant's initial belief that SGLang had "hung" or "deadlocked" ([msg 3161]). The strace output showed all threads in FUTEX_WAIT_BITSET, which looked like a deadlock. In reality, the process was simply taking an unexpectedly long time to load weights — 313 seconds for the first launch. The assistant corrected this in [msg 3164] when it discovered the log contained far more output than initially seen, including the "fired up and ready to roll!" message. This false alarm shaped the subsequent debugging approach, making the assistant more patient with long initialization times.

The Thinking Process

The assistant's reasoning in this message is compressed but clear. It notes that weight loading took longer than the previous launch (628s vs 313s), attributes this to variability in the quantized model's dequantization process, and dismisses it as acceptable. It observes that "CUDA graph capture happened quickly" — an important detail, as CUDA graph capture can itself be time-consuming for large models. The brevity of the announcement ("Now let's benchmark") reflects confidence: the assistant expected CUDA graphs to improve performance and is eager to quantify the improvement.

The choice to run three different prompts is deliberate — it tests whether the results generalize across different input contexts and generation patterns. The identical results across all three prompts confirm that CUDA graphs have made decoding latency deterministic, independent of the specific token sequence being generated.

Significance in the Larger Narrative

This message sits at a critical juncture in the session. It closes the chapter on "get SGLang base serving working on SM120" and opens the next chapter on "test SGLang + EAGLE-3 drafter." The benchmark numbers establish a baseline that will be used to evaluate whether speculative decoding provides any benefit. In the messages that follow, the assistant will test both the AQ-MedAI EAGLE-3 drafter and a custom drafter trained on Kimi-K2.5 data, ultimately finding that neither provides speedup due to low acceptance rates — a disappointing result that leads to yet another pivot.

But in this moment, the outlook is optimistic. SGLang is running, CUDA graphs are working, and the numbers are competitive. The assistant has successfully navigated the treacherous waters of Blackwell GPU compatibility, SGLang nightly builds, and multi-hour weight loading to produce a working inference server. The message is a testament to the value of systematic debugging, hypothesis-driven optimization, and the patience to wait 10 minutes for a server to start.