The Benchmark That Changed Everything: How CUDA Graphs Transformed SGLang's Performance on 8x Blackwell GPUs

In the middle of a marathon session tuning speculative decoding for the Kimi-K2.5 model on 8x RTX PRO 6000 Blackwell GPUs, the assistant produced a single message ([msg 3175]) that crystallized the entire trajectory of the investigation. This message—a concise benchmark comparison table followed by a decisive pivot—represents a moment of clarity after hours of confusion, deadlock debugging, and false starts. It is the fulcrum upon which the rest of the session turns.

The Message

Here is the message in full:

CUDA graphs made a huge difference for single-stream! Here's the comparison:

>

| Metric | vLLM | SGLang (no CG) | SGLang (with CG) | |--------|------|----------------|-------------------| | Single-stream | 82.5 tok/s (12.1ms) | 27 tok/s (37ms) | 63.6 tok/s (15.7ms) | | C=128 throughput | 1,536 tok/s | 2,216 tok/s | 2,370 tok/s |

>

SGLang single-stream is now 63.6 tok/s vs vLLM's 82.5 tok/s — 77% of vLLM, which is reasonable given the different attention backend (triton vs flashinfer). Peak throughput is better at 2,370 tok/s. The single-stream gap might be addressable with flashinfer backend, but the key goal is EAGLE-3. Let me kill this and start with the AQ-MedAI EAGLE-3 drafter.

On its surface, this is a routine progress update. But beneath the numbers lies a rich story of debugging, hardware characterization, and strategic decision-making.

The Road to This Message: A False Hang and a 628-Second Weight Load

To understand why this message matters, we must appreciate what preceded it. The assistant had spent the previous several messages ([msg 3161] through [msg 3174]) in a state of high alert, convinced that SGLang was deadlocked after weight loading. The server had started, allocated 76GB per GPU, and then gone silent—no further log output, 0% GPU utilization, port 8000 not listening. Strace showed 131 threads all stuck on futex waits. The assistant dove deep into SGLang's initialization code, tracing the sequence after weight loading: NCCL initialization, memory pool allocation, CUDA graph capture, warmup inference.

Then came the breakthrough at [msg 3164]: the assistant realized that the log file contained far more output than initially visible. All 8 tensor-parallel ranks had printed "Load weight end," the memory pool was allocated, and there was even a prefill batch logged. The server wasn't hung—it was just agonizingly slow. Weight loading took 313 seconds for the first attempt and 628 seconds for the second, because the quantized model (GLM-5-NVFP4, 547GB) required dequantization on load.

This discovery fundamentally changed the assistant's understanding of the system. What looked like a catastrophic deadlock was actually a slow initialization. The lesson was twofold: first, the model's weight format required significant computation to load, and second, the assistant's monitoring script was checking too early, triggering false alarms. The assistant adjusted its wait time from 5-second intervals to 15-second intervals over 80 iterations, demonstrating adaptive troubleshooting.

The CUDA Graph Revelation

The initial SGLang benchmark at [msg 3168] was disappointing: 27 tok/s single-stream, compared to vLLM's 82.5 tok/s. The assistant correctly identified the culprit: the --disable-cuda-graph flag. CUDA graphs are a critical optimization that captures a sequence of GPU kernel launches into a single executable graph, eliminating kernel launch overhead. Without them, each token generation step pays the full cost of launching individual kernels—a penalty that is especially severe for single-stream workloads where there is no batching to amortize the overhead.

The assistant killed the server, freed all GPUs, and relaunched without the --disable-cuda-graph flag. The result was dramatic: single-stream throughput jumped from 27 tok/s to 63.6 tok/s, a 2.4x improvement. Peak throughput at C=128 concurrency rose from 2,216 tok/s to 2,370 tok/s. CUDA graphs were not optional—they were essential.## Interpreting the Benchmark Table

The three-column comparison in the message tells a nuanced story. The vLLM baseline (82.5 tok/s single-stream, 1,536 tok/s peak) was established in earlier session segments and represented the gold standard the team was trying to match or exceed. SGLang without CUDA graphs was embarrassingly slow at 27 tok/s—barely a third of vLLM's single-stream performance. With CUDA graphs enabled, SGLang reached 63.6 tok/s, or 77% of vLLM.

The peak throughput numbers tell a different story. At C=128 concurrency, SGLang without CUDA graphs already outperformed vLLM (2,216 vs 1,536 tok/s), and with CUDA graphs it pulled further ahead (2,370 tok/s). This inversion is characteristic of systems where single-stream latency is dominated by kernel launch overhead (which CUDA graphs fix) but batch throughput is limited by compute and memory bandwidth (where SGLang's triton attention backend apparently excels).

The assistant's interpretation of the single-stream gap as "reasonable given the different attention backend (triton vs flashinfer)" reveals an important assumption: that the attention backend is the primary differentiator. This assumption would later be challenged when the assistant experimented with --attention-backend flashinfer in subsequent tuning, but at this moment it served as a working hypothesis that justified moving forward rather than getting bogged down in optimization.

The Strategic Pivot

The most consequential part of the message is the final sentence: "Let me kill this and start with the AQ-MedAI EAGLE-3 drafter." This decision embodies a critical trade-off that defines engineering work on large language model serving. The assistant had two paths:

  1. Optimize SGLang single-stream to match vLLM — chasing the remaining ~23% gap through attention backend selection, NCCL tuning, and other knobs.
  2. Test EAGLE-3 speculative decoding — the primary goal of the entire session, which could potentially deliver much larger gains than closing the single-stream gap. The assistant chose path 2, reasoning that "the key goal is EAGLE-3." This was a correct prioritization. Even if SGLang single-stream were optimized to match vLLM's 82.5 tok/s, the improvement would be at most 30%—while successful speculative decoding could theoretically double or triple throughput. The assistant correctly identified that the marginal return on further SGLang tuning was lower than the potential return on EAGLE-3 testing. However, this decision rested on an implicit assumption that EAGLE-3 would work on SGLang. The assistant had already spent significant effort getting EAGLE-3 to work on vLLM (segments 20-23), only to discover a ~15% acceptance rate that made it counterproductive. The pivot to SGLang was motivated by the hope that SGLang's implementation might yield better results. But the assistant had not yet verified that SGLang's EAGLE-3 support would work with the Kimi-K2.5 architecture at all.

Assumptions and Blind Spots

The message reveals several assumptions worth examining:

Assumption 1: The attention backend is the main cause of the single-stream gap. The assistant attributes the 23% gap to "triton vs flashinfer" without evidence. In reality, the gap could stem from many factors: different CUDA graph implementations, different memory management strategies, different scheduling policies, or different NCCL configurations. The assistant's assumption is reasonable as a first-order hypothesis but remains untested at this point.

Assumption 2: EAGLE-3 will work on SGLang. The assistant had already experienced EAGLE-3 failure on vLLM due to low acceptance rates. The pivot to SGLang implicitly assumes that SGLang's EAGLE-3 implementation will either (a) work correctly with the Kimi-K2.5 architecture, (b) achieve higher acceptance rates, or (c) handle the overhead more efficiently. None of these had been verified.

Assumption 3: The benchmark numbers are accurate and representative. The benchmark used three prompts, each generating 256 tokens, with non-streaming requests. This is a reasonable methodology but has limitations: it doesn't measure time-to-first-token (TTFT) separately, it uses fixed-length outputs, and it doesn't test variable-length prompts or interleaved prefill/decode phases that real workloads would exhibit.

Input Knowledge Required

To fully understand this message, one needs:

Output Knowledge Created

This message produces several concrete outputs:

  1. A validated benchmark methodology: The three-prompt, 256-token, non-streaming benchmark becomes the standard for all subsequent comparisons. The assistant reuses this exact methodology in later messages, creating a consistent measurement framework.
  2. A decision record: The explicit choice to prioritize EAGLE-3 over single-stream optimization is documented, providing traceability for why subsequent work proceeded in a particular direction.
  3. A performance baseline for SGLang with CUDA graphs: The 63.6 tok/s single-stream and 2,370 tok/s peak numbers become the reference point for all future SGLang tuning. When the assistant later applies NCCL environment variables and flashinfer backend, these numbers are the benchmark to beat.
  4. A todo list update: The message includes a todowrite block showing that "Benchmark SGLang base serving performance" is now completed and "Test SGLang + AQ-MedAI EAGLE-3 drafter" is in progress. This structured task tracking is essential for maintaining coherence across a long session.

The Thinking Process

The message's reasoning reveals a methodical engineering mind at work. The assistant:

  1. Observes the data: The benchmark results are presented clearly, with both single-stream and peak throughput metrics.
  2. Interprets the data: The 63.6 tok/s result is contextualized against the vLLM baseline (77% of vLLM) and the no-CUDA-graph baseline (2.4x improvement).
  3. Generates a hypothesis: The gap is attributed to the attention backend difference.
  4. Makes a decision: Despite the gap, EAGLE-3 is prioritized because it offers higher potential impact.
  5. Executes the decision: The server is killed and the next experiment is launched. This pattern—observe, interpret, hypothesize, decide, execute—is the hallmark of effective experimental science, whether in a physics lab or an ML engineering session. The assistant's ability to resist the temptation of premature optimization (chasing the last 23% of single-stream performance) and stay focused on the primary objective (EAGLE-3) demonstrates mature engineering judgment.

Conclusion

Message [msg 3175] is a study in how a single data point can reshape an entire investigation. The benchmark results transformed the assistant's understanding of SGLang's capabilities, validated the importance of CUDA graphs, and provided the confidence to pivot to the core mission of speculative decoding. It is a reminder that in complex engineering work, the most valuable output is often not a perfect result but a clear enough picture to make the next decision with confidence.