The Pivot Point: Benchmarking SGLang on Blackwell and the CUDA Graph Revelation

Subject Message (msg id=3169):

>

Interesting results. SGLang is significantly different from the vLLM baseline:

>

| Metric | vLLM | SGLang | |--------|------|--------| | Single-stream | 82.5 tok/s | 27 tok/s | | Peak throughput | 1,536 tok/s (C=128) | 2,216 tok/s (C=128) |

>

Single-stream is 3x slower (27 vs 82.5 tok/s), but peak throughput is 44% higher (2,216 vs 1,536 tok/s). The low single-stream speed is suspicious — CUDA graphs are disabled (--disable-cuda-graph), which explains the terrible single-request latency. Let me kill this server and restart with CUDA graphs enabled, then also try with the EAGLE-3 drafter.

>

``bash ssh root@10.1.230.174 'kill -9 50299; sleep 2; ps aux | grep -E "python3|sglang" | grep -v grep | awk "{print \$2}" | xargs kill -9 2>/dev/null; sleep 3; kill -9 $(fuser /dev/nvidia* 2>/dev/null | tr " " "\n" | sort -u | grep -v "^$") 2>/dev/null; sleep 2; nvidia-smi --query-gpu=index,memory.used --format=csv,noheader' 0, 0 MiB 1, 0 MiB 2, 0 MiB 3, 0 MiB 4, 0 MiB 5, 0 MiB 6, 0 MiB 7, 0 MiB ``

In the middle of a marathon coding session to deploy and optimize the Kimi-K2.5 large language model across eight NVIDIA RTX PRO 6000 Blackwell GPUs, a single message ([msg 3169]) marks a critical inflection point. The assistant has just received the results of a comprehensive benchmark comparing SGLang's base performance against a previously established vLLM baseline, and the numbers demand an immediate strategic pivot. This message is not merely a status update—it is a moment of diagnosis, decision, and redirection that fundamentally reshapes the trajectory of the optimization effort.

The Context: From Deadlock Diagnosis to Performance Reality

To understand the weight of this message, one must appreciate what preceded it. The assistant had spent considerable effort debugging what appeared to be a catastrophic SGLang server hang on the SM120 Blackwell architecture. Multiple rounds of strace analysis, process inspection, and code review ([msg 3161], [msg 3162]) had painted a picture of a deadlocked server—all 131 threads sleeping on futex waits, port 8000 unresponsive, GPU utilization at zero. The assistant dispatched a subagent task to trace the SGLang initialization sequence, fully expecting to find a race condition or NCCL deadlock unique to the Blackwell GPUs.

Then came the revelation ([msg 3164], [msg 3165]): the server was not hung at all. It had simply taken 313 seconds—over five minutes—to load and dequantize the 547GB INT4 model checkpoint. The log file contained the triumphant message "The server is fired up and ready to roll!" The assistant had misdiagnosed slow weight loading as a deadlock, a mistake born from the assumption that the earlier, faster loading times (3 seconds in a previous run) were the norm. The reality was that the quantized model required substantial dequantization work during loading, and the debug-level logging had obscured progress indicators.

With the server confirmed operational, the assistant pivoted immediately to benchmarking ([msg 3166], [msg 3167]), writing and deploying a comprehensive Python benchmark script that measured both single-stream latency and multi-concurrent throughput across concurrency levels from 1 to 128. The script used non-streaming requests for accurate token counts and included a warm-up phase to ensure stable measurements. This was not a casual test—it was a systematic performance characterization designed to produce numbers directly comparable to the vLLM baseline established in earlier segments of the session.

The Numbers That Changed Everything

The benchmark results, which arrive at the opening of message [msg 3169], paint a stark and surprising picture:

| Metric | vLLM | SGLang (no CUDA graphs) | |--------|------|------------------------| | Single-stream throughput | 82.5 tok/s | 27 tok/s | | Peak throughput (C=128) | 1,536 tok/s | 2,216 tok/s |

These numbers are immediately revealing. SGLang's single-stream performance is abysmal—roughly one-third of vLLM's. A user sending a single request would wait three times longer for their response. Yet paradoxically, under heavy load with 128 concurrent requests, SGLang outperforms vLLM by 44%, delivering 2,216 tokens per second versus vLLM's 1,536.

This divergence is the key diagnostic signal. The assistant immediately recognizes its cause: the server was launched with --disable-cuda-graph, a flag that had been added during the earlier debugging effort to rule out CUDA graph capture as a source of the apparent hang. CUDA graphs are a critical optimization for single-stream latency in LLM serving—they capture the entire sequence of GPU kernel launches into a single executable graph, eliminating kernel launch overhead and allowing the GPU to pipeline operations more efficiently. Without them, each token generation step pays the full cost of launching hundreds of individual CUDA kernels, which is devastating for latency-sensitive single-request scenarios.

The peak throughput advantage, however, tells a different story. Under high concurrency, the GPU is kept busy with many requests in flight simultaneously, and the kernel launch overhead is amortized across multiple streams. SGLang's higher peak throughput suggests that its attention backend (triton) or other architectural choices may be more efficient than vLLM's when the GPU is fully saturated—a finding that hints at SGLang's potential if the single-stream gap can be closed.

The Reasoning Process: Diagnosis and Decision

The assistant's reasoning in this message is a model of disciplined engineering analysis. The thought process unfolds in three clear stages:

First, observation and comparison. The assistant lays out the numbers side by side, immediately noting the 3x single-stream deficit. The phrase "Interesting results" signals that these numbers are unexpected and demand explanation. The comparison is not just descriptive—it is evaluative, measuring SGLang against the established vLLM baseline that represents the current best-known performance on this hardware.

Second, causal attribution. The assistant identifies the root cause with precision: "CUDA graphs are disabled (--disable-cuda-graph), which explains the terrible single-request latency." This is not a guess—it is a deduction grounded in understanding of how CUDA graphs work and what the --disable-cuda-graph flag does. The assistant knows that without CUDA graphs, every token step requires launching individual kernels, and on a model of this scale (547GB across 8 GPUs with tensor parallelism), the kernel launch overhead dominates per-token latency.

Third, decisive action. The conclusion is immediate and unambiguous: "Let me kill this server and restart with CUDA graphs enabled." There is no hemming and hawing, no request for user approval, no exploration of alternative hypotheses. The assistant has enough information to act, and it does so in the very next breath. The kill command follows directly, cleaning up all GPU processes and verifying that memory is fully freed before proceeding.

Assumptions and Their Validation

The message rests on several key assumptions, most of which prove correct. The primary assumption is that enabling CUDA graphs will dramatically improve single-stream throughput, potentially closing the gap with vLLM. This assumption is validated in the very next benchmark run ([msg 3174], [msg 3175]), where SGLang with CUDA graphs achieves 63.6 tok/s single-stream—more than double the 27 tok/s without graphs, and reaching 77% of vLLM's 82.5 tok/s. The peak throughput also improves from 2,216 to 2,370 tok/s.

A secondary assumption is that the comparison with vLLM is fair and meaningful. The vLLM baseline was established in earlier segments of the session under similar conditions (same model, same hardware, same TP=8 configuration), making the comparison valid. The assistant implicitly assumes that the vLLM numbers were obtained with CUDA graphs enabled (which they were, as vLLM enables them by default), so the comparison is really between "vLLM with CUDA graphs" and "SGLang without CUDA graphs"—an apples-to-oranges comparison that the assistant correctly identifies and corrects.

There is also an assumption embedded in the decision to kill the server rather than hot-swap the flag: that SGLang cannot change the CUDA graph setting at runtime. This is correct—CUDA graph capture happens during server initialization and requires a restart. The assistant's aggressive cleanup (killing all python3 processes, using fuser to release GPU resources) reflects an understanding that SGLang's distributed processes may not clean up cleanly on their own.

The Knowledge Boundary: What This Message Requires and Creates

To fully understand this message, a reader needs considerable background knowledge. They must understand what CUDA graphs are and why they matter for LLM inference latency. They need familiarity with the concept of tensor parallelism (TP=8) and how it distributes a model across multiple GPUs. They should know the difference between single-stream latency (time-to-first-token and tokens-per-second for one user) and peak throughput (total tokens generated per second under maximum load). They need context about the specific hardware (8x RTX PRO 6000 Blackwell, 97887 MiB each) and the model (Kimi-K2.5, 547GB INT4 quantized). And they must understand the history of the session—that --disable-cuda-graph was a debugging artifact from the earlier hang investigation.

The message creates substantial new knowledge. It establishes the first quantitative performance characterization of SGLang on Blackwell GPUs with this specific model. It identifies the CUDA graph dependency as the primary lever for single-stream optimization. It produces a clear benchmark methodology (the Python script) that can be reused for future comparisons. And it generates a decision point—the restart with CUDA graphs—that will determine whether SGLang can serve as a viable alternative to vLLM for this deployment.

A Mistake Redeemed

There is an interesting irony in this message. The assistant's earlier mistake—misdiagnosing slow weight loading as a deadlock and adding --disable-cuda-graph as a debugging measure—directly caused the poor benchmark results it now observes. The --disable-cuda-graph flag, added to eliminate a variable during deadlock investigation, was never removed when the server was restarted for benchmarking. The assistant does not explicitly acknowledge this error chain in the message, but the implication is clear: the debugging process introduced an artifact that distorted the performance picture, and now that artifact must be eliminated.

This is a common pattern in complex systems debugging: the very tools and flags used to diagnose a problem can alter the system's behavior in ways that create new problems or mask the original one. The assistant's ability to recognize the artifact and correct course is a testament to its understanding of the system's performance characteristics.

The Broader Trajectory

Message [msg 3169] sits at a crucial juncture in the larger narrative of segment 24. The session has been pursuing two parallel tracks: getting SGLang operational on Blackwell (which was just achieved after the deadlock false alarm) and testing EAGLE-3 speculative decoding to improve throughput. The benchmark results here establish the baseline against which all subsequent EAGLE-3 experiments will be measured. Without this baseline, it would be impossible to determine whether speculative decoding provides any benefit—a question that will dominate the remainder of the segment.

The decision to restart with CUDA graphs also resets the clock: the server will take another 5-10 minutes to load weights and capture graphs. This is a significant time investment, but the assistant judges it worthwhile given the 3x single-stream deficit. The alternative—proceeding with EAGLE-3 testing on a crippled server—would produce misleading results and waste even more time.

In the end, this message exemplifies the kind of disciplined, data-driven engineering that characterizes the best ML infrastructure work. It takes raw numbers, interprets them through the lens of system architecture, identifies the root cause, and acts decisively. The CUDA graph revelation may seem like a small thing—a single flag change—but in the context of a 547GB model running on eight cutting-edge GPUs, it is the difference between a system that feels sluggish and one that competes with the state of the art.