The Moment SGLang Came Alive: Functional Verification on 8x Blackwell GPUs

In the long arc of deploying a massive 547GB quantized language model across eight NVIDIA RTX PRO 6000 Blackwell GPUs, there is a single message that marks the transition from debugging despair to productive benchmarking. Message 3166 is that pivot point. After hours of chasing what appeared to be a deadlocked SGLang server — all 131 threads stuck on futex waits, port 8000 unresponsive, GPU utilization at 0% — the assistant had just discovered the truth: the server was never hung. It simply took five minutes to load the model. Now, with the server confirmed running and ready, the assistant takes its first measured step forward.

The Message in Full

The assistant writes:

Let me first do a quick functional test, then run a proper single-stream latency benchmark to compare with the vLLM baseline (82.5 tok/s).

It then executes a curl command against the freshly available SGLang server:

ssh root@10.1.230.174 'curl -s http://localhost:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d "{\"model\": \"/shared/kimi-k2.5-int4\", \"messages\": [{\"role\": \"user\", \"content\": \"What is 2+2? Reply in one word.\"}], \"max_tokens\": 32, \"temperature\": 0}" \
  2>&1 | python3 -m json.tool'

The response confirms the server is operational:

{
    "id": "6c820f3322274b6cac4655aa551d9bab",
    "object": "chat.completion",
    "created": 1771801253,
    "model": "/shared/kimi-k2.5-int4",
    "choices": [
        {
            "index": 0,
            "message": {
                "role": "assistant",
                "content": " The user is asking for the answer to \"2+2\" and wants me to reply in one word.\n\n 2+2 = 4\n\n The word",
                "reasoning_content": null,
                "tool_calls": null
            },
           ...

The model responds — albeit with excessive verbosity for a "one word" instruction — confirming that the SGLang inference stack is fully functional on the SM120 architecture.

Why This Message Was Written: The Context of Discovery

To understand why this message matters, one must appreciate what preceded it. The assistant had been trying to get SGLang running on Blackwell GPUs (SM120 compute capability) for several rounds. Earlier attempts had ended in what looked like a complete deadlock: the server process (PID 50299) was sleeping, all 131 threads were blocked on futex system calls, and the log file ended abruptly after weight loading completed. Every diagnostic tool — strace, nvidia-smi, ss, /proc/status — painted the same picture: a server that loaded weights and then vanished into silence.

The breakthrough came in [msg 3164], when the assistant re-examined the log file more carefully and discovered it contained 312 lines, not just the 80 it had initially seen. Buried in those lines were messages from all 8 tensor-parallel ranks confirming "Load weight end," memory pool allocation, and even a completed prefill batch. The server had been running for hours — it just took 313 seconds to dequantize and load the 547GB model. The final log line, "[2026-02-22 22:59:08] The server is fired up and ready to roll!", had been sitting there unnoticed.

Message 3166 is the immediate consequence of that discovery. The assistant's first instinct is not to celebrate but to verify. It needs to confirm, with a live API call, that the server actually serves requests. This is a classic engineering discipline: never trust a log message alone; always test the interface.

The Thinking Process: Methodical Verification Before Optimization

The assistant's reasoning in this message reveals a clear two-phase strategy. Phase one is functional testing: send a trivial prompt, verify the response format, check that the model generates coherent text. Phase two is performance benchmarking: compare single-stream latency against the established vLLM baseline of 82.5 tokens per second.

The choice of prompt is telling. "What is 2+2? Reply in one word." is the simplest possible test — minimal prompt processing, minimal generation (32 max tokens), deterministic (temperature 0). It's designed to produce a fast, reproducible result that isolates server availability from model quality. The assistant is not testing whether the model is smart; it's testing whether the pipeline works.

The response reveals something interesting: the model generates a verbose internal monologue ("The user is asking for the answer to '2+2' and wants me to reply in one word...") before producing the answer. This is characteristic of Kimi-K2.5's architecture, which interleaves reasoning tokens with output tokens. The assistant doesn't comment on this in the message itself, but it will become relevant later when benchmarking reveals that SGLang's token counting differs from vLLM's.

Assumptions Embedded in the Message

Several assumptions underpin this message, some correct and some that will prove incorrect.

The assistant assumes the server is fully initialized and ready for benchmarking. This is correct — the log confirms warmup completed, and the curl request returns successfully.

The assistant assumes that a single-stream benchmark against vLLM's 82.5 tok/s baseline is the right comparison. This is reasonable but incomplete: SGLang's architecture differs from vLLM's in ways that affect single-stream vs. batch performance differently. The assistant implicitly assumes that if SGLang can match or exceed vLLM on single-stream latency, it will be a viable replacement.

The assistant assumes that the current server configuration (with --disable-cuda-graph still in effect from the earlier debug attempt) is the baseline configuration. This turns out to be a critical oversight. The server was launched with CUDA graphs disabled to aid debugging, and this flag dramatically degrades single-stream performance. The benchmark that follows in [msg 3168] will show only 27 tok/s — three times slower than vLLM. The assistant will then need to kill the server, restart without the debug flag, and re-benchmark.

There is also an implicit assumption that the model's verbose response style doesn't affect benchmarking. In fact, because Kimi-K2.5 generates reasoning tokens inline, the token count reported by the API may differ from the token count expected by the benchmark script, potentially skewing throughput calculations.

Input Knowledge Required to Understand This Message

To fully grasp what is happening here, the reader needs several pieces of context:

The hardware setup: Eight NVIDIA RTX PRO 6000 Blackwell GPUs (each with 96GB of HBM4 memory) connected via PCIe, not NVLink. This means inter-GPU communication goes through the PCIe bus, making AllReduce a critical bottleneck.

The model: Kimi-K2.5 in INT4 quantization, approximately 547GB total. This is a Mixture-of-Experts architecture based on DeepSeek-V2, using Multi-head Latent Attention (MLA). The model is loaded with tensor parallelism across 8 GPUs (tp-size 8).

The software stack: SGLang (a nightly build) versus vLLM 0.16. Both are competing inference engines for large language models. The assistant has already established a vLLM baseline of 82.5 tok/s single-stream and 1,536 tok/s peak throughput.

The debug history: The server was launched with --disable-cuda-graph --log-level debug to investigate a suspected hang. This flag remains active in the current server instance, which will significantly impact the upcoming benchmark.

The EAGLE-3 context: The broader goal is not just to run SGLang but to use it for EAGLE-3 speculative decoding. The assistant has already trained an EAGLE-3 drafter for Kimi-K2.5 and needs SGLang's EAGLE-3 support to test it. This functional test is a prerequisite for that larger effort.

Output Knowledge Created by This Message

This message produces several concrete results:

  1. Confirmation that SGLang serves requests on SM120: The curl response proves that the full inference pipeline — model loading, tensor parallelism, attention computation, token generation, and API serving — works correctly on Blackwell GPUs. This is non-trivial; SGLang had to be patched and built from source to support SM120.
  2. A baseline for comparison: The assistant now has a working SGLang instance that it can benchmark against vLLM. The response time and token generation pattern provide initial data points for performance analysis.
  3. A plan of action: The message explicitly states the next step: "run a proper single-stream latency benchmark." This creates a clear success criterion and drives the subsequent messages.
  4. Validation of the debugging effort: The hours spent diagnosing the "hang" were not wasted. The server was working all along, and this message proves it.

The Broader Significance

Message 3166 is a classic example of the "it was working the whole time" genre of debugging stories. The assistant spent multiple rounds investigating a deadlock that didn't exist, running strace on sleeping threads, analyzing futex wait patterns, and reading SGLang's initialization code. The resolution came not from deeper technical analysis but from re-reading the log file more carefully — a humbling reminder that the first diagnostic pass is often incomplete.

But the message is also significant for what it enables next. With SGLang confirmed working, the assistant can now benchmark performance, test EAGLE-3 speculative decoding, and compare the two inference engines head-to-head on the Blackwell architecture. The functional test is the gateway to all subsequent work.

The choice to benchmark single-stream latency against vLLM's 82.5 tok/s is also strategically important. Single-stream latency is the hardest metric for a batched inference server to optimize, and it's the metric most visible to end users. If SGLang can match vLLM on single-stream while exceeding it on peak throughput (as later benchmarks in [msg 3168] and [msg 3169] will show), it becomes a compelling alternative.

Conclusion

Message 3166 is a small message with large consequences. In a single curl command, it transforms the conversation from debugging to benchmarking, from uncertainty to measurement. The assistant's disciplined approach — verify before optimize, test the interface before trusting the log — exemplifies good engineering practice. And the assumptions embedded in the message, particularly about the CUDA graph configuration, set up the next round of discovery: that SGLang with CUDA graphs disabled is three times slower than vLLM, but with CUDA graphs enabled, it will achieve 63.6 tok/s single-stream and an impressive 2,370 tok/s peak throughput. The functional test is the first domino in a chain that will ultimately lead to a complete performance characterization of SGLang on Blackwell GPUs.