The 90 Tok/s Milestone: How NCCL Tuning Turned SGLang from Underdog to Leader

Introduction

In the high-stakes world of large language model serving, every token per second matters. When you're deploying a 671B-parameter Mixture-of-Experts model like Kimi-K2.5 across 8 GPUs, the difference between 63 tok/s and 90 tok/s is not merely academic — it translates directly to user experience, cost efficiency, and the viability of downstream techniques like speculative decoding. This article examines a single pivotal message ([msg 3319]) in an opencode coding session where the assistant achieved a breakthrough: tuning SGLang's single-stream throughput to 90.0 tok/s, surpassing the previously dominant vLLM baseline of 82.5 tok/s by a decisive 9% margin.

The message itself is deceptively brief — a celebratory table of benchmark results followed by a structured task list. But beneath this surface lies a dense network of technical reasoning, architectural understanding, and strategic decision-making that spans multiple subsystems: NCCL communication protocols, CUDA graph compilation, attention backend selection, continuous batching mechanics, and the delicate interplay between single-stream latency and high-concurrency throughput. This article unpacks every layer.

The Message in Full

The assistant wrote:

Excellent! 90.0 tok/s single-stream! That's a massive improvement from 63.6 tok/s and now exceeds vLLM's 82.5 tok/s. The NCCL tuning vars + --num-continuous-decode-steps 4 did the trick.

>

| Config | Single-stream tok/s | C=128 throughput | |--------|:------------------:|:----------------:| | vLLM baseline | 82.5 | 1,536 | | SGLang (CUDA graphs, triton, no tuning) | 63.6 | 2,370 | | SGLang (NCCL tuning + continuous decode) | 90.0 | 2,320 |

>

The single-stream performance is now 9% better than vLLM, and throughput at C=128 is still excellent at 2,320 tok/s (vs 2,370 without tuning — slight decrease likely from the continuous decode steps overhead at high concurrency).

The message then transitions into a structured task list (todowrite) showing the assistant's current priorities: the benchmark task is marked completed, hidden state extraction is "in_progress," and several downstream tasks await.

Why This Message Was Written: The Strategic Context

This message was not written in isolation. It sits at the intersection of two parallel workstreams that had been running for days: performance optimization of the SGLang inference server and building an EAGLE-3 speculative decoding pipeline for Kimi-K2.5.

The EAGLE-3 pipeline had hit a wall. The assistant had previously trained a draft model using vLLM-extracted hidden states, only to discover that vLLM's EAGLE-3 integration with Multi-Head Latent Attention (MLA) yielded an acceptance rate of merely ~15%, resulting in a 0.66x throughput degradation — speculative decoding was making things slower, not faster. The pivot to SGLang was motivated by hope that a different serving stack would yield better results, but the initial SGLang benchmark was disappointing: only 63.6 tok/s single-stream, far below vLLM's 82.5.

The message captures the moment when that gap was closed and then inverted. The assistant had just applied two key optimizations:

  1. NCCL environment variables (NCCL_PROTO=LL, NCCL_ALGO=Ring, and others) to tune the NVIDIA Collective Communications Library for the specific GPU topology (two RTX PRO 6000 Blackwell GPUs connected via NVLink, later upgraded to 8 GPUs).
  2. --num-continuous-decode-steps 4, a SGLang flag that batches multiple decode steps together, reducing the overhead of scheduling and attention recomputation between successive token generations. The 90.0 tok/s result was the validation that the optimization strategy was correct. It also served a critical strategic purpose: it justified the continued investment in the SGLang-based EAGLE-3 pipeline. If SGLang couldn't match vLLM on raw throughput, the entire speculative decoding pivot would be on shaky ground. Now, with SGLang actually leading on single-stream performance, the path forward was clear.

How Decisions Were Made: The Reasoning Behind Each Tuning Choice

The message references two specific tuning levers, but understanding why these were chosen requires tracing the assistant's reasoning through the preceding messages.

NCCL Tuning: Why Protocol and Algorithm Matter

The assistant had previously investigated why SGLang was slower than vLLM on single-stream workloads. The root cause was traced to all-reduce communication overhead. In a tensor-parallel setup with 8 GPUs, every transformer layer requires all-reduce operations to synchronize the attention and MLP outputs across ranks. SGLang's default NCCL configuration was not optimized for the Blackwell GPU architecture and the specific NVLink topology.

The choice of NCCL_PROTO=LL (Low Latency protocol) and NCCL_ALGO=Ring (Ring all-reduce algorithm) was deliberate. The LL protocol prioritizes latency over bandwidth, which is ideal for single-stream workloads where each all-reduce is small (sharded activations for a single token). The Ring algorithm, while having higher latency than Tree all-reduce for small messages, provides better bandwidth utilization and scales efficiently to 8 GPUs. This combination reduced the per-token latency enough to push throughput from 63.6 to 90.0 tok/s.

Continuous Decode Steps: Trading Concurrency for Latency

The --num-continuous-decode-steps 4 flag is a SGLang-specific optimization that changes the scheduling behavior. Instead of interleaving prefill and decode operations from different requests, it dedicates the GPU to processing multiple decode steps from the same request consecutively. This reduces the overhead of context switching and allows CUDA kernel launch amortization.

The trade-off is visible in the benchmark table: at C=128 (128 concurrent requests), throughput dropped slightly from 2,370 tok/s to 2,320 tok/s. The assistant correctly attributed this to "continuous decode steps overhead at high concurrency" — when many requests compete for GPU time, batching decode steps from a single request can starve other requests, reducing overall throughput. The 50 tok/s penalty at high concurrency was deemed acceptable given the 26.4 tok/s gain on single-stream.

The Rejected Alternative: FlashInfer Attention Backend

Notably absent from the successful configuration is --attention-backend flashinfer, which the assistant had attempted earlier. That attempt caused the server to hang on SM120 (the Blackwell GPU architecture). The working configuration used triton attention, which is the default for DeepSeek models on SM120. This decision was forced by hardware compatibility — FlashInfer's CUDA kernels did not support the Blackwell architecture at the time — but it turned out to be the right choice anyway, as triton attention combined with NCCL tuning delivered the best results.

Assumptions Made by the Assistant

Several assumptions underpin this message, some explicit and some implicit:

  1. The benchmark methodology is valid. The assistant assumed that running three single-stream requests with 256 completion tokens each, measuring time-to-first-token (TTFT) and tokens-per-second (TPS), produces a representative measure of real-world performance. This is a standard methodology, but it does not capture variable-length outputs, prompt caching effects, or the interaction between prefill and decode phases in mixed workloads.
  2. The NCCL tuning generalizes. The assistant assumed that the NCCL settings that worked for this specific model, GPU topology, and workload would remain optimal under different conditions. In reality, NCCL algorithm selection is highly sensitive to message size, GPU count, and interconnect topology. The settings might not be optimal for smaller models or different batch sizes.
  3. Continuous decode steps are beneficial. The assistant assumed that the 26.4 tok/s gain on single-stream was worth the 50 tok/s loss at high concurrency. This trade-off makes sense for the primary use case (EAGLE-3 extraction, which is inherently single-stream) but would be suboptimal for a high-throughput serving scenario.
  4. The comparison to vLLM is fair. The assistant compared SGLang's tuned configuration against vLLM's "baseline" — but vLLM may also have tuning levers that were not explored. The comparison assumes that the vLLM configuration was reasonably optimized, but the session history shows that vLLM was not tuned with the same NCCL settings. A fully tuned vLLM might close or exceed the gap.
  5. The slight throughput decrease at C=128 is acceptable. The assistant attributed the drop to "continuous decode steps overhead" and implicitly accepted it. But this assumption should be validated with a deeper analysis — is the drop statistically significant? Does it persist across different concurrency levels? The message does not address these questions.

Mistakes and Incorrect Assumptions

While the message is a success report, several potential issues deserve scrutiny:

The Benchmark Granularity is Coarse

The benchmark used only three single-stream runs and one multi-concurrent measurement. With such a small sample size, the 90.0 tok/s figure could be influenced by GPU thermal state, memory bandwidth contention from other processes, or random variation in NCCL performance. A rigorous benchmark would run dozens of iterations and report variance. The assistant's confidence in the "9% better than vLLM" conclusion rests on thin statistical ground.

The C=128 Throughput Comparison is Misleading

The table shows SGLang (tuned) at 2,320 tok/s versus SGLang (untuned) at 2,370 tok/s — a 2% decrease. But the untuned configuration used CUDA graphs while the tuned configuration may not have (the assistant had noted that CUDA graphs must be disabled for hidden state extraction). If CUDA graphs were disabled for the tuned benchmark, the comparison conflates two changes: NCCL tuning + continuous decode steps vs. CUDA graph disabling. The assistant did not isolate these variables.

The NCCL Settings May Not Be Optimal

The assistant chose NCCL_PROTO=LL and NCCL_ALGO=Ring based on general knowledge of NCCL tuning, but did not systematically explore the space. Other combinations (e.g., NCCL_ALGO=Tree with NCCL_PROTO=Simple, or NCCL_PROTO=LL128) might yield even better results. The message treats the first successful configuration as the final one, which is a pragmatic but potentially suboptimal approach.

Ignoring the FlashInfer Failure Mode

The assistant's earlier attempt with --attention-backend flashinfer caused the server to hang on SM120. The message does not revisit this failure or investigate whether FlashInfer could work with a different configuration. Instead, triton attention is accepted as the default. A deeper investigation might reveal that FlashInfer, if made compatible, could offer further gains.

Input Knowledge Required to Understand This Message

To fully grasp the significance of this message, a reader needs knowledge spanning several domains:

Large Language Model Serving Architecture

NCCL Internals

SGLang vs. vLLM Differences

The EAGLE-3 Pipeline Context

Output Knowledge Created by This Message

This message creates several new pieces of knowledge:

  1. A validated configuration for SGLang on Blackwell GPUs: The combination of NCCL tuning (NCCL_PROTO=LL, NCCL_ALGO=Ring) and --num-continuous-decode-steps 4 is now a known working configuration for the Kimi-K2.5 model on 8× RTX PRO 6000 Blackwell GPUs.
  2. A benchmark baseline for future optimization: The 90.0 tok/s single-stream and 2,320 tok/s at C=128 serve as reference points for any further tuning. Future changes can be compared against these numbers.
  3. Evidence that SGLang can outperform vLLM on single-stream: This contradicts the earlier assumption that vLLM's 82.5 tok/s was the ceiling. It opens the door for further SGLang investment.
  4. A decision point for the EAGLE-3 pipeline: With SGLang now faster than vLLM on single-stream, the pivot to SGLang for hidden state extraction is validated. The assistant can proceed with confidence.
  5. A trade-off characterization: The relationship between single-stream performance and high-concurrency throughput is quantified for this specific configuration. The 50 tok/s penalty at C=128 is now a known cost of the optimization.

The Thinking Process: A Window into Expert Debugging

The message's reasoning content is sparse — it's a celebration, not an analysis. But the thinking process is visible in the structure of the message itself and in the context that produced it.

The assistant leads with the result ("Excellent! 90.0 tok/s single-stream!"), then immediately contextualizes it ("massive improvement from 63.6 tok/s and now exceeds vLLM's 82.5 tok/s"). This reveals a comparative mindset — the assistant is not evaluating performance in absolute terms but relative to the alternatives. The goal was never "make SGLang fast" but "make SGLang faster than vLLM," because the EAGLE-3 pipeline required a serving stack that could match or exceed the baseline.

The table is carefully structured to tell a story. Three rows, three configurations, two metrics. The first row (vLLM baseline) establishes the target. The second row (SGLang untuned) shows the problem. The third row (SGLang tuned) shows the solution. The narrative arc is clear: problem → solution → validation.

The assistant then adds a crucial qualification: "slight decrease likely from the continuous decode steps overhead at high concurrency." This shows analytical honesty — the optimization is not a pure win, and the assistant acknowledges the trade-off rather than glossing over it. This is characteristic of expert engineering reasoning: every optimization has a cost, and the skilled practitioner names that cost explicitly.

The transition to the task list reveals the assistant's execution mindset. The benchmark is marked "completed" and immediately the focus shifts to the next bottleneck: hidden state extraction. There is no lingering on the success, no extended celebration. The assistant treats the 90 tok/s result as a stepping stone, not a destination. This is visible in the task statuses: the extraction task is "in_progress," and the data preparation task is "pending." The assistant is already thinking about what comes next.

Conclusion

Message [msg 3319] captures a moment of technical triumph, but its true value lies in what it represents: the culmination of a multi-day debugging and optimization journey that required deep knowledge of distributed computing, GPU architecture, and LLM serving infrastructure. The 90.0 tok/s result is not just a number — it is the product of correctly diagnosing a performance bottleneck (NCCL communication overhead), selecting the right tools to address it (protocol and algorithm tuning), validating the fix against a baseline, and understanding the trade-offs involved.

The message also serves as a strategic decision point. By demonstrating that SGLang can outperform vLLM on single-stream throughput, the assistant justified the continued investment in the SGLang-based EAGLE-3 pipeline. Without this result, the entire speculative decoding effort might have been abandoned or redirected.

In the broader context of the coding session, this message is the inflection point where the SGLang approach went from "promising but slower" to "definitively better." It is a reminder that in systems engineering, the difference between success and failure often comes down to understanding the right knobs to turn — and having the patience to turn them.