The EAGLE-3 Verdict and the Strategic Pivot: SGLang Performance Tuning on 8× Blackwell GPUs

Introduction

Segment 24 of this opencode session represents one of the most consequential inflection points in the entire Kimi-K2.5 deployment campaign. Over the course of nearly 100 messages, the assistant navigates a phantom server hang, benchmarks base SGLang against vLLM, patches the Kimi-K2.5 model wrapper for EAGLE-3 compatibility, tests two separate draft models (both of which fail to deliver speedup), discovers a subtle SM120 detection bug in SGLang's attention backend selection, and ultimately pivots to a two-track plan of NCCL tuning and SGLang-based hidden state extraction for a retrained drafter. The narrative arc moves from speculative decoding (a promising but ultimately unsuccessful optimization) to performance tuning (a more direct attack on the latency bottleneck), all while maintaining the discipline to accept negative results and redirect effort strategically.

This article synthesizes the key technical findings, debugging methodologies, and engineering decisions that define this segment, drawing on the detailed analysis in the chunk articles [1][2] and the raw conversation data.

Part I: The Phantom Hang — Discovering That SGLang Was Working All Along

The segment opens with the assistant deep in a debugging spiral. SGLang had been launched on the 8× Blackwell GPU system (SM120 architecture) and appeared to be hanging after weight loading. The server process was alive but sleeping, GPU utilization was 0%, port 8000 never opened, and the log file ended abruptly after model loading completed. The assistant had tried multiple diagnostic approaches — checking process state with ps aux, examining GPU memory with nvidia-smi, reading the debug log, and even relaunching with --disable-cuda-graph --log-level debug to isolate the issue.

The breakthrough came when the assistant re-examined the log file more carefully. Buried in the full 312-line log were messages from all 8 tensor-parallel ranks confirming "Load weight end," memory pool allocation, and even a completed prefill batch. The final log line read: "[2026-02-22 22:59:08] The server is fired up and ready to roll!" The server had been running for hours — it just took 313 seconds to dequantize and load the 547GB model. The assistant had been killing the server prematurely, mistaking a long initialization for a deadlock.

This discovery transformed the debugging session into a benchmarking campaign. The assistant immediately ran a functional test via curl, confirming that SGLang served requests correctly on SM120. The response was verbose but coherent — the model was alive.

The lesson here is one of patience and careful log analysis. When dealing with models of this scale — 547GB of INT4-quantized weights across 8 GPUs — initialization times of 5–10 minutes are normal. The assistant's instinct to suspect a hang was reasonable given the silent failure modes common in distributed GPU systems, but the correct response was to wait and read the logs more carefully rather than kill and restart. This pattern — prematurely aborting a slow but correct initialization — is a common pitfall in large-scale ML deployments.

Part II: Benchmarking Base SGLang — A Tale of Two Metrics

With the server confirmed operational, the assistant benchmarked base SGLang performance. The results revealed a fascinating split between two critical metrics. In single-stream mode (one request at a time, measuring latency), SGLang achieved 63.6 tok/s, significantly behind vLLM's 82.5 tok/s baseline. But at high concurrency (C=128 concurrent requests), SGLang dominated with 2,370 tok/s peak throughput — a 54% improvement over vLLM's 1,536 tok/s peak.

This divergence told a clear story about the two engines' architectural strengths. vLLM's more mature CUDA graph capture and scheduling pipeline gave it superior per-request latency. But SGLang's batching and memory management were substantially more efficient under load, allowing it to extract more total throughput from the 8-GPU Blackwell system. The assistant had a clear target: close the single-stream gap while preserving the throughput advantage.

The initial benchmark was conducted with CUDA graphs disabled (a leftover from the debugging configuration). When the assistant restarted with CUDA graphs enabled, single-stream performance jumped from 27 tok/s to 63.6 tok/s, confirming that CUDA graph capture was essential for competitive latency. This is a well-known pattern in ML inference optimization: CUDA graphs eliminate kernel launch overhead by pre-recording the entire execution graph, which is particularly beneficial for the small, frequent operations in autoregressive decoding.

Part III: Patching Kimi-K2.5 for EAGLE-3 — The Three-Line Delegation

With base SGLang benchmarked, the assistant turned to the primary goal: EAGLE-3 speculative decoding. The first attempt to launch SGLang with the AQ-MedAI EAGLE-3 drafter crashed with an AttributeError: KimiK25ForConditionalGeneration didn't have the set_eagle3_layers_to_capture method required by SGLang's speculative decoding framework.

The assistant traced the issue through SGLang's codebase, discovering that the KimiK25ForConditionalGeneration class is a multimodal wrapper around a DeepseekV3ForCausalLM language model (accessible via self.language_model). The inner DeepSeek model already implemented the full EAGLE-3 interface — set_eagle3_layers_to_capture, get_embed_and_head, and set_embed_and_head — but the wrapper didn't delegate to it.

The fix was minimal: three delegation methods that simply forwarded calls to self.language_model. The assistant found the reference pattern in mllama4.py and applied it to kimi_k25.py. After clearing cached .pyc files and restarting, the EAGLE-3 server launched successfully.

This patch is a textbook example of how modern LLM serving frameworks handle model heterogeneity. The Kimi-K2.5 model is built on the DeepSeekV3 architecture but wrapped in a multimodal interface. The EAGLE-3 integration was designed for the base DeepSeek model, and the wrapper class simply hadn't been updated to expose the speculative decoding interface. The fix was straightforward once the assistant understood the class hierarchy.

Part IV: Testing the AQ-MedAI EAGLE-3 Drafter — 42% Acceptance, No Speedup

The first EAGLE-3 test used the pre-trained AQ-MedAI drafter, which had been explicitly tested with SGLang on the earlier Kimi-K2 model. The results were underwhelming. Single-stream throughput was 62.9 tok/s — essentially identical to the base SGLang rate of 63.6 tok/s. At high concurrency, speculative decoding was actually worse: 849 tok/s vs 2,370 tok/s for the base server, because SGLang automatically limited max_running_requests to 48 in speculative mode versus 2048 for base.

The acceptance rate was approximately 42%, meaning only 1.65 out of 4 proposed draft tokens were accepted on average. This was well below the 60–80% typically needed for speculative decoding to break even, considering the overhead of running the draft model and the extra verification step. The assistant diagnosed the root cause: the AQ-MedAI drafter was trained for Kimi-K2, not Kimi-K2.5, and architectural differences between the model versions explained the mediocre acceptance rate.

The max_running_requests limitation was particularly damaging. In base mode, SGLang could batch up to 2,048 concurrent requests, fully utilizing the 8-GPU system's compute capacity. In speculative mode, the limit dropped to 48, meaning the batching efficiency that gave SGLang its throughput advantage was effectively disabled. This is not a bug but a design constraint: speculative decoding requires maintaining separate KV caches for the draft and target models, which increases memory pressure and limits batch sizes.

Part V: Testing the Custom K2.5-Trained EAGLE-3 Drafter — Completely Broken

The assistant then tested the custom drafter that had been painstakingly trained on 10,000 samples of actual Kimi-K2.5 inference data. The results were devastating. Single-stream throughput was 40.8 tok/s — barely two-thirds of the base SGLang performance and less than half of vLLM's 82.5 tok/s. The acceptance logs revealed the truth: an acceptance length of exactly 1.00 and an acceptance rate of 25% (1/4). This meant the drafter was accepting exactly zero draft tokens per step — the "1.00" represented only the original token that was always going to be generated. The drafter was, in the assistant's words, "completely broken" and "essentially random."

The root cause analysis was sophisticated. The assistant identified a likely hidden state mismatch: the training data for the custom drafter was extracted from the INT4 quantized model using vLLM's inference path, but SGLang's internal hidden state extraction mechanism may produce different hidden states than vLLM's. Even small differences in quantization numerics or layer indexing conventions could shift the hidden state distribution enough that the drafter's predictions became useless.

The assistant's investigation into this mismatch is documented in detail in [2]. The key finding was that both frameworks capture mathematically equivalent hidden states — hidden_states + residual after each target layer — but they index them differently. SGLang uses a +1 offset convention where it captures the input to layer N+1 (which equals the output of layer N), while the speculators library captures the output of layer N directly. These are the same values, just indexed differently. This ruled out the simplest explanation and forced the assistant to confront deeper questions about numerical consistency between frameworks.

Part VI: The Verdict on EAGLE-3 — No Benefit on This Hardware

The assistant synthesized the results into a definitive comparison table:

| Config | Single-stream | Peak throughput | Accept rate | |--------|:------------:|:----------------:|:-----------:| | vLLM baseline | 82.5 tok/s | 1,536 tok/s | — | | SGLang base (CUDA graphs) | 63.6 tok/s | 2,370 tok/s | — | | SGLang + AQ-MedAI EAGLE-3 | 62.9 tok/s | 849 tok/s | ~42% | | SGLang + Custom K2.5 EAGLE-3 | 40.8 tok/s | 597 tok/s | 25% (broken) |

The conclusion was stark: EAGLE-3 speculative decoding provided no benefit on this hardware configuration. The base SGLang server with CUDA graphs achieved the best throughput, and the single-stream leader remained vLLM. The assistant killed the speculative decoding server and restored the base SGLang configuration.

This verdict represents a critical inflection point. The assistant had invested significant effort in building the EAGLE-3 training pipeline, generating 10,000 synthetic samples, training the drafter, and debugging the integration. The decision to abandon this approach was not easy, but it was correct. The data clearly showed that speculative decoding was not just failing to improve performance — it was actively degrading it, especially at high concurrency where the max_running_requests limit cut throughput by more than half.

Part VII: The SM120 Detection Bug — A Single Integer with Cascading Consequences

While investigating why SGLang's single-stream performance lagged behind vLLM's, the assistant traced through SGLang's attention backend selection logic and found a critical flaw. SGLang's server_args.py contains a code path for DeepSeekV3 and KimiK25 models that selects the optimized trtllm_mla attention backend on SM100+ GPUs. The condition guarding this selection uses is_sm100_supported(), which checks for compute capability major version 10. But the Blackwell RTX PRO 6000 GPUs have compute capability 12.0 — major version 12. The check returned False, and SGLang silently fell through to the default triton backend.

The bug was a classic off-by-one-major-version error. The SGLang developers had written the code when only SM100 (compute capability 10.0) Blackwell GPUs existed, and didn't anticipate SM120 (compute capability 12.0) having a different major version number. A separate function is_sm120_supported() existed in the codebase but was never consulted in the attention backend selection logic for DeepSeek/KimiK25 models.

The performance implications were significant. The trtllm_mla backend is specifically optimized for Multi-head Latent Attention (MLA) — the attention mechanism used by DeepSeekV3 and KimiK25 models — on NVIDIA's TensorRT-LLM library. On SM100+ hardware, it can leverage hardware-specific optimizations like FP8 tensor cores and improved memory bandwidth. The triton backend, while functional, is a general-purpose fallback that lacks these optimizations. The difference between a hand-tuned MLA kernel and a generic triton kernel could easily account for 10-20% of decode latency — a significant portion of the gap between vLLM's 82.5 tok/s and SGLang's 63.6 tok/s.

Part VIII: The Pivot — NCCL Tuning and the Two-Track Plan

The user redirected the effort with a clear mandate: apply the same NCCL tuning that made vLLM fast to SGLang, and retrain the EAGLE-3 drafter using SGLang-based hidden state extraction. The assistant immediately began executing both tracks in parallel.

The NCCL tuning involved six environment variables from vLLM's systemd service file: NCCL_PROTO=LL, NCCL_ALGO=Ring, NCCL_P2P_LEVEL=SYS, NCCL_MAX_NCHANNELS=16, NCCL_BUFFSIZE=16777216, NCCL_NTHREADS=512. These variables control the NCCL communication library's protocol selection, algorithm choice, P2P level, channel count, buffer size, and thread count. The NCCL_PROTO=LL (Low Latency) protocol and NCCL_ALGO=Ring algorithm were particularly important, as they reduce communication overhead for the small message sizes typical in tensor-parallel inference.

The assistant killed the running server and restarted with these variables set, along with --attention-backend flashinfer, --num-continuous-decode-steps 4, and --disable-custom-all-reduce. The flashinfer attention backend was chosen as an alternative to the unavailable trtllm_mla backend, and --num-continuous-decode-steps 4 enables continuous batching with multiple decode steps per iteration. The --disable-custom-all-reduce flag disables SGLang's custom all-reduce implementation in favor of NCCL's native all-reduce, which may be more efficient with the tuned NCCL settings.

Part IX: Planning the New EAGLE-3 Training Pipeline

For the retraining track, the assistant faced a choice: build a native SGLang-based hidden state extraction pipeline, or reuse the existing vLLM-based extraction with a targeted fix. The SGLang-based approach ran into complications — SGLang's attention layers require a ForwardBatch object for KV cache management that is deeply tied to the server's memory pool and not easily accessible from a standalone script.

The assistant made a pragmatic decision: reuse the existing vLLM extraction pipeline, which already works and achieves 3,165 tok/s throughput, but fix the layer capture convention to match SGLang's expectations. The key insight was that SGLang's EAGLE-3 integration uses a +1 offset in its layers_to_capture list and captures hidden states before the layer forward pass. By adjusting the layer IDs in the extraction script to match this convention, the hidden states should align.

While waiting for the tuned SGLang server to load (a 10-minute process for the 547GB model), the assistant inspected the existing 10K-sample dataset and planned the expansion to 15K samples. The data pipeline was ready — 5K more inference samples needed to be generated, then hidden states re-extracted using the corrected layer IDs, and the drafter retrained.

Part X: The Hidden State Epiphany — Unraveling the +1 Offset

One of the most intellectually satisfying moments in this segment came when the assistant was planning the SGLang-based hidden state extraction pipeline. While examining SGLang's deepseek_v2.py model implementation, the assistant discovered a critical detail about how EAGLE-3 hidden states are captured during inference.

The key finding was that SGLang captures hidden states before the layer runs — the line aux_hidden_states.append(hidden_states + residual) appears before hidden_states, residual = layer(...) in the forward pass. And the layers_to_capture list uses a +1 offset: layers_to_capture = [val + 1 for val in layer_ids]. So when the training pipeline specifies layers [2, 30, 58], SGLang captures at positions [3, 31, 59], but because the capture happens before the layer forward pass, the captured value is actually the output of layers 2, 30, and 58 respectively.

This subtle implementation detail had profound implications. If the training pipeline extracted hidden states at layers [2, 30, 58] using a different method — for example, by running a forward pass and grabbing the output after each layer — those hidden states would not match what SGLang's EAGLE-3 integration actually uses during inference. The mismatch could explain the low acceptance rates observed in earlier testing.

The assistant's "But actually — wait" moment represents a classic debugging epiphany: the connection between two pieces of information that had been sitting separately in context — the +1 offset convention from the code inspection, and the layer IDs from the training pipeline — suddenly clicked into place. This insight directly informed the pragmatic decision to reuse the vLLM extraction pipeline with corrected layer IDs rather than building a complex new SGLang-based extraction system.

Part XI: The Deeper Puzzle — Why the Drafter Failed

Despite the assistant's thorough investigation, the root cause of the 25% acceptance rate remains unresolved at the end of this segment. The layer offset hypothesis has been ruled out — both frameworks capture the same values at the same logical positions. The remaining hypotheses are:

  1. INT4 dequantization divergence: vLLM and SGLang may use different dequantization kernels for the MXFP4 format, producing numerically different hidden states even with the same input and weights.
  2. Attention backend differences: vLLM uses PagedAttention while SGLang uses flashinfer. These different attention implementations can produce slightly different numerical results, especially at the boundaries of the attention computation.
  3. CUDA graph compilation: SGLang uses CUDA graphs to optimize inference, which can change the order of operations and introduce numerical differences compared to vLLM's eager execution.
  4. Residual connection handling: Even though both frameworks capture hidden_states + residual, the internal computation of hidden_states and residual might differ due to different normalization placement or different handling of the residual stream. The pragmatic response — re-extract using SGLang — bypasses the mystery entirely. By using the same framework for both training data extraction and inference, any cross-framework inconsistencies are eliminated. This is the engineering equivalent of "if you can't fix it, work around it."

Conclusion: A Masterclass in Disciplined Engineering

Segment 24 of this opencode session is a masterclass in disciplined engineering under uncertainty. The assistant navigated a phantom hang, benchmarked two inference engines, patched a model wrapper for speculative decoding, tested two draft models (both of which failed), diagnosed a subtle SM120 detection bug, and pivoted to a two-track optimization plan. The willingness to accept negative results — to conclude that EAGLE-3 provided no benefit and move on — is a hallmark of effective experimental science.

The subsequent pivot to NCCL tuning and retraining with SGLang-based extraction represents a strategic redirection grounded in the hard-won knowledge of what works and what doesn't on this specific hardware configuration. The tuned SGLang server is loading, the data pipeline is being prepared, and the next phase of the optimization campaign is underway.

For practitioners building similar systems, the lessons are clear: