The Turning Point: CUDA Graphs Finally Work with EAGLE-3 Speculative Decoding

In the high-stakes world of large language model deployment, every millisecond counts. When the team behind the Kimi-K2.5 model deployment set out to accelerate inference using EAGLE-3 speculative decoding, they encountered a cascade of obstacles — a hidden state concatenation bug, low acceptance rates, and a particularly stubborn problem with CUDA graphs that consistently caused the server to hang. Message 3634 marks the moment when that last barrier finally broke.

The Message in Context

The message is deceptively brief:

CUDA graphs didn't hang this time! The server started successfully. Let me benchmark: [bash] ssh root@10.1.230.174 'grep -E "cuda graph|graph capture|CUDA" /data/eagle3/sglang_eagle3_cudagraph.log | tail -10' [2026-02-23 17:52:13 TP6] Capture draft extend cuda graph end. Time elapsed: 1.42 s. mem usage=0.49 GB. avail mem=6.98 GB. [2026-02-23 17:52:13 TP4] Capture draft extend cuda graph end. Time elapsed: 1.42 s. mem usage=0.49 GB. avail mem=6.98 GB. [2026-02-23 17:52:13 TP5] Capture draft extend cuda graph end. Time elapsed: 1.42 s. mem usage=0.49 GB. avail mem=6.98 GB. [2026-02-23 17:52:13 TP7] Capture draft extend cuda graph end. Time elapsed: 1.40 s. mem usage=0.49 GB. avail mem=7.02 GB. [2026-02...

The assistant then runs a grep command to verify that CUDA graph capture succeeded across all eight tensor-parallel (TP) ranks, and the logs confirm it: each of TP4 through TP7 (and presumably TP0-TP3 as well) completed their "draft extend" CUDA graph capture in approximately 1.4 seconds, using only 0.49 GB of memory per rank. This is the first successful CUDA graph capture for the EAGLE-3 speculative decoding pipeline in this entire session.

Why This Message Was Written

To understand the significance, we must trace the history. The team had been battling a multi-layered performance problem. After fixing the critical bug where the server was started with --speculative-algorithm EAGLE instead of EAGLE3 — a single character difference that caused the draft model to receive 7168-dim single-layer hidden states instead of the expected 21504-dim concatenated states — they achieved a functional speculative decoding pipeline. But the performance was disappointing: without CUDA graphs, the best configuration achieved only 53.2 tok/s, far below the 90 tok/s non-speculative baseline.

CUDA graphs had been tried before and consistently caused the server to hang or deadlock, particularly on the SM120 architecture of the RTX PRO 6000 Blackwell GPUs. The previous attempts were abandoned, and the team fell back to running with --disable-cuda-graph. But the overhead of launching individual CUDA kernels for each speculative step was crippling — each verification step required multiple kernel launches, and with an accept length of only ~2.1 tokens per step, the overhead of speculation was exceeding its benefit.

Message 3632 launched a new server instance with CUDA graphs enabled, using the same configuration that had previously hung. Message 3633 showed the agonizingly long wait — the assistant polled the health endpoint every 10 seconds for over 4 minutes, watching the server slowly initialize. The subject message is the triumphant verification that this time, it worked.

The Thinking Process

The assistant's reasoning is visible in the structure of the message itself. The opening line — "CUDA graphs didn't hang this time!" — reveals the expectation of failure that had been built up through repeated attempts. The exclamation mark conveys genuine surprise and relief. The assistant immediately pivots to benchmarking, showing a pragmatic engineering mindset: the goal was never just to get CUDA graphs working, but to measure whether they actually improved throughput.

The choice of grep command is revealing. Rather than running a full benchmark immediately, the assistant first checks the server logs for CUDA graph capture messages. This is a diagnostic step — confirming that the graphs were actually captured before investing time in a benchmark run. The log lines show that each TP rank captured its "draft extend" CUDA graph in about 1.4 seconds, using a modest 0.49 GB of memory. The consistency across ranks (TP4, TP5, TP6, TP7 all showing nearly identical timings) indicates that the graph capture scaled well across the 8-GPU tensor-parallel configuration.

Input Knowledge Required

Understanding this message requires familiarity with several concepts:

• CUDA Graphs: A feature that allows a sequence of CUDA kernel launches to be captured and replayed as a single unit, eliminating kernel launch overhead. For speculative decoding, where the same pattern of draft model inference and target model verification repeats many times, CUDA graphs can dramatically reduce latency.
• Tensor Parallelism (TP): The model is sharded across 8 GPUs, with each GPU handling a portion of the computation. The log lines show TP4 through TP7, indicating the last four of eight ranks.
• EAGLE-3 Speculative Decoding: A draft model generates multiple candidate tokens, and the target model verifies them in parallel. The "draft extend" phase refers to running the draft model forward to generate candidates.
• SGLang Server Architecture: The server is launched with a complex set of NCCL environment variables (NCCL_PROTO=LL, NCCL_ALGO=Ring, etc.) and SGLang-specific flags that control memory allocation, speculative decoding parameters, and logging.

Output Knowledge Created

This message produces several important pieces of knowledge:

1. CUDA graphs are compatible with EAGLE-3 on SM120: Previous hangs were not fundamental incompatibilities but likely timing or initialization issues that were resolved by the specific configuration used.
2. Graph capture time is acceptable: At ~1.4 seconds per rank, the overhead of capturing CUDA graphs is negligible compared to the total server startup time (several minutes).
3. Memory overhead is low: Each rank uses only 0.49 GB for the captured graph, which is modest on 96 GB GPUs.
4. The configuration is reproducible: The NCCL environment variables, model path, speculative parameters, and other settings from msg 3632 produce a working server. The subsequent messages (3635-3637) build on this knowledge, showing that CUDA graphs boosted throughput from 53.2 tok/s to 74.9 tok/s — a 41% improvement — though still short of the 90 tok/s baseline. The assistant correctly diagnoses that the remaining gap is due to the low accept length (~2.1 tokens) rather than kernel launch overhead.

Assumptions and Their Validity

The assistant makes several assumptions in this message:

• That CUDA graphs would improve throughput: This assumption proved correct — the 41% improvement from 53.2 to 74.9 tok/s validated the decision to debug the CUDA graph hang.
• That the grep output confirms full functionality: The assistant assumes that successful graph capture implies correct operation during inference. This is a reasonable assumption, but the real test comes in the benchmark (msg 3635).
• That the same configuration would work without modification: The server was launched with the same parameters that previously caused hangs. The fact that it worked suggests the hang was intermittent or related to system state (e.g., residual GPU memory from a previous process) rather than a deterministic bug.

Broader Significance

This message sits at a critical juncture in the deployment pipeline. The team had invested enormous effort in training a custom EAGLE-3 drafter on 10,000 samples of Kimi-K2.5 hidden states, only to find that the accept rate was too low to justify speculation without CUDA graphs. The successful CUDA graph capture changed the economics: with graphs, the overhead per speculative step dropped enough that even a modest accept length of ~2.1 tokens produced a meaningful fraction of the baseline throughput.

However, the 74.9 tok/s result (revealed in msg 3635) still fell short of the 90 tok/s non-speculative baseline. This led to the insight that the fundamental lever was not further optimization of the inference pipeline, but more training data for the draft model. The EAGLE-3 paper shows a clear scaling relationship between training data volume and acceptance rate, and the team's next major effort — scaling the training dataset by 10× through an 83K-prompt inference pipeline — was a direct consequence of the analysis that followed this message.

In the end, message 3634 represents a bittersweet milestone: a technical victory that proved CUDA graphs could work, but also the moment when the team realized that no amount of inference optimization could substitute for a better-trained draft model. The path forward was clear: more data, better training, and a second attempt at the full EAGLE-3 pipeline.