Chasing the CUDA Graph: Debugging EAGLE-3 Verify Performance in SGLang

In the middle of an intense performance debugging session, a single message captures a pivotal moment of investigation. The message, <msg id=4897>, is deceptively brief — a single line of reasoning followed by a grep command — but it represents the culmination of a multi-hour effort to understand why EAGLE-3 speculative decoding was performing worse than the baseline model, despite months of development and training.

The Context: A Performance Regression That Wasn't

To understand this message, we need to step back. The user had been working on deploying a Kimi-K2.5 model (a 1-trillion-parameter Mixture-of-Experts model) with EAGLE-3 speculative decoding — a technique where a small "draft" model generates candidate tokens that a large "target" model verifies in parallel, theoretically achieving higher throughput than running the large model alone. Earlier in the session, the assistant had measured EAGLE-3 speculation at 94 tok/s, which seemed promising. But subsequent testing revealed a harsh reality: the true baseline for the target model alone was 82-83 tok/s, and EAGLE-3 was delivering only 59-61 tok/s — a 27% regression rather than an improvement.

The assistant had already ruled out several hypotheses. A git pull that updated SGLang was not the culprit — reverting to the old commit produced identical baseline performance. NCCL tuning environment variables had been meticulously propagated through the worker spawn chain (engine.py, scheduler.py, sitecustomize.py) without effect. The performance numbers were stable and reproducible.

The real issue was hiding in plain sight: the EAGLE-3 "verify" step — where the target model processes the draft tokens — was taking approximately 29ms to verify 3 tokens, while a single-token baseline decode took only ~12ms. That's 9.7ms per token for verify versus 12ms per token for decode — actually faster per-token in raw compute, but the verify processes 3 tokens serially in one forward pass, meaning the total latency per speculation cycle is 29ms versus 12ms for a single decode step. With an average acceptance length of ~2.0 tokens per speculation cycle, EAGLE-3 was producing 2 tokens per 29ms cycle, while the baseline produced 1 token per 12ms — 60 tok/s versus 82 tok/s.

The Hypothesis: CUDA Graphs as the Missing Piece

In <msg id=4895>, the assistant formulated a critical hypothesis. The baseline decode uses CUDA graphs — a feature that captures a sequence of GPU kernel launches into a single replayable graph, eliminating kernel launch overhead. With 61 transformer layers and 8-way tensor parallelism, each layer requires an allreduce operation across GPUs. In CUDA graph mode, all 61 allreduce kernel launches are pre-recorded and replayed with near-zero overhead. Without CUDA graphs, each allreduce requires a dynamic kernel launch, and the cumulative overhead of 61 launches adds up significantly.

The verify step, however, was running in "extend" mode (also called prefill mode) because it processes a batch of draft tokens against the existing KV cache. The SGLang code explicitly set can_run_cuda_graph=False for this path (visible in <msg id=4896> at line 322 of eagle_worker.py). The assistant realized something crucial: with EAGLE-3's topk=1 configuration, the number of draft tokens is fixed — it's always num_steps + 1 tokens. This means the verify batch size is constant, which is the key prerequisite for CUDA graph capture. If the verify step could use CUDA graphs, the 29ms cycle time might drop significantly.

The Subject Message: Following the Variable Chain

This brings us to <msg id=4897>. The assistant writes:

can_run_cuda_graph is returned from the target model's forward pass. Let me check what controls this: [bash] ssh root@10.1.230.174 'grep -n "can_run_cuda_graph\|can_run.*graph" /root/sglang/python/sglang/srt/model_executor/model_runner.py | head -20'

This is a textbook debugging maneuver: trace the variable backward. The assistant had already seen that can_run_cuda_graph flows through the verify path in eagle_worker.py — it's returned from batch_result.can_run_cuda_graph and passed to downstream functions. But where does this boolean originate? What logic determines whether the model runner believes it can use CUDA graphs for a given forward pass?

The command searches model_runner.py — the core file that orchestrates the actual model forward pass — for all references to can_run_cuda_graph or can_run.*graph. The output reveals several important locations:

What This Reveals About the Thinking Process

The assistant's reasoning here is methodical and layered. Let me unpack the cognitive stack:

Layer 1: Observation. The verify step is 29ms for 3 tokens while baseline decode is 12ms for 1 token. The per-token compute is actually similar, but the total cycle time is 2.4x worse.

Layer 2: Hypothesis formation. The difference must be CUDA graph usage. Baseline decode uses CUDA graphs (eliminating kernel launch overhead across 61 layers × allreduce operations). Verify does not.

Layer 3: Code tracing. The assistant traces can_run_cuda_graph from eagle_worker.py (where it's consumed) back to model_runner.py (where it's produced). This is the step captured in the subject message.

Layer 4: Feasibility analysis. The assistant had already reasoned (in <msg id=4895>) that with topk=1, the verify batch size is constant, making CUDA graphs theoretically applicable. The question is whether the SGLang codebase has the plumbing to support this, or whether a modification would be needed.

Assumptions and Knowledge Required

To understand this message, several pieces of background knowledge are necessary:

  1. CUDA Graphs: A CUDA feature that captures a sequence of kernel launches into a graph object that can be replayed with minimal overhead. Critical for high-throughput inference because it eliminates CPU-side launch latency for each kernel.
  2. EAGLE-3 Speculative Decoding: A draft-then-verify architecture where a small model generates candidate tokens and the large target model processes them in a single forward pass (verify). The verify processes num_steps + 1 tokens at once.
  3. Tensor Parallelism and Allreduce: With 8-way tensor parallelism, each transformer layer's output must be synchronized across all 8 GPUs via an allreduce operation. With 61 layers, that's 61 allreduces per forward pass.
  4. SGLang Architecture: The model_runner.py file orchestrates the model forward pass, while eagle_worker.py handles the speculative decoding loop. The can_run_cuda_graph flag is a coordination mechanism between them.
  5. The 1T MoE Model: Kimi-K2.5 is a Mixture-of-Experts model with approximately 1 trillion parameters, running across 8 GPUs connected via PCIe. PCIe interconnects add latency to allreduce operations, making CUDA graph optimization even more important.

The Deeper Significance

This message is a perfect example of performance debugging at the systems level. The assistant isn't looking at model architecture or training hyperparameters — it's looking at kernel launch overhead, NCCL communication patterns, and the interaction between speculative decoding and GPU execution models. This is the kind of debugging that separates a working deployment from an optimized one.

The can_run_cuda_graph variable is a thin thread connecting two complex systems: the speculative decoding infrastructure (which decides whether to attempt CUDA graph capture) and the model runner (which actually executes the forward pass). By following this thread, the assistant is attempting to answer a question with significant practical implications: can the verify step be made to use CUDA graphs, and if so, by how much would it close the performance gap?

The answer to this question would determine the entire trajectory of the project. If CUDA graphs could reduce verify time from 29ms to, say, 15ms (closer to the decode cost), then EAGLE-3 would immediately become viable. If not, the only path forward would be improving the draft model's acceptance rate through more training data — a much more expensive and time-consuming approach.

What Comes Next

The message ends with the grep output showing the relevant code locations. The assistant would then need to examine each of these locations to understand the logic. The critical path is at lines 2315-2336 and 2448-2460, where can_run_graph is computed. The assistant needs to understand: under what conditions does model_runner.py decide that CUDA graphs are usable? Is it gated on batch size, sequence length, attention mode, or something else? And crucially, can the verify path satisfy those conditions?

The grep output shows can_run_piecewise_cuda_graph appearing at line 2163 — this is a separate feature (piecewise CUDA graphs for multi-layer capture) that might interact with the verify path. The assistant would need to trace through these conditions to determine whether a simple configuration change could enable CUDA graphs for verify, or whether a code modification would be required.

This single message, for all its brevity, represents the turning point in a debugging session. The assistant has moved from asking "what is wrong?" to asking "what controls this specific behavior?" — a shift from diagnosis to intervention planning. The grep command is the first step toward understanding whether the fix is a configuration toggle, a code patch, or an impossibility that forces a strategic pivot.