The Piecewise CUDA Graph Blacklist: A Detective Story in Performance Optimization

The Message

ssh root@10.1.230.172 'sed -n "1446,1505p" /root/ml-env/lib/python3.12/site-packages/sglang/srt/configs/model_config.py' 2>&1
piecewise_cuda_graph_disabled_model_archs = [
    "DeepseekV32ForCausalLM",
    "Qwen3NextForCausalLM",
    "GlmMoeDsaForCausalLM",
    "BailingMoeV2_5ForCausalLM",
    "LLaDAModelLM",
]

if external_mm_model_arch := envs.SGLANG_EXTERNAL_MM_MODEL_ARCH.get():
    multimodal_model_archs.append(external_mm_model_arch)


def is_multimodal_model(model_architectures: List[str]):
    if any(
        multi_model_arch in model_architectures
        for multi_model_arch in multimodal_model_archs
    ):
  ...

This seemingly innocuous command — reading a few dozen lines from a Python configuration file — represents the climax of a performance investigation that began with a simple user concern: "Cuda graphs/nccl? Don't leave perf on the table." The message is the moment of discovery, the point at which a hypothesis about a configuration error transforms into an understanding of an intentional design constraint.

Why This Message Was Written: The Investigation's Climax

The message was written to resolve a specific question that had been building across multiple rounds of the conversation. The user had deployed the Qwen3.6-27B model on a CT129 server with 2× A6000 GPUs and was getting approximately 50-57 tok/s with 3-step MTP speculative decoding. While this was a significant improvement over the earlier 35 tok/s with 1-step MTP, the user suspected there might be additional performance available through CUDA graph optimizations and NCCL configuration tweaks.

The assistant's investigation had already revealed several clues. Earlier in the conversation ([msg 8200]), the assistant had dumped the server arguments and noticed that disable_piecewise_cuda_graph=True was set — piecewise CUDA graphs were disabled. This immediately raised a red flag: piecewise CUDA graphs are an optimization that captures GPU operations into reusable graphs, avoiding the overhead of re-launching kernels on every decode step. If they were disabled, it could mean significant performance was being left on the table.

But the assistant didn't stop at the surface observation. Instead of blindly enabling the flag, they traced the logic through the SGLang codebase. In [msg 8201], they discovered that the disablement wasn't a user configuration choice — it was being set automatically by the _handle_piecewise_cuda_graph method in server_args.py. Further digging in <msg id=8202-8204> revealed that the decision flowed through is_piecewise_cuda_graph_disabled_model(), which checked the model's architecture against a hardcoded list.

Message 8206 is the moment that list is revealed. The assistant reads the exact source code lines containing piecewise_cuda_graph_disabled_model_archs, and there it is: &#34;Qwen3NextForCausalLM&#34; sits alongside four other model architectures (DeepseekV32ForCausalLM, GlmMoeDsaForCausalLM, BailingMoeV2_5ForCausalLM, LLaDAModelLM) in the exclusion list.

The Reasoning and Decision-Making Process

The assistant's thinking process here is a textbook example of systematic debugging. The chain of reasoning went through several stages:

Stage 1 — Observation: The server logs showed disable_piecewise_cuda_graph=True. This looked like a configuration problem.

Stage 2 — Hypothesis Testing: The assistant checked the server arguments help text to understand what flags controlled CUDA graphs ([msg 8200]). They found --enable-breakable-cuda-graph, --disable-cuda-graph, and --enable-profile-cuda-graph among the options.

Stage 3 — Root Cause Tracing: Instead of jumping to conclusions, the assistant traced the disablement logic through the code. They found that disable_piecewise_cuda_graph was being set to True automatically in _handle_piecewise_cuda_graph(), not by user configuration.

Stage 4 — Following the Dependency Chain: The assistant then traced the condition that triggered the disablement — self.get_model_config().is_piecewise_cuda_graph_disabled_model — which led to the is_piecewise_cuda_graph_disabled_model() function in model_config.py.

Stage 5 — The Reveal: Message 8206 reads the actual blacklist, confirming that Qwen3NextForCausalLM is intentionally excluded from piecewise CUDA graph support.

The crucial decision made in this message is implicit: the assistant chooses to read the source code rather than speculate or guess. This is a deliberate methodological choice. Rather than asking "Can we enable piecewise CUDA graphs and test it?" — which would be the faster path — the assistant asks "Why are they disabled?" This reflects a deeper understanding that if SGLang's developers intentionally excluded this model architecture, there's likely a fundamental incompatibility that no amount of flag-toggling will fix.

Assumptions and Their Validity

Several assumptions underpin this investigation:

Assumption 1: The performance ceiling is not yet reached. The user's question "Don't leave perf on the table" assumes there is performance to be gained. The assistant's investigation validates this assumption as a reasonable hypothesis worth testing, but the discovery in message 8206 ultimately refutes it — the performance is likely already at the ceiling for this model on this hardware.

Assumption 2: CUDA graph disablement might be a configuration error. The assistant initially treated the disabled piecewise CUDA graphs as potentially suboptimal. This was a reasonable working hypothesis, but the investigation proved it wrong — the disablement is intentional and architecture-specific.

Assumption 3: The SGLang codebase is well-structured enough to trace. The assistant assumed that by following the code paths, they could find the root cause. This assumption was validated — the code was cleanly organized with clear separation between server arguments, model configuration, and the architecture blacklist.

Assumption 4: The model architecture name is discoverable. The assistant assumed that Qwen3.6-27B would use an architecture matching one of the entries in the blacklist. This was correct — Qwen3NextForCausalLM is indeed the architecture for this model (the "Next" suffix indicating MTP support).

Mistakes and Incorrect Assumptions

One subtle mistake in the assistant's approach: the investigation could have been faster. The assistant spent several rounds tracing through server_args.py, model_config.py, and the architecture blacklist. A more direct approach would have been to check the model's architecture name first (via model.config.architectures) and then search for it in the blacklist. However, this "mistake" is really a tradeoff — the thorough approach provided deeper understanding of the codebase structure and confirmed that no other configuration flags were interfering.

Another potential blind spot: the assistant didn't check whether the piecewise CUDA graph disablement applies to the draft model (MTP) path or the base model path. The Qwen3NextForCausalLM architecture includes both the base model and the MTP heads. It's possible that piecewise CUDA graphs could work for the base model's decode but not for the MTP verification step. The blacklist is a blanket exclusion, and the assistant didn't explore whether a more nuanced configuration could partially enable the optimization.

Input Knowledge Required

To fully understand this message, one needs:

  1. Understanding of CUDA graphs: CUDA graphs allow capturing a sequence of GPU kernel launches into a single graph that can be replayed with minimal CPU overhead. Piecewise CUDA graphs break this into segments, allowing dynamic batching while still amortizing launch overhead.
  2. Knowledge of SGLang's architecture: SGLang is a serving framework for LLMs. It uses a scheduler that batches requests and dispatches them to GPU workers. CUDA graphs are a key optimization for reducing latency in the decode phase.
  3. Understanding of MTP (Multi-Token Prediction): MTP is a speculative decoding technique where a model predicts multiple future tokens simultaneously using auxiliary heads. The Qwen3.6-27B model uses this, and the "Next" in its architecture name refers to this capability.
  4. Familiarity with the model architecture blacklist pattern: The concept of maintaining a list of architectures that are incompatible with certain optimizations is common in ML serving frameworks. It represents accumulated engineering knowledge about which features work with which model designs.
  5. Context of the performance investigation: The preceding messages establish that the server is running on 2× A6000 GPUs with specific memory constraints, and that the user is performance-conscious.

Output Knowledge Created

This message creates several pieces of valuable knowledge:

  1. A confirmed root cause: Piecewise CUDA graphs are disabled for Qwen3NextForCausalLM because the architecture is explicitly listed in the exclusion blacklist. This is not a bug or misconfiguration.
  2. A documented constraint: The five architectures in the blacklist (DeepseekV32ForCausalLM, Qwen3NextForCausalLM, GlmMoeDsaForCausalLM, BailingMoeV2_5ForCausalLM, LLaDAModelLM) share common characteristics — they are all models with speculative decoding or mixture-of-experts architectures where the dynamic computation graph makes piecewise CUDA graph capture unreliable or counterproductive.
  3. A boundary for optimization: The investigation establishes that the current ~50-57 tok/s is likely the practical ceiling for this model on this hardware. Any further optimization would need to target different bottlenecks (memory bandwidth, attention kernel efficiency, or the speculative decoding acceptance rate).
  4. A reusable debugging methodology: The assistant's approach — observe a suspicious flag, trace its origin through the codebase, follow the dependency chain to the authoritative source — is a transferable skill demonstrated through the investigation.

The Thinking Process Visible in the Reasoning

The assistant's thinking is visible in the progression of tool calls across messages 8200-8206. Each step reveals a narrowing focus:

  1. Broad reconnaissance ([msg 8200]): Dump all server args and grep for CUDA graph, NCCL, and overlap-related flags. This establishes the landscape of available optimizations.
  2. Pattern matching ([msg 8201]): Notice that disable_piecewise_cuda_graph=True is set. Cross-reference with the help text to understand what flags control this behavior.
  3. Root cause tracing (<msg id=8201-8202>): Read the _handle_piecewise_cuda_graph method in server_args.py to see how the flag is set. Discover it's automatic, not manual.
  4. Dependency chain ([msg 8203]): Follow the condition to is_piecewise_cuda_graph_disabled_model in model_config.py. This is the critical juncture — the assistant recognizes that the model configuration is the authoritative source.
  5. The blacklist (<msg id=8204-8205>): Find the function and the list of disabled architectures. Confirm that Qwen3NextForCausalLM is in the list.
  6. The reveal ([msg 8206]): Read the exact source code to show the complete list, providing irrefutable evidence. This progression mirrors the scientific method: observe, hypothesize, test, trace, conclude. The assistant never assumes; it verifies at each step by reading the actual source code rather than relying on documentation or guesses.

Broader Implications

The discovery in message 8206 has implications beyond this single deployment. It reveals an important architectural constraint in SGLang: models with speculative decoding architectures (like Qwen3NextForCausalLM) or dynamic MoE routing (like DeepseekV32ForCausalLM) cannot use piecewise CUDA graphs because their computation graphs change between steps. When a model has multiple possible execution paths (e.g., different MTP heads fire on different steps, or different experts are activated), a captured CUDA graph would be invalid for the un-captured path.

This is a fundamental tension in LLM serving optimization: the more dynamic the model architecture, the harder it is to amortize launch overhead through graph capture. For the Qwen3.6-27B deployment, this means the path to higher throughput lies not in CUDA graph optimization but in improving the speculative decoding acceptance rate — which is exactly what the DFlash drafter training pipeline (the other major thread of this segment) aims to achieve.

The message thus serves as a pivot point: it closes the door on one optimization path and implicitly validates the parallel effort to build a better drafter model. The performance ceiling is not a software configuration problem; it is a fundamental architectural constraint that can only be overcome by improving the model's speculative decoding capabilities.