The Performance Audit: Forcing CUDA Graphs and Overlap Scheduling in SGLang

In the high-stakes world of large language model serving, every token per second counts. When a user running Qwen3.6-27B on dual NVIDIA RTX PRO 6000 (A6000) GPUs noticed throughput had dropped from ~70 tok/s to ~35-40 tok/s, the assistant quickly diagnosed the issue: the server had been started with a single-step MTP (Multi-Token Prediction) speculative decoding configuration instead of the original 3-step NEXTN setup. Restoring the original configuration brought throughput back to ~50-57 tok/s, a solid improvement but still below the remembered peak. Then came the user's pointed challenge in message 8199: "Cuda graphs/nccl? Don't leave perf on the table." This single remark triggered a deep forensic investigation into SGLang's performance optimization flags, culminating in message 8216 — a carefully constructed server launch command that represents the assistant's best attempt to squeeze every last drop of performance from the hardware.

The Investigation: Tracing Through SGLang's Source Code

Message 8216 is not a standalone action; it is the conclusion of a meticulous debugging session spanning messages 8200 through 8215. The assistant's response to the user's challenge was immediate: rather than guessing at optimization flags, it went directly to the running server's logs and SGLang's source code to understand exactly what was enabled and disabled.

The first clue came from parsing the server's startup arguments, which revealed disable_piecewise_cuda_graph=True. Piecewise CUDA graphs are a SGLang optimization that captures the prefill and extend phases of inference into CUDA graphs, reducing kernel launch overhead. Having this disabled was a red flag. The assistant then cross-referenced the available flags via --help, identifying three promising candidates: --enforce-piecewise-cuda-graph, --language-only, and --enable-single-batch-overlap.

But the real detective work began when the assistant asked why piecewise CUDA graphs were being disabled. By tracing through server_args.py in the SGLang source tree, the assistant discovered a chain of conditional checks (messages 8201-8213). The culprit was condition #8: if the model's architecture is classified as multimodal (VLM), SGLang automatically disables piecewise CUDA graphs. The Qwen3.6-27B model uses the Qwen3_5ForConditionalGeneration architecture, which is indeed a multimodal architecture capable of handling images. Even though this deployment was purely text-based, SGLang's auto-detection flagged it as VLM and turned off the optimization.

Further investigation revealed that --language-only exists precisely for this scenario — it tells SGLang to skip loading vision encoder weights and treat the model as text-only. However, the assistant correctly noted that --language-only might not bypass the piecewise CUDA graph check, which is based on the model's architecture class rather than runtime flags. This is why --enforce-piecewise-cuda-graph was needed as a separate override.

The third flag, --enable-single-batch-overlap, addresses a different bottleneck: overlapping computation with communication during tensor-parallel allreduce operations. On the A6000 (SM86 architecture), NCCL NVLS (NVLink Sharp) is unavailable — that requires SM90+ (H100/B100). But the custom allreduce implementation can still benefit from overlapping the allreduce communication with subsequent computation within a single batch, if the scheduler supports it.

The Launch Command: A Synthesis of Discoveries

Message 8216 presents the final launch command, which incorporates all three performance flags into the previously restored configuration:

SGLANG_ENABLE_SPEC_V2=1 nohup ~/ml-env/bin/python3 -m sglang.launch_server \
  --model-path /root/models/Qwen3.6-27B \
  --port 30000 \
  --host 0.0.0.0 \
  --tp-size 2 \
  --mem-fraction-static 0.88 \
  --context-length 131072 \
  --max-running-requests 16 \
  --mamba-full-memory-ratio 0.5 \
  --mamba-scheduler-strategy extra_buffer \
  --reasoning-parser qwen3 \
  --tool-call-parser qwen3_coder \
  --speculative-algorithm NEXTN \
  --speculative-num-steps 3 \
  --speculative-eagle-topk 1 \
  --speculative-num-draft-tokens 4 \
  --trust-remote-code \
  --language-only \
  --enforce-piecewise-cuda-graph \
  --enable-single-batch-overlap

The command is a study in deliberate engineering. Every flag serves a purpose, and the three new additions are the direct result of the source code audit:

--language-only: Skips loading the vision encoder weights, saving approximately 1-2 GB of GPU memory. This is safe because the deployment only serves text requests. The freed memory can be used for larger KV cache allocations or higher batch sizes.
--enforce-piecewise-cuda-graph: Overrides SGLang's auto-detection and forces the use of piecewise CUDA graphs for prefill and extend operations, even though the model architecture is classified as multimodal. This is the key optimization that the user was asking about — it captures the prefill computation into a CUDA graph, reducing kernel launch overhead and improving latency for the first few tokens of each request.
--enable-single-batch-overlap: Enables computation-communication overlap within a single batch during tensor-parallel inference. On 2× A6000 with TP=2, every forward pass requires an allreduce across GPUs. This flag allows the scheduler to overlap that communication with subsequent computation, theoretically improving GPU utilization.

Assumptions and Potential Pitfalls

The assistant made several assumptions in constructing this command, some of which deserve scrutiny.

First, the assumption that --enforce-piecewise-cuda-graph would work correctly on this model. The flag exists precisely to override the auto-disable logic, but there may be a reason the SGLang developers chose to disable it for VLM architectures — perhaps certain multimodal operations (like cross-attention with image features) are not compatible with CUDA graph capture. Since this deployment uses the model in language-only mode, the risk is mitigated, but it's not zero. If the CUDA graph capture fails or produces incorrect results, the server could silently produce degraded outputs.

Second, the assumption that --language-only is compatible with the MTP speculative decoding setup. The Qwen3.6-27B model with NEXTN speculative decoding uses a draft model structure that may have different weight layouts than the base model. If the language-only flag strips weights that the MTP heads depend on, the server could crash or produce nonsense. The assistant implicitly trusts that the model's MTP heads are part of the language backbone, not the vision encoder — a reasonable but unverified assumption.

Third, the assumption that --enable-single-batch-overlap provides meaningful benefit on A6000. The overlap optimization is most effective when compute and communication can be pipelined, which requires sufficient compute to hide the allreduce latency. On A6000s connected via NVLink (which these likely are, given the TP=2 configuration), the allreduce bandwidth is high and the latency is low, so the overlap window may be small. The flag may provide single-digit percentage improvements at best.

Input Knowledge Required

To fully understand this message, one needs knowledge of several domains:

SGLang server architecture: Understanding what piecewise CUDA graphs are, how they differ from decode-phase CUDA graphs, and why they're beneficial for prefill/extend operations.
CUDA graph capture: The concept of capturing GPU kernel launches into a graph that can be replayed with minimal CPU overhead, and the limitations (graphs must be static — no dynamic control flow, no variable tensor shapes).
Tensor parallelism and allreduce: How TP=2 splits the model across two GPUs and requires allreduce operations to synchronize gradients/activations, and how overlapping communication with compute can improve utilization.
GPU architecture differences: SM86 (A6000/RTX 3090) vs SM90 (H100) vs SM100 (B100) capabilities, particularly regarding NVLS and other advanced communication primitives.
SGLang's VLM detection logic: The model architecture classification system that auto-disables certain optimizations for multimodal models.
MTP/NEXTN speculative decoding: How multi-token prediction works, what the --speculative-num-steps 3 and --speculative-num-draft-tokens 4 flags do, and how they interact with CUDA graph capture.

Output Knowledge Created

This message produces a running SGLang inference server with all available performance optimizations enabled. The immediate output is a PID (50277) and a log file at /root/sglang_serve.log. The broader output is a validated configuration that can be reused for future deployments — either via the systemd service (which was set up in message 8197) or as a reference command for similar setups.

More importantly, the message creates knowledge about which flags matter and why. The investigation revealed that SGLang's auto-detection is conservative — it disables optimizations by default for VLM architectures even when the deployment is text-only. The pattern of --language-only + --enforce-piecewise-cuda-graph is a reusable recipe for any multimodal model being used for text-only inference. This is the kind of hard-won deployment knowledge that rarely appears in documentation.

The Thinking Process

The assistant's reasoning in this message is notable for its systematic, evidence-driven approach. Rather than trying random flags or relying on intuition, the assistant:

Inspected the current state: Parsed the server's startup arguments to see what was actually enabled.
Identified discrepancies: Found that disable_piecewise_cuda_graph=True despite the model not being in the explicit disabled list.
Traced the root cause: Followed the code path through server_args.py, checking each conditional that could disable piecewise CUDA graphs, until finding the VLM architecture check.
Verified hardware constraints: Checked whether NVLS was available (no, A6000 is SM86) and whether flashinfer allreduce fusion applied (no, needs SM90+).
Formulated a targeted fix: Identified three flags that address the specific issues found, rather than a shotgun approach.
Cleaned state: Killed the old server and verified GPUs were idle before launching the new one. This process exemplifies the ideal debugging workflow for deployment infrastructure: measure, inspect, trace, understand, fix, verify.

Conclusion

Message 8216 is a masterclass in performance-driven server configuration. It represents the culmination of a deep investigation into SGLang's optimization internals, translating source code analysis into three targeted flags that unlock piecewise CUDA graphs, save memory by skipping vision encoder loading, and enable computation-communication overlap. Whether these flags restore the remembered ~70 tok/s remains to be seen — the next messages in the conversation will reveal the benchmark results. But regardless of the outcome, the methodology is sound: understand why an optimization is disabled before deciding to override it.