The Ceiling: Why ~55 tok/s Is All You Get from 2× A6000 with MTP

In the long arc of an opencode coding session spanning environment setup, model deployment, speculative decoding training, and performance optimization, there comes a moment when the numbers stop moving. Message 8224 is that moment. It is a short, almost anticlimactic message — the assistant reports that after an extensive investigation into CUDA graphs, NCCL configuration, and tensor-parallel overlap scheduling, the throughput remains stubbornly at 49–57 tok/s, essentially unchanged from before. The message is notable not for what it achieves, but for what it honestly concedes: that on this particular hardware, with this particular model and speculative decoding configuration, the performance is at its ceiling.

The Context: A User Who Wants Every Drop of Performance

The story begins with the user's sharp challenge at message 8199: "Cuda graphs/nccl? Don't leave perf on the table." This is not a casual comment. It comes from a user who clearly understands that inference serving involves many knobs — CUDA graph capture, NCCL allreduce tuning, overlap scheduling, memory management — and that default configurations often leave performance on the table. The user is asking the assistant to do what any good engineer would: verify that every optimization has been explored before accepting the current throughput numbers.

The assistant had just finished deploying Qwen3.6-27B on a CT129 server with two NVIDIA RTX A6000 GPUs, using SGLang with 3-step MTP (Multi-Token Prediction) speculative decoding. The initial benchmark showed ~50–57 tok/s with an acceptance length of ~3.0–3.5 tokens (see [msg 8197]). The user remembered seeing 70 tok/s at some point and wanted to know why the current numbers were lower.

The Investigation: Tracing Through SGLang's Source Code

What follows is a textbook example of systematic performance debugging. The assistant does not guess or hand-wave. It SSHes into the server, inspects the actual server arguments from the log, and cross-references them against the SGLang source code to understand which optimizations are enabled and which are not.

The investigation reveals several findings ([msg 8201]):

Piecewise CUDA graph is disabled (disable_piecewise_cuda_graph=True). This is significant because piecewise CUDA graphs can accelerate the prefill/extend phase by capturing the entire computation as a single graph, reducing kernel launch overhead.
Custom allreduce is enabled (disable_custom_all_reduce=False) — good.
NVLS (NVLink Shared Storage) is disabled (enable_nccl_nvls=False) — but this is expected because A6000 GPUs are SM86 architecture, and NVLS requires SM90+ (H100/B100).
Overlap scheduler is enabled (disable_overlap_schedule=False) — good.
Single-batch overlap is disabled (enable_single_batch_overlap=False) — this could potentially help overlap computation with communication within a single batch.
Two-batch overlap is disabled (enable_two_batch_overlap=False). The assistant then dives into the SGLang source code to understand why piecewise CUDA graphs are being disabled. Through a series of grep and sed commands on the server's Python installation, it traces the logic through server_args.py and model_config.py (<msg id=8201–8206>). The culprit is condition #8 in the _handle_piecewise_cuda_graph method: the Qwen3.6-27B model uses the Qwen3_5ForConditionalGeneration architecture, which SGLang classifies as multimodal/VLM. For multimodal models, piecewise CUDA graphs are automatically disabled because the vision encoder paths use custom operations that torch.compile cannot handle. The assistant also discovers that the model architecture is NOT in the explicit piecewise_cuda_graph_disabled_model_archs list ([msg 8206]), which means the disable is coming from the multimodal check, not an explicit exclusion. This opens the door to forcing it on with --enforce-piecewise-cuda-graph.

The Attempt: Restarting with All Flags

Armed with this understanding, the assistant restarts the server with two additional flags: --enforce-piecewise-cuda-graph to force-enable piecewise CUDA graphs despite the VLM classification, and --enable-single-batch-overlap to try overlapping computation and communication within a batch ([msg 8216]). The first attempt fails because the assistant also tried --language-only, which requires --encoder-urls for encoder disaggregation mode — a configuration that doesn't apply here (<msg id=8217–8218>). After dropping that flag, the server starts successfully with piecewise CUDA graphs enabled.

The logs confirm that piecewise CUDA graphs are indeed being compiled: the server captures 50 token-length variants during startup ([msg 8219]). The assistant waits for the compilation to complete and then benchmarks the new configuration (<msg id=8222–8223>).

The Result: No Improvement

And then comes message 8224 — the subject of this article. The assistant reports the benchmark results plainly:

Roughly the same — 49–57 tok/s, accept len 2.7–3.35. The piecewise graphs help prefill (not showing here since it's a short prompt), and single-batch overlap won't show much on only TP=2 with PCIe (not NVLink).

This is the moment of truth. The assistant has done the work, enabled the optimizations, measured the result, and found no material improvement. The explanation is precise and technically grounded:

Piecewise CUDA graphs accelerate chunked prefill, not decode. The benchmark uses a short 33-token prompt, so the prefill phase is negligible. The benefit would only appear with long-context inputs (thousands of tokens of context), where chunked prefill dominates.
Single-batch overlap doesn't help with TP=2 over PCIe. The bottleneck during decode is the NCCL allreduce that synchronizes the two GPUs after each layer. With only two GPUs connected via PCIe (not NVLink), the allreduce latency is the dominant term, and overlapping it with computation provides minimal gain because the computation itself is also memory-bound.
The A6000s are connected via PCIe, not NVLink. This is the fundamental hardware limitation. NVLink provides 600 GB/s bidirectional bandwidth between GPUs, while PCIe Gen4 x16 provides ~32 GB/s. The allreduce for a 27B model's weights (27 GB of parameters, split across two GPUs) must transfer ~13.5 GB across this narrow pipe during each decode step. That transfer time dominates the step latency.

The Honest Assessment: ~55 tok/s Is the Ceiling

The assistant then delivers the conclusion that the user needs to hear but may not want to:

The honest answer: ~55 tok/s is close to the ceiling for this hardware with 3-step MTP. The A6000s are connected via PCIe, not NVLink — TP allreduce is the bottleneck during decode, and no CUDA graph trick fixes that.

This is a crucial moment in any engineering conversation: the point where investigation exhausts the available optimization space and the answer is "this is as fast as it gets." The assistant does not hedge or offer false hope. It explains why the remembered 70 tok/s was likely an outlier — either a shorter prompt, warmer caches, or thinking tokens that decode faster because repetitive patterns have higher MTP acceptance.

The message then outlines the two real paths to higher throughput:

Better speculative decoding: The DFlash drafter being trained in parallel sessions targets an acceptance length of 5–6 tokens, which would directly increase decode throughput by reducing the number of target model invocations per token.
NVLink interconnect: Physically connecting the GPUs via NVLink would reduce allreduce latency by an order of magnitude, but this is not available on the CT129 server hardware.

What Makes This Message Significant

Message 8224 is a model of how to deliver bad news in an engineering context. It is brief but complete: it states the result, explains why the attempted optimizations didn't work, identifies the root cause (PCIe allreduce bottleneck), and provides the actionable paths forward. It does not apologize or equivocate. It does not suggest trying more flags or tweaking more parameters. It accepts the hardware limitation and redirects to the strategies that can actually move the needle.

The message also demonstrates a critical skill: knowing when to stop optimizing. The assistant could have continued tweaking — trying different CUDA graph batch sizes, adjusting NCCL timeout parameters, experimenting with different tensor parallelism strategies. But the investigation had already identified the fundamental bottleneck: PCIe bandwidth between the two GPUs. No software optimization can overcome a hardware bottleneck of this nature. Continuing to tweak would be wasting time that could be better spent on the DFlash drafter training or other productive work.

The Broader Lesson: Hardware Constraints Are Real

For anyone working with LLM inference serving, this message encapsulates a hard truth that many practitioners learn through painful experience: software optimizations can only go so far. CUDA graphs, overlap scheduling, attention kernel selection, and memory management can all help, but when the bottleneck is raw data movement across a narrow interconnect, no amount of software cleverness will change the physics.

The two A6000 GPUs on the CT129 server are powerful accelerators — 48 GB of memory each, respectable compute throughput — but they are connected by the same PCIe bus that also handles storage and network I/O. For tensor-parallel inference, where every layer requires an allreduce of the activations, this interconnect becomes the dominant constraint. The assistant's analysis correctly identifies that the decode step spends 83% of its time reading 27 GB of weights (as established in the segment's chunk summary), and the allreduce adds additional latency on top of that.

Conclusion: A Message That Closes a Chapter

Message 8224 closes the chapter on performance optimization for the Qwen3.6-27B deployment on CT129. It is the answer to the user's challenge — yes, the assistant investigated CUDA graphs and NCCL, and no, there is no hidden performance to unlock. The numbers are what they are because the hardware is what it is.

The message also serves as a pivot point. With the ceiling established, the conversation can move on to the strategies that can improve throughput: the DFlash drafter training that is already underway in parallel sessions, and potentially hardware upgrades. The assistant's final line — "To actually get back to 70+, you'd need either: Higher MTP acceptance (our DFlash drafter at accept len 5-6 would do it) or NVLink between the GPUs (not available on CT129)" — is both a conclusion and a roadmap.

In the broader narrative of the coding session, this message represents the moment when empirical measurement meets honest communication. The assistant did the work, gathered the data, and delivered the verdict without spin or sugar-coating. It is a small message — barely a paragraph of analysis — but it carries the weight of the twenty-plus messages of investigation that preceded it, and it shapes the direction of everything that follows.