The Plateau: Recognizing the PCIe Bottleneck in Blackwell MoE Inference

Introduction

In the course of deploying a 744-billion-parameter Mixture-of-Experts (MoE) model—GLM-5-NVFP4—across eight RTX PRO 6000 Blackwell GPUs, a pivotal moment arrived when the assistant, after successfully resolving a critical NaN crash during decode and enabling CUDA graphs, confronted an unexpected plateau. Message 240 in this coding session captures the precise instant when the assistant realized that the primary performance lever it had been pulling—CUDA graph capture—was not yielding the expected gains. This article examines that message in depth: the reasoning that led to the benchmark, the conclusion drawn from the results, the pivot toward new tuning strategies, and the assumptions that shaped the investigation.

The Context: A Hard-Won Victory

To understand message 240, one must appreciate the arduous path that preceded it. The session began with the deployment of GLM-5-NVFP4, a massive MoE model with 256 experts and approximately 40 billion active parameters per forward pass, on a system of eight RTX PRO 6000 Blackwell GPUs connected only via PCIe—no NVLink, no NVSwitch. The initial server launches had crashed with NaN errors during decode, a problem that required extensive debugging of attention backends. The breakthrough came when the assistant discovered that the --nsa-decode-backend trtllm and --nsa-prefill-backend trtllm flags were necessary for the SM120 architecture of the Blackwell GPUs, resolving the numerical instability.

With a working server, the assistant established baseline throughput: approximately 225 output tokens per second and 516 total tokens per second with 64 concurrent requests. The next logical step was tuning. The assistant increased the memory fraction from 0.85 to 0.92, expanding the KV cache from 370K to 498K tokens, and—most significantly—enabled CUDA graphs by omitting the --disable-cuda-graph flag. CUDA graphs are a powerful optimization that captures GPU kernel launch sequences into a single graph, reducing kernel launch overhead and improving utilization. The graphs captured successfully for batch sizes up to 64, and a sanity check confirmed the server produced coherent output ("Paris" to a simple question). Everything seemed poised for a performance leap.

The Benchmark That Changed the Narrative

Message 240 opens with the assistant reporting the results of a decode-heavy benchmark:

Interesting — with decode-heavy workload (128 in, 512 out): - Output throughput: 236 tok/s — slightly better - CUDA graphs confirmed (cuda graph: True in server log) - Server-side peak was ~200 tok/s with 22 concurrent

The numbers are striking. Despite the successful CUDA graph capture, the output throughput moved from 225 tok/s to 236 tok/s—a mere 5% improvement, well within the noise of benchmark variance. The server-side peak of ~200 tok/s with 22 concurrent requests was essentially unchanged from the pre-CUDA-graph configuration. The assistant had invested significant effort in enabling CUDA graphs—restarting the server, waiting through the capture process, verifying correctness—only to find that the bottleneck lay elsewhere.

This is the moment of diagnostic insight. The assistant articulates it clearly:

The throughput is capped around 200-236 tok/s regardless of CUDA graphs. This suggests the bottleneck is the MoE expert computation + all-reduce over PCIe, not kernel launch overhead.

This sentence is the heart of the message. It represents a shift from a "kernel launch overhead" mental model to a "communication-bound" mental model. The assistant is reasoning that if CUDA graphs—which specifically optimize kernel launch latency—do not improve throughput, then the bottleneck must be in the computation or communication that happens between kernel launches, not in the launches themselves.

The Reasoning Process: From Observation to Hypothesis

The assistant's reasoning chain is worth reconstructing in detail. The observation is: CUDA graphs improved throughput negligibly. Why would that be?

CUDA graphs eliminate the CPU overhead of launching individual GPU kernels. They are most effective when a model has many small, fast kernels where the launch latency is a significant fraction of total execution time. In large transformer models, this often applies to attention kernels and element-wise operations. The fact that CUDA graphs did not help implies that the GPU is spending most of its time in large, long-running kernels or in communication waits—neither of which CUDA graphs can accelerate.

For an MoE model like GLM-5-NVFP4, the dominant computation is the expert feed-forward networks. With 256 experts and only 8 activated per token, the model must route each token to the appropriate GPUs, compute the expert outputs, and then combine them. In a tensor-parallel configuration (TP8), every token's computation is split across all 8 GPUs, requiring an all-reduce operation after every transformer layer. With 61 layers, this means 61 all-reduce operations per forward pass. On a PCIe-only topology, each all-reduce must traverse the PCIe bus, which has a theoretical bandwidth of ~32 GB/s per direction (PCIe Gen5 x16) compared to the ~900 GB/s of NVLink. The all-reduce latency dominates the per-token time.

The assistant's hypothesis is therefore that the MoE expert computation combined with PCIe all-reduce is the bottleneck. This is a well-informed conclusion: it correctly identifies that the communication-to-computation ratio for this model on this hardware is unfavorable, and that no amount of kernel launch optimization can fix it.

The Pivot: Exploring New Tuning Parameters

Having identified the bottleneck, the assistant does not stop at diagnosis. It immediately pivots to explore other tuning options:

Let me look into whether there are more tuning options. Let me check the --schedule-policy, --chunked-prefill-size, and DeepGemm options, and also measure single-stream latency.

This is a strategic shift. The assistant is now looking for parameters that affect the compute and communication profile rather than kernel launch overhead. Each of these options targets a different aspect:

The Bash Command: A Closer Look

The bash command in message 240 is a compound operation that does two things:

  1. Streaming test: Sends a chat completion request with stream: true and pipes the output through head -5 to verify the streaming format works. This tests the server's ability to handle streaming responses, which is critical for real-time user interaction.
  2. Timed non-streaming test: Sends a non-streaming request with time prefix to measure wall-clock latency, then pipes the response through a Python one-liner that extracts token counts from the usage field. The prompt is "Count from 1 to 20, one number per line" with max_tokens: 512, which should produce a decode-heavy workload. The command structure reveals the assistant's thoroughness: it tests both streaming and non-streaming paths, and it extracts precise token counts to compute tokens-per-second. The use of time for wall-clock measurement is important because the assistant cannot directly measure server-side latency from the client side.

Assumptions and Their Validity

Several assumptions underpin the assistant's reasoning in this message:

Assumption 1: CUDA graphs should improve throughput if kernel launch overhead were the bottleneck. This is sound. CUDA graphs are specifically designed to reduce kernel launch latency, and their lack of effect strongly suggests that launch overhead is not the dominant factor.

Assumption 2: The bottleneck is MoE expert computation + all-reduce over PCIe. This is a reasonable hypothesis but not yet proven. The assistant does not have direct measurements of PCIe bandwidth utilization or all-reduce latency. The subsequent conversation will validate this assumption through bandwidth testing and investigation of virtualization overhead.

Assumption 3: The throughput plateau is consistent across workload types. The assistant tested two workloads: 256 in/256 out (balanced) and 128 in/512 out (decode-heavy). Both showed similar throughput (~225 and ~236 tok/s respectively). This consistency supports the hypothesis that the bottleneck is fundamental to the hardware topology rather than workload-dependent.

Assumption 4: The server configuration is otherwise optimal. The assistant assumes that the current configuration (mem-fraction 0.92, CUDA graphs enabled, trtllm NSA backends, flashinfer attention backend) is near-optimal, and that further gains must come from different parallelism strategies or infrastructure changes. This assumption is reasonable given the systematic tuning already performed, but it may miss opportunities in other areas (e.g., MoE runner backend, quantization format).

Input Knowledge Required

To fully understand message 240, one needs:

  1. Knowledge of the GLM-5-NVFP4 model architecture: It is a 744B MoE with 256 experts, 61 layers, and 8 activated experts per token. The model is quantized to NVFP4 (NVIDIA FP4 format).
  2. Knowledge of tensor parallelism (TP): The current configuration uses TP8, meaning every operation is split across all 8 GPUs with all-reduce after each layer.
  3. Knowledge of CUDA graphs: A CUDA optimization that captures kernel launch sequences into a graph to reduce CPU launch overhead. Effective when many small kernels are launched sequentially.
  4. Knowledge of PCIe vs NVLink bandwidth: PCIe Gen5 x16 provides ~32 GB/s bidirectional, while NVLink provides ~900 GB/s. The system has no NVLink.
  5. Knowledge of MoE communication patterns: MoE models require all-to-all communication for expert routing in addition to the all-reduce for attention and shared layers.
  6. Knowledge of SGLang server parameters: The flags --schedule-policy, --chunked-prefill-size, and DeepGemm are SGLang-specific tuning options.

Output Knowledge Created

Message 240 produces several important pieces of knowledge:

  1. Empirical result: CUDA graphs do not significantly improve throughput for GLM-5-NVFP4 on 8x Blackwell GPUs over PCIe. This is a non-obvious result—one might expect CUDA graphs to help any transformer model—and it has practical implications for deployment decisions.
  2. Bottleneck diagnosis: The throughput ceiling of ~200-236 tok/s is attributed to MoE expert computation + PCIe all-reduce. This diagnosis guides all subsequent optimization efforts.
  3. Single-stream latency baseline: The streaming test begins to establish a single-request latency profile, which is important for understanding the user experience under low concurrency.
  4. Tuning roadmap: The identification of --schedule-policy, --chunked-prefill-size, and DeepGemm as next targets provides a structured path forward for the investigation.
  5. Methodology: The assistant demonstrates a rigorous benchmarking methodology: compare with and without the optimization, check server-side logs for confirmation, test multiple workload types, and measure both concurrent throughput and single-stream latency.

Mistakes and Incorrect Assumptions

While the assistant's reasoning is largely sound, there are potential issues:

The sample size is small. The benchmark uses 64 prompts with random input/output lengths. While this is sufficient for a rough throughput estimate, it may not capture variability due to prompt content, cache state, or scheduling dynamics. The 5% improvement from CUDA graphs could be real but masked by noise.

The assistant does not measure PCIe utilization directly. The conclusion that PCIe all-reduce is the bottleneck is inferred from the lack of CUDA graph improvement, not from direct measurement. In the subsequent conversation, bandwidth tests will confirm this inference, but at this point it remains a hypothesis.

The assistant assumes CUDA graphs are working optimally. The server log confirms cuda graph: True, but the assistant does not verify that the graphs are actually being hit for all batch sizes. If the workload's batch sizes fall outside the captured graph sizes, CUDA graphs would not be used.

The assistant does not consider other potential bottlenecks. Memory bandwidth, tensor core utilization, or CPU-side scheduling could also contribute to the throughput ceiling. The focus on PCIe all-reduce is reasonable but potentially incomplete.

The Broader Significance

Message 240 is a turning point in the session. Before this message, the assistant was in a "tuning" phase—adjusting parameters within the existing configuration to squeeze out more performance. After this message, the investigation shifts to a "rethinking" phase: questioning whether the fundamental parallelism strategy (TP8) is appropriate for this hardware topology, and whether alternative approaches like expert parallelism or data parallelism could bypass the PCIe bottleneck.

This shift is visible in the very next user message ([msg 243]), where the user asks: "Would expert-parallel be faster here given it's 8x massive gpu but on pcie / no nvlink?" The assistant's diagnosis in message 240 directly enables this question. Without the realization that PCIe all-reduce is the bottleneck, the user would not have thought to ask about expert parallelism, which changes the communication pattern from all-reduce to all-to-all.

The message also demonstrates a key skill in systems engineering: knowing when to stop tuning one parameter and pivot to another. The assistant could have spent hours trying different CUDA graph configurations, varying batch sizes, or adjusting graph capture parameters. Instead, it recognized the negative result as diagnostic information and used it to redirect the investigation toward more promising avenues.

Conclusion

Message 240 is a masterclass in diagnostic reasoning in large-scale ML inference. The assistant, after successfully enabling CUDA graphs, runs a benchmark that reveals the optimization's limited impact. Rather than dismissing the result or trying harder with the same approach, the assistant uses the negative evidence to identify the true bottleneck: PCIe-bound all-reduce in MoE expert computation. This insight reframes the entire optimization problem, shifting the focus from kernel launch overhead to communication topology. The message captures the precise moment when a "tuning" mindset gives way to a "redesign" mindset—a transition that is essential for making progress on complex systems where the obvious levers have been exhausted.