The 11 Tok/s Wall: Diagnosing PCIe-Bound MoE Throughput on 8× Blackwell GPUs

In message 242 of an extensive opencode session, the assistant reaches a pivotal moment in performance characterization. After successfully deploying the GLM-5-NVFP4 model—a 744-billion-parameter Mixture-of-Experts (MoE) transformer—across eight NVIDIA RTX PRO 6000 Blackwell GPUs, and after resolving a critical NaN crash during decode by selecting the correct TensorRT-LLM (trtllm) NSA backends, the assistant now turns to understanding the system's fundamental latency characteristics. This single message captures two experiments: a clean single-stream measurement that reveals the core bottleneck, and a flawed dual-stream test whose nonsensical output inadvertently tells its own story about the challenges of benchmarking in distributed environments.

The Single-Stream Revelation

The message opens with a clear, declarative result:

Single stream: 2048 tokens in 183.8 seconds = ~11.1 tok/s for single request. That's the single-stream decode speed — bottlenecked by MoE sequential expert dispatch + PCIe all-reduce for each token.

This is a moment of diagnostic clarity. The assistant had previously measured batch throughput at approximately 225 output tokens per second with 64 concurrent requests—respectable for a PCIe-bound deployment. But single-stream latency tells a different story. At 11 tokens per second, a single user would experience multi-second delays for even modest responses. The assistant immediately identifies the root cause: the MoE architecture requires sequential dispatch to multiple experts per token, and with eight GPUs connected only via PCIe (no NVLink), each all-reduce operation for expert aggregation must traverse the PCIe bus. This creates a latency floor that no amount of batching can eliminate for individual requests.

The reasoning here is sophisticated. The assistant has synthesized information from multiple sources: knowledge of the GLM-5 architecture (which uses MoE with multiple experts per token), the hardware topology (eight GPUs on PCIe without NVLink, as confirmed earlier in the session), and the empirical data from the just-completed test. The conclusion—that the bottleneck is "MoE sequential expert dispatch + PCIe all-reduce"—represents a genuine insight that will shape all subsequent optimization efforts.

The Dual-Stream Test and Its Hidden Bug

Having established single-stream performance, the assistant immediately designs a follow-up experiment to measure how the system behaves under two concurrent requests:

Let me also measure dual-stream (2 concurrent requests):

The test launches two background curl processes in parallel, each requesting 1024 tokens from the server on different topics (general relativity and quantum mechanics). The script captures start and end timestamps using date +%s%N (seconds since epoch concatenated with nanoseconds), waits for both requests to complete, then calculates elapsed time and combined throughput.

The output reveals a catastrophic measurement error:

Request 1: 1024 tokens, Request 2: 1024 tokens, Total: 2048 tokens
Wall time: 1771461862608ms
Combined TPS: 0.0

The wall time of 1,771,461,862,608 milliseconds corresponds to approximately 20.5 days—clearly nonsensical for what should have been a multi-minute test. The combined throughput of 0.0 tokens per second is the arithmetic consequence of dividing by this absurdly large denominator.

Anatomy of a Benchmarking Bug

What went wrong? The timing script uses a common but fragile pattern in bash. The date +%s%N command produces a very large integer (e.g., 1771461862608123456 for a timestamp of 1,771,461,862 seconds and 608,123,456 nanoseconds). The script stores this in start_time, launches background processes, waits for them, then captures end_time from a second date invocation.

The most likely explanation involves variable scoping or arithmetic issues after the wait call. In bash, background processes execute in subshells, but variables set in the parent shell should persist. However, the wait built-in can interact poorly with certain shell configurations, and the arithmetic expansion $(( (end_time - start_time) / 1000000 )) may have encountered an issue where end_time was empty or unset. If end_time evaluated to zero in the arithmetic context, the result would be -start_time / 1000000—a negative number close in magnitude to what we see. But the output shows a positive value, suggesting a different failure mode, perhaps one where the subtraction produced zero and the script fell through to print the raw timestamp.

The critical observation is that neither the assistant nor the user notices the bug. The assistant does not comment on the impossible wall time or the zero throughput. The user's follow-up question (message 243) asks about expert parallelism, completely bypassing the broken measurement. This is a realistic failure mode in fast-paced engineering work: when a tool produces obviously wrong output, the natural instinct is to mentally discard it and move on rather than debug the measurement infrastructure.

Context, Motivation, and Assumptions

This message was written after a long chain of infrastructure work spanning multiple sessions. The assistant had: installed NVIDIA drivers and CUDA toolkits on Ubuntu 24.04, resolved flash-attention build issues, upgraded the machine to eight GPUs, deployed the GLM-5-NVFP4 model using SGLang nightly builds, debugged NaN crashes by selecting trtllm NSA backends, captured CUDA graphs successfully, and established batch throughput baselines. The single-stream test was the natural next step: after proving that the server could handle concurrent load, the assistant needed to understand the latency profile for individual requests.

Several assumptions underpin this message. The assistant assumes that the date +%s%N timing approach is reliable, that background process management with wait preserves shell variable state, and that the resulting throughput calculation is trustworthy. The assistant also assumes that 2048 tokens is a sufficient sample for a stable single-stream measurement (reasonable, given the 183-second duration). The diagnosis of the bottleneck assumes that PCIe all-reduce is the dominant term—an assumption that would later be validated by deeper investigation into virtualization overhead and cross-GPU P2P latency.

Knowledge Created and Broader Significance

This message creates important output knowledge. First, it establishes the single-stream latency baseline at approximately 11 tokens per second—a critical number for any application-level reasoning about user experience. Second, it articulates a clear bottleneck hypothesis (MoE sequential expert dispatch + PCIe all-reduce) that frames all subsequent optimization work. Third, the failed dual-stream test, while producing no usable performance data, reveals the fragility of ad-hoc benchmarking scripts in distributed environments.

The broader significance lies in what this message represents: the transition from "making it work" to "understanding performance." The NaN crashes are resolved, the server is stable, CUDA graphs are captured, and now the real work of optimization begins. The 11 tok/s single-stream figure is the number that must be improved—through expert parallelism, kernel fusion, or hardware topology changes. This message marks the moment when the assistant stops being a deployment engineer and becomes a performance analyst.

Conclusion

Message 242 captures a microcosm of real-world systems engineering: a clean diagnostic insight followed by a flawed experiment, with neither the assistant nor the user catching the measurement error. The single-stream result—11 tokens per second, bottlenecked by PCIe all-reduce—is genuine and valuable. The dual-stream test is broken, but its failure is instructive. Together, they illustrate the iterative, imperfect process of understanding complex distributed ML systems, where every measurement must be questioned and every assumption tested.