The 43 Tok/s Ceiling: Analyzing a Turning Point in GLM-5 Inference Optimization

Introduction

In the long arc of deploying a massive 744B-parameter Mixture-of-Experts model (GLM-5) on eight RTX PRO 6000 Blackwell GPUs, there are moments of triumph, moments of frustration, and moments of clear-eyed assessment. Message [msg 1997] belongs firmly in the last category. It is the message where the assistant, having just received the results of a comprehensive benchmark comparing CUDAGraph-accelerated inference against the previous enforce-eager mode, steps back to evaluate where things stand. The raw numbers are in hand: single-request decode has improved from 20 tok/s to 43 tok/s — a respectable 2.15x gain — but the user's target of 100 tok/s remains stubbornly out of reach. This message is the pivot point between celebration and renewed problem-solving, between the solved mystery of garbage output and the unsolved challenge of hardware-imposed throughput limits.

The Message in Full

The assistant writes:

CUDAGraph with the full benchmark is looking good:

>

| Concurrency | enforce-eager (before) | CUDAGraph (after) | Improvement | |---|---|---|---| | 1 | 20.0 tok/s | 42.7 tok/s | 2.14x | | 2 | 38.2 tok/s | 71.5 tok/s | 1.87x | | 10 | 154.0 tok/s | 128.7 tok/s | 0.84x | | 64 | 292.9 tok/s | 303.7 tok/s | 1.04x | | 256 | 995.4 tok/s | 1065.6 tok/s | 1.07x | | 1024 | 1049.1 tok/s | 1060.7 tok/s | 1.01x |

>

CUDAGraph gives 2x improvement at low concurrency (where NCCL overhead dominates). At high concurrency the benefit is smaller because batching already amortizes kernel launch overhead.

>

We're at 43 tok/s single-request now, still below the 100 tok/s target. The remaining ~23ms per decode step is pure GPU compute (memory-bandwidth-bound GGUF matmuls through 78 layers). Let me check if we can get more performance with other approaches.

The message then appends an updated todo list marking the CUDAGraph work as completed and flagging further optimization investigations as in-progress, with options including trying TP=4 to halve NCCL traffic at the cost of doubling per-GPU memory bandwidth pressure.

The Road to This Message

To understand why [msg 1997] matters, one must understand the journey that led to it. The session had been a marathon of debugging. The GLM-5 model — a 744B-parameter MoE architecture — had been quantized to GGUF Q4_K_XL format and was being served through vLLM on eight Blackwell GPUs connected only via PCIe (no NVLink). Earlier in the segment, the model had loaded successfully but produced completely incoherent garbage output. The root cause was eventually traced to two bugs: a shard ordering issue in the GGUF dequantization layer for fused projections (fixed with a simple sorted(shard_id) in gguf.py), and a red herring involving a custom PyTorch op in the Triton MLA attention backend that the assistant had initially patched by bypassing the custom op entirely.

The shard ordering fix turned out to be the real root cause. Once it was applied, the upstream custom op path worked correctly, and CUDAGraph — which had previously produced garbage output due to the direct-call bypass — became viable again. This was confirmed in [msg 1993], where the assistant reported "Excellent! This is a major win."

With CUDAGraph working, the assistant ran the full benchmark suite in [msg 1996], producing the raw data that [msg 1997] now analyzes. The benchmark script, written to avoid shell escaping issues with f-strings in zsh, tested concurrency levels from 1 to 1024 and produced clean results with zero errors across all levels.

Deep Analysis of the Benchmark Results

The table in [msg 1997] tells a nuanced story. At concurrency 1, CUDAGraph delivers a 2.14x improvement (20.0 → 42.7 tok/s). This is exactly what the profiling in [msg 1986] predicted: with enforce-eager, each decode step spent 42% of its time (21ms out of 50.5ms) on NCCL allreduce calls — 157 separate calls per step, each taking ~135μs. CUDAGraph eliminates this overhead by capturing the entire computation graph once and replaying it, batching all kernel launches and NCCL operations into a single coordinated execution.

At concurrency 2, the improvement drops to 1.87x (38.2 → 71.5 tok/s). This is still substantial, but the diminishing returns are already visible. By concurrency 10, something remarkable happens: the CUDAGraph result (128.7 tok/s) is actually lower than the enforce-eager result (154.0 tok/s). This 0.84x ratio is the first sign that CUDAGraph is not a universal win. The assistant's explanation — "batching already amortizes kernel launch overhead" — is correct but incomplete. What's really happening is that at higher concurrency, the GPU is already saturated with compute work, and the NCCL overhead becomes a smaller fraction of total time. CUDAGraph's benefit shrinks proportionally, and in some configurations, its overhead (memory for captured graphs, reduced flexibility in scheduling) may slightly outweigh its benefits.

At concurrency 64 and above, the two approaches converge to near-identical throughput: 303.7 vs 292.9 tok/s at C=64, 1065.6 vs 995.4 at C=256, and 1060.7 vs 1049.1 at C=1024. The improvements are 1.04x, 1.07x, and 1.01x respectively — within the noise of measurement variation. This convergence tells us something fundamental about the system: at high throughput, the bottleneck is no longer NCCL or kernel launch overhead, but the raw memory bandwidth of the GPUs themselves.

The Reasoning Behind the Analysis

The assistant's thinking in [msg 1997] is a model of disciplined engineering analysis. It does three things in sequence:

First, it quantifies the gain by presenting a clean before/after comparison table. This is not just reporting numbers — it's establishing that the CUDAGraph fix was worthwhile and understanding where it helps most.

Second, it explains the pattern with a concise physical intuition: "CUDAGraph gives 2x improvement at low concurrency (where NCCL overhead dominates). At high concurrency the benefit is smaller because batching already amortizes kernel launch overhead." This explanation ties the empirical data back to the profiling results from earlier in the session, creating a coherent narrative of cause and effect.

Third, it identifies the remaining gap and characterizes it: "The remaining ~23ms per decode step is pure GPU compute (memory-bandwidth-bound GGUF matmuls through 78 layers)." This is a crucial framing. By labeling the bottleneck as "memory-bandwidth-bound," the assistant is implicitly acknowledging that this is a hardware limitation, not a software bug. The 78 layers of the GLM-5 model, each requiring quantized matrix multiplications against the GGUF weights, are fundamentally limited by how fast data can be moved from GPU memory to compute units. No amount of software optimization can exceed the memory bandwidth of eight RTX PRO 6000 Blackwell GPUs connected via PCIe.

Assumptions and Potential Blind Spots

The message rests on several assumptions that deserve scrutiny. The first is that the remaining 23ms is indeed "pure GPU compute" with no further overhead to eliminate. This is a reasonable inference from the profiling data, but it assumes that the profiling captured all relevant costs. In practice, there may be CPU-side overheads (scheduling, Python interpreter, request parsing) that become visible only at higher throughputs, or GPU kernel launch overheads that CUDAGraph didn't fully capture.

The second assumption is that the 100 tok/s target is achievable with this hardware configuration. The assistant doesn't state this explicitly, but the todo list's inclusion of "Option A: Try TP=4" suggests the assistant believes further gains are possible. However, the physics of the situation are daunting: 43 tok/s with 8 GPUs means ~5.4 tok/s per GPU. To reach 100 tok/s would require ~12.5 tok/s per GPU — a 2.3x improvement in memory-bandwidth-bound compute. This would require either significantly faster memory (impossible without new hardware) or a reduction in model size (defeating the purpose of deploying the full model).

The third assumption, implicit in the todo list, is that tensor parallelism (TP) tuning might help. Reducing TP from 8 to 4 would halve the NCCL allreduce traffic (since fewer GPUs need to synchronize) but double the per-GPU memory bandwidth requirement (since each GPU would hold larger shards of each weight matrix). Given that the current bottleneck is already memory-bandwidth-bound, reducing TP would likely worsen performance, not improve it. The assistant seems to recognize this tension by listing it as an option to investigate rather than a sure win.

The Todo List as a Decision Record

The todo list appended to [msg 1997] is worth examining as a record of the assistant's decision-making process. It shows three completed items (CUDAGraph enabled, full benchmark done, and the earlier profiling) and one in-progress item (investigating further optimizations). The pending options include:

The Broader Context: From Debugging to Optimization

[msg 1997] marks a transition in the session's narrative arc. The earlier messages in this segment were dominated by debugging — finding and fixing the garbage output bug, understanding why CUDAGraph produced incorrect results, reverting the direct-call patch. With the correctness issues resolved and CUDAGraph working, the focus shifts to pure optimization. The tone changes from "why is this broken?" to "how can we make this faster?"

This transition is visible in the language. Earlier messages used words like "diagnose," "fix," "root cause," and "bug." Message [msg 1997] uses words like "improvement," "bottleneck," "amortizes," and "optimizations." The assistant is no longer a debugger; it is a performance engineer.

The Knowledge Created

[msg 1997] creates several pieces of valuable knowledge:

  1. A quantitative benchmark of CUDAGraph vs enforce-eager across six concurrency levels, establishing the performance envelope of the current configuration.
  2. A validated explanation of why CUDAGraph helps at low concurrency but not high concurrency, tying empirical results to the NCCL profiling data.
  3. A characterization of the remaining bottleneck as memory-bandwidth-bound GGUF matmuls, setting realistic expectations for future optimization efforts.
  4. A decision framework for the next optimization steps, encoded in the todo list's options.
  5. A baseline measurement (43 tok/s single-request) against which future improvements can be compared.

Conclusion

[msg 1997] is not the most dramatic message in this session — it doesn't contain a breakthrough fix or a spectacular performance leap. But it is perhaps the most honest. It reports progress (2x improvement), acknowledges limits (still below target), and characterizes the remaining challenge in physical terms (memory bandwidth). It demonstrates the kind of disciplined engineering thinking that separates genuine optimization from wishful thinking: measure, explain, identify the bottleneck, then decide where to invest effort next.

The message also reveals something about the assistant's approach to problem-solving. It doesn't celebrate prematurely or make unfounded promises. It presents the data, explains what it means, and lays out options for the next steps. This is the voice of an engineer who has learned that the hardest problems in ML infrastructure are not the ones you can fix with a clever patch, but the ones you have to work around because they're written into the physics of the hardware.

In the end, the 100 tok/s target would prove elusive — the session would later achieve ~57 tok/s through NCCL protocol tuning, but the fundamental memory bandwidth ceiling remained. [msg 1997] is the moment when that ceiling first became visible, and the assistant chose to face it squarely rather than pretend it didn't exist.