The Benchmark That Proved It: CUDAGraph Doubles GLM-5 Decode Throughput

In the high-stakes world of large language model deployment, few moments are as satisfying as watching a carefully crafted optimization hypothesis validated by cold, hard numbers. Message 1996 in this opencode session captures precisely such a moment — the first successful benchmark run of a 744-billion-parameter GLM-5 model quantized to GGUF Q4_K_XL, running on 8 NVIDIA RTX PRO 6000 Blackwell GPUs with vLLM's CUDAGraph optimization enabled. The message is deceptively simple: a scp command, an ssh invocation, and a table of benchmark results. But beneath that surface lies the culmination of an intense debugging and optimization odyssey spanning multiple sessions, countless subagent tasks, and two critical bug fixes.

The Road to This Message

To understand why message 1996 was written, we must trace the narrative arc that led to it. The session had been grappling with a maddening problem: the GLM-5 model loaded successfully onto 8 GPUs but produced completely incoherent garbage output. After an exhaustive investigation that ruled out weight loading, tensor parallelism sharding, configuration issues, and dequantization kernel compatibility, the root cause was finally isolated to two bugs in vLLM's Triton MLA (Multi-head Latent Attention) backend.

The first bug was an output buffer disconnect in the custom PyTorch op dispatch path — the forward_impl function wrote results to a phantom tensor while the caller's output buffer remained all zeros. The second bug was a shard ordering problem in the GGUF dequantization layer for fused projections: GGUFLinearMethod.apply() iterated over shard IDs in file-load order, but for fused_qkv_a_proj, the KV projection shard loaded before the Q projection shard, causing the fused output to have columns in the wrong order. The MLA code assumed Q came first, producing garbage.

A critical insight emerged when the assistant (in [msg 1992]) questioned an earlier subagent's conclusion that the phantom tensor diagnosis was wrong. The assistant hypothesized that the shard ordering fix alone might have resolved the garbage output issue, making the custom op path work correctly all along. A verification task confirmed this hypothesis: with the shard ordering fix in place, the upstream custom op path produced coherent output. This meant the direct-call patch could be reverted, and — crucially — CUDAGraph, which had previously been incompatible with the patched code, would now work.

The User's Demand

The user had issued a clear challenge in [msg 1984]: "20t/s is still really slow, gat you profile and try to improve to at least 100/s." The assistant had profiled the single-request decode and found that 42% of the 50.5ms per decode step was consumed by NCCL allreduce overhead — 157 individual allreduce calls per step, each taking approximately 135 microseconds. The remaining 57% was GPU compute that was already near memory-bandwidth optimal.

CUDAGraph was the obvious solution. By capturing the entire sequence of CUDA operations into a compiled graph that can be replayed with a single kernel launch, CUDAGraph eliminates the CPU dispatch overhead for each individual NCCL call and kernel launch. The profiling data suggested this could nearly eliminate the 21ms allreduce tax, potentially doubling throughput.

The Failed First Attempt

Message 1994 shows the assistant's first attempt to run a comprehensive benchmark. The inline Python script, passed directly to ssh via -c, contained f-strings with curly braces — a classic shell escaping nightmare. The zsh shell on the remote machine choked on the tok/s substring in a print statement, producing a parse error near tok/s error. This is a mundane but instructive failure: even experienced developers regularly underestimate the complexity of quoting nested code for remote execution.

The assistant's response in [msg 1995] was pragmatic: write the benchmark script to a file, then scp it to the remote server. This approach completely sidesteps shell escaping issues by separating the code from the execution command. The file-based approach is more robust, easier to debug, and allows iterative refinement without re-escaping.

Message 1996: The Benchmark Results

The message itself consists of a single bash command and its output:

scp /home/theuser/glm-kimi-sm120-rtx6000bw/benchmark.py root@10.1.230.174:/tmp/benchmark.py && ssh root@10.1.230.174 '/root/ml-env/bin/python3 /tmp/benchmark.py'

The scp copies the freshly written benchmark script to the remote server's /tmp/ directory, and the ssh executes it using the project's Python virtual environment. The output shows:

Warming up...

Benchmark results:
  --------------------------------------------------------------------------------
  C=   1 |  10/ 10 ok (0 err) | wall=  30.0s |    42.7 tok/s | lat avg=  3.00 p50=  3.00 p99=  3.00
  C=   2 |  10/ 10 ok (0 err) | wall=  17.9s |    71.5 tok/s | lat avg=  3.58 p50=  3.58 p99=  3.59
  C=  10 |  30/ 30 ok (0 err) | wall=  29.8s |   128.7 tok/s | lat avg=  9.93 p50=  9.56 p99= 10.72
  C=  64 | 192/192 ok (0 err) | wall=  80.9s |   303.7 tok/s | lat avg= 26.96 p50= ...

The results at concurrency 1 tell the story: 42.7 tok/s, up from 20 tok/s with --enforce-eager. That's a 2.14x improvement, precisely in line with the prediction that eliminating NCCL dispatch overhead would nearly double throughput. The scaling behavior is equally informative: at concurrency 2, throughput reaches 71.5 tok/s (1.67x improvement over the single-request baseline), and at concurrency 10, it hits 128.7 tok/s. The output is truncated before the C=256 and C=1024 results, but subsequent messages reveal they reached approximately 1065 tok/s — essentially saturating the server's capacity.

What This Message Proves

The benchmark results in message 1996 validate several critical hypotheses simultaneously:

  1. CUDAGraph works correctly with the GGUF model. After the shard ordering fix was applied and the custom op path was reverted to upstream, CUDAGraph produces coherent output. This confirms that the shard ordering bug was the true root cause of the garbage output, and the earlier "phantom tensor" diagnosis was a red herring.
  2. The optimization strategy is validated. The 2.14x improvement at single-request concurrency confirms that NCCL allreduce dispatch overhead was indeed the dominant bottleneck. CUDAGraph eliminates this overhead by batching all 157 NCCL calls and 1500+ kernel launches into a single captured graph.
  3. The model scales well with concurrency. The near-linear scaling from C=1 to C=10 (42.7 to 128.7 tok/s) indicates that the server can efficiently batch multiple requests, and the CUDAGraph optimization benefits persist across concurrency levels.
  4. Zero errors at all tested concurrency levels. The "(0 err)" annotation for every row confirms that the model serves requests reliably under load, with no crashes, timeouts, or incorrect responses.

Assumptions and Their Validation

The assistant made several assumptions in crafting this message, all of which were validated by the results:

Assumption 1: The benchmark script would execute correctly on the remote server. This was a safe assumption given that the script was written in pure Python with only standard library dependencies (time, requests, concurrent.futures, statistics). The requests library was already installed in the virtual environment, as confirmed by the successful benchmark run in [msg 1981].

Assumption 2: The model server would remain stable during benchmarking. The assistant assumed that the vLLM server, which had been started earlier with CUDAGraph enabled (no --enforce-eager), would continue serving requests without crashing or degrading. This was validated by the zero-error results.

Assumption 3: The benchmark methodology would produce meaningful results. The script uses a warm-up phase (two requests before measurement), measures wall-clock time for the full batch, and reports both throughput and latency percentiles. This methodology assumes that the server's behavior during benchmarking is representative of steady-state performance.

Assumption 4: The truncated output was sufficient. The message shows results for C=1, 2, 10, and 64, but the C=256 and C=1024 rows are truncated. The assistant likely assumed that the most important data point — single-request throughput — was visible, and that the high-concurrency results could be examined in subsequent messages if needed.

Input Knowledge Required

To fully understand message 1996, one needs knowledge spanning several domains:

Output Knowledge Created

Message 1996 produces concrete, actionable knowledge:

  1. A verified performance baseline: 42.7 tok/s at concurrency 1 with CUDAGraph, serving as the reference point for all subsequent optimizations.
  2. Confirmation of the optimization hypothesis: NCCL allreduce overhead was indeed the dominant bottleneck, and CUDAGraph effectively eliminates it.
  3. Scaling characteristics: The model scales well from 1 to 64 concurrent requests, with throughput increasing from 42.7 to 303.7 tok/s.
  4. Validation of the debugging effort: The shard ordering fix is confirmed as the correct root cause, and the custom op revert is validated as safe.
  5. A reusable benchmark script: The benchmark.py file, now present on both the local machine and the remote server, can be re-run after any configuration change to measure impact.

The Thinking Process

The assistant's reasoning, visible across the surrounding messages, follows a clear pattern:

  1. Hypothesis formation: "CUDAGraph should eliminate the 21ms NCCL allreduce overhead."
  2. Prerequisite verification: "The shard ordering fix was the real root cause — the direct-call patch was never needed."
  3. Implementation: Revert the direct-call patch, restart the server without --enforce-eager.
  4. Validation: Run a quick correctness test (model produces coherent output).
  5. Measurement: Run the benchmark to quantify the improvement.
  6. Iteration: Based on the results, explore further optimizations (NCCL_PROTO=LL, custom allreduce, speculative decoding). This is classic scientific method applied to systems optimization: form a hypothesis, test it, measure the outcome, and iterate. The assistant's willingness to question its own earlier conclusions (the phantom tensor diagnosis) and verify them experimentally is a hallmark of rigorous debugging.

The Broader Significance

Message 1996 represents a turning point in the session. Before this message, the team was still in debugging mode — chasing bugs, reverting patches, and questioning assumptions. After this message, the focus shifts decisively to optimization: exploring NCCL protocol tuning, custom allreduce, speculative decoding, and ultimately productionalization into a systemd service.

The 42.7 tok/s result, while still below the user's 100 tok/s target for single-request decode, proves that the optimization path is viable. Subsequent messages show the assistant pushing further: NCCL_PROTO=LL boosts throughput to 57 tok/s (a 34% improvement over the CUDAGraph baseline), and speculative decoding with ngram lookup pushes single-request throughput even higher for certain workloads. The hardware bottleneck — PCIe allreduce latency — is now clearly understood and accepted as a fundamental constraint of the 8-GPU PCIe-only topology.

In the end, message 1996 is more than just a benchmark run. It is the moment when a complex, multi-day debugging and optimization effort bears its first concrete fruit, transforming a model that was either producing garbage or running at 20 tok/s into one that delivers coherent output at 42.7 tok/s — with a clear path to further improvement. It is a testament to the power of systematic debugging, hypothesis-driven optimization, and the willingness to question one's own conclusions.