The 56 Tok/s Milestone: Validating NCCL Protocol Optimization on an 8-GPU Blackwell Inference Stack

Introduction

In the high-stakes world of large language model inference, every millisecond counts. When deploying a 744-billion parameter model (GLM-5) quantized to 4.5 bits across eight NVIDIA RTX PRO 6000 Blackwell GPUs connected only by PCIe, the difference between a usable system and an unusable one often comes down to a single environment variable. Message 2008 in this opencode session captures the exact moment when a carefully orchestrated sequence of optimizations culminated in a measurable breakthrough: single-request decode throughput reaching 56.1 tokens per second, nearly tripling the baseline of 20 tok/s achieved just hours earlier.

This message is deceptively simple — a bash command that copies a benchmark script to a remote server and runs it, followed by a table of results. But beneath this sparse surface lies the payoff of an intense debugging and optimization saga that spanned multiple sub-agents, profiler runs, and failed experiments. Understanding why this particular message was written, what assumptions it rested on, and what knowledge it produced reveals the methodical craft of productionizing cutting-edge ML inference.

The Message

The assistant writes:

Let me run the full benchmark with the current optimized config and then explore more options:

>

``bash scp /home/theuser/glm-kimi-sm120-rtx6000bw/benchmark.py root@10.1.230.174:/tmp/benchmark.py && ssh root@10.1.230.174 '/root/ml-env/bin/python3 /tmp/benchmark.py' ``

>

Warming up...

>

Benchmark results: `` -------------------------------------------------------------------------------- C= 1 | 10/ 10 ok (0 err) | wall= 22.8s | 56.1 tok/s | lat avg= 2.28 p50= 2.28 p99= 2.29 C= 2 | 10/ 10 ok (0 err) | wall= 13.8s | 92.5 tok/s | lat avg= 2.76 p50= 2.77 p99= 2.78 C= 10 | 30/ 30 ok (0 err) | wall= 26.4s | 145.6 tok/s | lat avg= 8.79 p50= 8.79 p99= 8.81 C= 64 | 192/192 ok (0 err) | wall= 77.0s | 319.2 tok/s | lat avg= 25.64 p50= ... ``

The output is truncated — the full results for C=256 and C=1024 are not shown in this message, though they were captured and analyzed in the subsequent message (msg 2009). This truncation is itself meaningful: the assistant's primary interest at this moment is the single-request (C=1) throughput, which is the headline metric for interactive use cases.

The Optimization Journey That Led Here

To understand why this message was written, one must trace the path that led to it. The session had been wrestling with the GLM-5 GGUF model on vLLM for hours. The initial baseline with --enforce-eager (which disables CUDAGraph) was a disappointing 20 tok/s — far too slow for any practical application. Profiling revealed that 87% of the ~50ms per decode step was spent in NCCL allreduce operations over PCIe, with the GPUs' massive 1792 GB/s GDDR7 bandwidth sitting largely idle at just 12% utilization.

The first breakthrough came with CUDAGraph, which doubled throughput to 43 tok/s by batching GPU kernel launches and eliminating the per-kernel dispatch overhead. But CUDAGraph could not eliminate the fundamental PCIe bottleneck — the allreduce operations still had to transfer data between GPUs, and each of the 158 allreduce calls per decode step incurred latency.

The second breakthrough came from an unexpected direction. vLLM's custom allreduce implementation (which uses shared memory and peer-to-peer copies for much lower latency) was disabled because the Blackwell GPUs (compute capability 12.0) were not in the supported list, and the PCIe-only topology triggered a safety check. After attempts to force-enable custom allreduce failed due to a C++ kernel bug on PCIe systems with more than 2 GPUs, the assistant turned to NCCL itself.

The discovery was that NCCL's default protocol (Simple) is optimized for large message throughput, not for the tiny 12KB allreduce payloads typical of transformer inference with tensor parallelism. Setting NCCL_PROTO=LL — the Low-Latency protocol — reduced allreduce latency by 34%, boosting single-request throughput from 43 tok/s to approximately 57 tok/s. Message 2008 is the rigorous validation of that finding.## Why This Message Was Written: The Need for Rigorous Validation

Message 2008 was not written casually. It represents a deliberate checkpoint in a systematic optimization workflow. The assistant had just received the result from the NCCL_PROTO=LL experiment (msg 2007), which reported "57 tok/s is a significant step up from 43." But before declaring victory, the assistant needed to do two things: first, confirm that the optimization was real and reproducible under a proper benchmark (not just a single request), and second, establish a new baseline against which further optimizations could be measured.

The choice to run the full benchmark suite — testing concurrency levels from 1 to 1024 — reveals a sophisticated understanding of performance characterization. A single-request measurement can be misleading; the true behavior of an inference system only emerges under varying load. The assistant was particularly interested in whether the NCCL_PROTO=LL optimization, which reduces per-allreduce latency, might have negative side effects at higher concurrency where the Simple protocol's higher bandwidth could become advantageous.

The phrase "and then explore more options" in the message preamble is equally telling. The assistant was not satisfied with 56 tok/s. The target was 100 tok/s, and the gap of ~44 tok/s demanded explanation. This message was the launching pad for the next round of investigation — the results would determine whether the remaining gap was fundamental (bounded by hardware) or whether further software optimizations could close it.

Assumptions Embedded in the Benchmark

Several assumptions are baked into this benchmark that are worth examining. First, the benchmark uses a single prompt ("Write a detailed step-by-step guide on how to build a basic web application using Python and Flask...") with 128 max tokens. This is a long, cache-friendly prompt that minimizes the impact of prompt processing time and measures pure decode throughput. The assumption is that decode performance is the primary bottleneck for interactive applications, which is reasonable but not universal — workloads with very short prompts and long generations would see different characteristics.

Second, the benchmark uses a single client thread per concurrency level, relying on Python's ThreadPoolExecutor. This assumes that network latency and HTTP overhead are negligible compared to model inference time, which holds for localhost communication but might not generalize to production deployments with real network latency.

Third, the benchmark measures throughput in tokens per second averaged over the entire wall-clock time, including both prompt processing and generation. For the 128-token generation, prompt processing is amortized, but at very short generation lengths it could become a significant fraction. The assistant implicitly assumes that the prompt processing overhead is small enough to ignore for this measurement.

Fourth, and most critically, the benchmark assumes that the server is in a steady state — that CUDAGraph has been captured, that the NCCL communicators are initialized, and that no cold-start effects remain. The warmup requests (two make_request() calls before the benchmark loop) are intended to ensure this, but there is no verification that CUDAGraph replay is actually occurring for every request.

What the Results Reveal

The benchmark results tell a nuanced story. At C=1, the throughput is 56.1 tok/s, consistent with the ~57 tok/s reported from the earlier single-request test. The latency is a remarkably consistent 2.28 seconds per request (p50 = p99), indicating that the CUDAGraph is delivering deterministic execution times with no outliers.

At C=2, throughput jumps to 92.5 tok/s — nearly double the single-request rate. This is the expected behavior for a system where batching improves GPU utilization: with two concurrent requests, the model can process them together, sharing the weight read overhead and amortizing the NCCL allreduce latency across both sequences. The fact that throughput nearly doubles (rather than increasing by a smaller factor) suggests that the GPU compute resources were heavily underutilized at single-request concurrency.

At C=10, throughput reaches 145.6 tok/s, but the scaling is sublinear relative to C=2. This is where the system begins to saturate some resource — likely the PCIe bandwidth for allreduce, or the GPU memory bandwidth for weight reads. The latency increases to 8.79 seconds, reflecting queueing delays as requests wait for the GPU to finish processing previous batches.

The truncated results for C=64, C=256, and C=1024 would later show throughput plateauing around 300-320 tok/s, confirming that the system is fundamentally bandwidth-limited. The key insight is that the NCCL_PROTO=LL optimization primarily benefits low-concurrency scenarios where allreduce latency dominates; at high concurrency, the benefit is marginal because batching already amortizes the per-step overhead.

Output Knowledge Created

This message produced several pieces of actionable knowledge. First, it established a validated benchmark result of 56.1 tok/s for single-request decode with the NCCL_PROTO=LL + CUDAGraph + TP=8 configuration. This became the new baseline for all subsequent optimization attempts.

Second, it revealed the scaling characteristics of the system: near-linear improvement from C=1 to C=2, sublinear from C=2 to C=10, and saturation beyond C=64. This scaling profile is diagnostic of a system where the bottleneck shifts from allreduce latency (at low concurrency) to memory bandwidth (at high concurrency).

Third, the consistency of the latency distribution (p50 ≈ p99) provided evidence that the system was stable and not experiencing sporadic slowdowns from garbage collection, NCCL reinitialization, or other transient effects. This was important for ruling out hidden bugs in the custom vLLM patches for GLM-5 GGUF support.

Fourth, the benchmark implicitly validated that the NCCL_PROTO=LL environment variable was correctly applied to the running server. Since the server was started with the environment variable set, and the benchmark showed the expected improvement, this confirmed that the optimization was effective in production conditions.

The Thinking Process Visible in the Message

While the message itself is terse, the reasoning behind it can be reconstructed from the surrounding context. The assistant had just completed a sub-agent task (msg 2007) that tested multiple NCCL configurations and found NCCL_PROTO=LL to be the winner. The sub-agent's report included a single throughput number (~57 tok/s), but the assistant knew that a single measurement could be noisy or misleading. The decision to run the full benchmark suite reflects a commitment to rigorous evaluation.

The assistant also knew that the benchmark results would inform the next decision: whether to continue pursuing further optimizations or to declare the system "good enough." The 56 tok/s result, while impressive relative to the 20 tok/s baseline, was still far from the 100 tok/s target. The next message (msg 2009) shows the assistant immediately pivoting to explore more aggressive approaches like fusing allreduce with RMSNorm, confirming that the benchmark served its purpose as a decision gate.

The choice to run the benchmark via scp + ssh rather than executing Python inline also reveals practical wisdom. Earlier attempts to run Python code directly through SSH had failed due to shell escaping issues with f-strings and curly braces (msg 1994-1995). Writing the script to a local file and copying it to the remote server avoided these quoting problems and ensured that the benchmark code was version-controlled and reusable.

Mistakes and Limitations

One subtle limitation of this benchmark is that it measures throughput over the entire request lifetime, including both prompt processing and generation. For a 128-token generation, the prompt processing time is amortized, but the benchmark does not separately report time-to-first-token (TTFT) or decode-only throughput. This means that the 56.1 tok/s figure slightly understates the true decode throughput, since it includes the initial prompt evaluation.

Another limitation is that the benchmark uses a single prompt. Different prompt lengths and structures could yield different throughput numbers, particularly if the model's MoE routing causes different experts to be activated. A more thorough evaluation would test multiple prompt types.

The assistant also did not verify that the server was actually using CUDAGraph during the benchmark. While the server was started without --enforce-eager, and earlier log inspection confirmed CUDAGraph compilation, there was no runtime verification that graph replay was occurring for every decode step. If CUDAGraph had fallen back to eager mode for some reason (e.g., a dynamic shape not covered by the captured graph), the benchmark results would reflect a mix of graph and eager execution.

Conclusion

Message 2008 is a milestone in the journey to productionalize GLM-5 GGUF inference on Blackwell GPUs. It captures the moment when a carefully discovered optimization — NCCL's Low-Latency protocol — was rigorously validated against a comprehensive benchmark suite, producing a confirmed 56.1 tok/s for single-request decode. This represented a 2.8x improvement over the initial 20 tok/s baseline, achieved through a combination of CUDAGraph (2.15x) and NCCL protocol tuning (1.34x).

The message's true significance lies not in the number itself, but in what it enabled: a clear-eyed assessment of the remaining gap to 100 tok/s, and a data-driven decision about which optimization avenues to pursue next. It exemplifies the methodical, measurement-first approach that distinguishes effective ML engineering from guesswork.