The Benchmark That Changed Everything: Proving Compute-Bound Inference on Blackwell GPUs

In the high-stakes world of large language model inference, every millisecond counts. When you're deploying a 405-billion-parameter model like GLM-5-NVFP4 across eight NVIDIA RTX PRO 6000 Blackwell GPUs, the difference between a well-tuned system and a mediocre one can be measured in thousands of tokens per second. This article examines a single, pivotal message in an opencode coding session — message index 857 — where the assistant delivered the final, conclusive benchmark results that shattered a key assumption and redirected the entire optimization effort.

The Message in Full

The subject message reads:

[assistant] 256 concurrency: 370 output tok/s, 727 total tok/s. TP8 was 622 output tok/s, 1,867 total tok/s — still much worse. Let me try 512: [bash] ssh root@10.1.230.174 'source /root/ml-env/bin/activate && python3 -m sglang.bench_serving \ --backend sglang \ --host 127.0.0.1 --port 8000 \ --model glm-5 \ --tokenizer lukealonso/GLM-5-NVFP4 \ --dataset-name random \ --random-input-len 128 --random-output-len 128 \ --num-prompts 512 --request-rate 999 \ 2>&1 | tail -30' Total input text tokens: 32489 Total generated tokens: 32671 Total generated tokens (retokenized): 32647 Request throughput (req/s): 10.36 Input token throughput (tok/s): 657.13 Output token throughput (tok/s): 660.81 Peak output token throughput (tok/s): 1276.00 Peak concurrent requests: 512 Total token throughput (tok/s): 1317.94 Concurrency: ...

At first glance, this looks like just another benchmark run — a routine measurement in a long optimization session. But this message represents a critical inflection point. It is the moment when a carefully constructed hypothesis collapsed under the weight of empirical evidence, forcing a fundamental rethinking of where the true bottleneck lay.

The Context: A Hypothesis About Communication

To understand why this message matters, we must understand the reasoning that led to it. The session's hardware setup was unusual: eight RTX PRO 6000 Blackwell GPUs spread across two NUMA sockets in a Proxmox virtualized environment. Earlier analysis had revealed that cross-socket GPU communication suffered from severely degraded P2P bandwidth — approximately 40 GB/s compared to the 54 GB/s available within a single socket. The virtualization layer imposed additional latency penalties.

The conventional wisdom in LLM inference is that allreduce communication — the collective operation that synchronizes gradients and activations across GPUs during tensor parallelism — is often the primary bottleneck, especially when GPUs span NUMA boundaries. The assistant and user had discussed this extensively in preceding messages (<msg id=835-836>). The proposed solution was Pipeline Parallelism (PP): split the model's 78 layers in half, assign layers 0-38 to GPUs 0-3 (socket 0) and layers 39-77 to GPUs 4-7 (socket 1), with each group using 4-way tensor parallelism internally. This TP4+PP2 configuration would eliminate cross-socket allreduce entirely — each allreduce would stay within its NUMA-local group of four GPUs — and replace it with a single inter-stage activation transfer per layer boundary.

The expected outcome, based on prior research with a similar model (GLM-4.7-FP8), was a 23% throughput improvement. The assistant invested significant effort in setting up this configuration: verifying that the model architecture supported PP ([msg 838]), fixing argument parsing issues ([msg 843]), writing a launch script ([msg 847]), and monitoring the server startup across both PP stages (<msg id=849-851>).

The Systematic Unfolding of Disappointment

The assistant's benchmarking methodology was methodical. After confirming the server was running and producing coherent output ([msg 852]), the assistant ran a single-stream test ([msg 854]) which yielded 5.6 output tok/s — worse than TP8's ~8-11 tok/s. This was expected: pipeline parallelism introduces "bubble" overhead where one stage waits while the other computes, hurting single-stream latency. The real test was at higher concurrency, where the pipeline should fill and throughput should recover.

At 64 concurrency ([msg 855]), TP4+PP2 achieved 104 output tok/s and 213 total tok/s. TP8 had achieved 200 output tok/s and 700 total tok/s. The TP4+PP2 configuration was roughly half as fast — and the gap in total throughput was even larger (213 vs 700), suggesting the prefill phase was particularly penalized.

At 256 concurrency ([msg 856]), the results were 370 output tok/s and 727 total tok/s, compared to TP8's 622 output tok/s and 1,867 total tok/s. The pattern held: roughly 2x slower.

Then comes message 857, the subject of this article. The assistant reports the 256 result with a tone of resignation — "still much worse" — and pushes on to 512 concurrency, hoping that perhaps at extreme batch sizes the pipeline bubble would be amortized enough for TP4+PP2 to catch up. The 512-concurrency result: 660.81 output tok/s and 1,317.94 total tok/s. TP8 at 512 concurrency had achieved approximately 2,800 total tok/s (from earlier benchmarks not shown in this message but referenced in the todo list). The gap persisted.## The Deeper Implication: A Compute-Bound Revelation

The numbers in message 857 told a devastating story, but the real insight was in what they implied. The assistant's subsequent analysis in message 859 — responding to the user's astute observation "Doesn't the halving of perf sort of imply we're compute bound -> kernels not tuned to the GPU?" — laid out the logic with crystalline clarity.

With TP8, each GPU computes 1/8th of each layer's matrix multiplications. With TP4+PP2, each GPU computes 1/4th of each layer — double the per-GPU compute — but only half the layers. The total compute per token per GPU is the same. The key difference is that the matrix dimensions per GEMM operation are twice as large under TP4.

If the bottleneck had been allreduce communication, halving the allreduce participants from 8 to 4 and keeping them NUMA-local should have improved throughput. Instead, throughput roughly halved. The ~2x slowdown directly correlated with the 2x larger GEMMs per GPU. This was the signature of a compute-bound system: the GPUs were spending their time computing, not communicating, and making each individual computation larger (by reducing TP degree) made things worse, not better.

This was a pivotal realization. The entire optimization strategy up to this point had been built on the assumption that cross-socket allreduce was the primary bottleneck. The assistant had spent hours setting up the LXC container to bypass virtualization overhead (<msg id=839-851> of earlier segments), tuning NCCL parameters, and investigating P2P DMA workarounds — all under the premise that communication was the enemy. Message 857, combined with the user's insight in message 858, proved that assumption wrong.

The Thinking Process Visible in the Message

The subject message reveals a clear thinking process through its structure. The assistant does not simply present the 512-concurrency benchmark in isolation. It first reports the 256-concurrency result, explicitly comparing it to the TP8 baseline: "370 output tok/s, 727 total tok/s. TP8 was 622 output tok/s, 1,867 total tok/s — still much worse." The phrase "still much worse" is telling — it shows the assistant was hoping, perhaps against accumulating evidence, that higher concurrency would eventually favor the TP4+PP2 configuration. The word "still" implies a sequence of disappointments: single-stream was worse, 64 was worse, 256 was worse, and now 512 was also worse.

The decision to push on to 512 concurrency despite three consecutive negative data points reveals a methodical, scientific mindset. The assistant was not looking for confirmation of a preferred hypothesis; it was stress-testing the configuration to find its breaking point. Would TP4+PP2 ever catch up? The 512 result answered that question definitively: no. At 512 concurrency, TP4+PP2 achieved 1,318 total tok/s. TP8 at similar concurrency had achieved roughly 2,800 total tok/s — still more than double.

The assistant's choice of benchmark parameters is also noteworthy. Using --random-input-len 128 --random-output-len 128 with --request-rate 999 (effectively infinite request rate, saturating the server) created a controlled, reproducible workload. The 128-token input and output lengths are short enough to run many iterations quickly while being long enough to represent realistic chatbot interactions. The --num-prompts 512 parameter ensured the server was fully saturated with concurrent requests, maximizing throughput and minimizing the impact of pipeline bubbles.

Input Knowledge Required

To fully understand this message, one needs substantial background knowledge. The reader must understand the concept of tensor parallelism (TP) — splitting individual matrix multiplications across GPUs — and pipeline parallelism (PP) — splitting layers across GPU groups. The reader must know what allreduce is and why it can become a bottleneck in multi-GPU inference. The reader must understand the hardware topology: two NUMA sockets, each with four GPUs connected via NVLink/NVSwitch, with cross-socket communication going through the PCIe fabric and virtualization layer.

The reader must also be familiar with the SGLang serving framework and its bench_serving tool, the GLM-5-NVFP4 model architecture (a mixture-of-experts transformer with FP4 quantization), and the Blackwell SM120 microarchitecture. The benchmark parameters — random-input-len, random-output-len, num-prompts, request-rate — are domain-specific terminology from LLM serving.

The numerical comparison to TP8 results requires knowledge of those earlier benchmarks, which were conducted in previous segments of the conversation. The assistant is implicitly referencing a shared mental model of the baseline performance.

Output Knowledge Created

This message produced concrete, actionable knowledge. It established that TP4+PP2 on this hardware achieves approximately 660 output tok/s and 1,318 total tok/s at 512 concurrency — roughly half the throughput of TP8. This was a definitive empirical result that falsified the hypothesis that allreduce was the primary bottleneck.

The message also implicitly validated a new hypothesis: that the model is compute-bound, meaning the FP4 GEMM kernels on SM120 are the limiting factor. This redirected the entire optimization effort toward kernel tuning, CUTLASS autotuning, shared memory analysis, and expert parallelism — all of which became the focus of subsequent messages in the session. The assistant's todo list in message 860 was updated to reflect this new direction: "TP4+PP2 result: ~2x SLOWER than TP8 — confirms compute-bound, not comm-bound" was marked completed, and "Investigate FP4 GEMM kernel efficiency on SM120" was added as the next priority.

Mistakes and Incorrect Assumptions

The primary incorrect assumption was that cross-socket allreduce latency was the dominant bottleneck. This assumption was reasonable given the earlier P2P bandwidth measurements showing degraded cross-socket performance, and it was supported by prior research on a similar model (GLM-4.7-FP8) where TP4+PP2 had yielded a 23% improvement. However, the GLM-5-NVFP4 model is larger (405B vs ~200B parameters) and uses FP4 quantization, which may have different compute characteristics.

A secondary assumption was that the pipeline bubble overhead would be amortized at high concurrency. The benchmarks showed that even at 512 concurrent requests — far beyond typical production loads — the TP4+PP2 configuration could not match TP8. The pipeline bubble, combined with the larger per-GPU GEMM sizes, created an insurmountable gap.

There was also an implicit assumption that the SGLang framework's PP implementation was fully optimized for this model and hardware. While the assistant verified that the model architecture supported PP ([msg 838]), there may have been unaccounted overheads in the inter-stage communication, synchronization, or memory management that penalized PP more than expected.

The Turning Point

Message 857 is, in retrospect, the turning point of the entire optimization session. Before this message, the assistant was pursuing a communication-optimization strategy: reduce cross-socket traffic, improve P2P bandwidth, use PP to localize allreduce. After this message, the assistant pivoted to a compute-optimization strategy: analyze FP4 GEMM kernel efficiency, tune CUTLASS tactics, investigate shared memory limits, explore expert parallelism, and ultimately achieve a 28% throughput improvement through server parameter tuning (as documented in the chunk summary for segment 7).

The message demonstrates a crucial principle in systems optimization: always benchmark your assumptions. The assistant invested significant effort in setting up TP4+PP2 — writing launch scripts, debugging argument parsing, monitoring server startup — and then rigorously benchmarked it across multiple concurrency levels. When the results contradicted the hypothesis, the assistant accepted the evidence and pivoted. This scientific rigor, combined with the user's sharp analytical insight, turned a failed experiment into a valuable discovery that ultimately led to significant performance improvements.