The Pivot: How Concurrent Benchmarking Uncovered the True Bottleneck in GLM-5 Inference

Introduction

In a single, deceptively simple bash command, an AI assistant running an ML deployment session made a critical pivot that transformed the understanding of a performance bottleneck. Message 90 of this coding session contains the execution of a Python benchmark script on a remote Ubuntu 24.04 server equipped with 8 RTX PRO 6000 Blackwell GPUs, testing concurrent request throughput for the GLM-5-UD-Q4_K_XL model running under vLLM. The output reveals a dramatic finding: while single-request throughput had plateaued at approximately 57 tok/s despite extensive NCCL tuning, concurrent requests achieved 97.4 tok/s aggregate with 2 requests and 144.4 tok/s with 4 requests. This single data point shattered the assumption that NCCL communication was the primary bottleneck and revealed that the true constraint was latency-bound overhead that could be amortized through batching.

The message itself is brief:

ssh -o StrictHostKeyChecking=no root@10.1.230.174 '/root/ml-env/bin/python3 /tmp/bench.py'
2-concurrent req 0: 2.63s, 128 tokens, 48.7 tok/s
2-concurrent req 1: 2.63s, 128 tokens, 48.7 tok/s
2-concurrent aggregate: 256 tokens in 2.63s = 97.4 tok/s
4-concurrent aggregate: 512 tokens in 3.55s = 144.4 tok/s

But the implications ripple backward through dozens of prior optimization attempts and forward into a fundamentally new optimization strategy.

The Context: A Long Optimization Journey

To understand the significance of message 90, one must appreciate the exhaustive optimization work that preceded it. The assistant had spent multiple rounds attempting to squeeze more performance out of single-request decode throughput. The model in question, GLM-5, is a massive Mixture-of-Experts architecture with 78 layers, 256 routed experts (8 active per token), a hidden size of 6144, and a total of 8 tensor-parallel GPU workers. Each decode step requires approximately 156 allreduce operations (78 layers × 2 allreduces per layer), each transferring roughly 12KB of data.

The assistant had systematically tested:

Anatomy of the Message

The message consists of two parts: a command and its output. The command executes a Python script (/tmp/bench.py) that was written to the remote machine in the previous message (message 89). The script uses Python's threading module to send concurrent HTTP requests to the vLLM API server running on localhost:8000. It measures completion tokens and elapsed time for each request, then computes aggregate throughput.

The output shows three key data points:

  1. Single-request baseline (implicit from prior measurements): ~57 tok/s per request
  2. 2-concurrent requests: Each individual request slows to 48.7 tok/s (a 15% drop from the single-request rate), but the aggregate throughput jumps to 97.4 tok/s — a 1.7× improvement over single-request throughput
  3. 4-concurrent requests: Aggregate throughput reaches 144.4 tok/s — a 2.5× improvement over the single-request baseline The individual per-request slowdown (from 57 to 48.7 tok/s) is expected — the GPU must split its compute resources across multiple requests. But the aggregate gain reveals that the GPU was significantly underutilized during single-request processing. The fixed overheads of each decode step (model loading, attention computation, allreduce latency) are amortized across multiple requests when they are batched together.

The Reasoning Behind the Benchmark

The assistant's decision to run a concurrent benchmark represents a fundamental shift in optimization strategy. Up to this point, the working hypothesis was that NCCL allreduce communication was the bottleneck. The assistant had invested significant effort in tuning NCCL parameters, even going so far as to research the differences between Ring and Tree algorithms for small-message allreduce on PCIe Gen5 interconnects.

The concurrent benchmark was motivated by a specific observation: if NCCL tuning made no difference across multiple independent experiments, then either (a) NCCL was already optimally configured, or (b) NCCL was not the bottleneck. The assistant chose to test hypothesis (b) by examining whether the system could handle more work. If the GPU was truly compute-bound, adding concurrent requests would not increase aggregate throughput — each request would simply compete for the same fixed compute resources. If instead the bottleneck was latency-bound (waiting for allreduce operations to complete, or for model layers to load), then concurrent requests could fill those idle cycles.

The benchmark design itself reveals careful thought. The assistant used Python threading to send concurrent HTTP requests, measured completion tokens rather than wall-clock time for the entire script, and computed aggregate throughput as the ratio of total tokens to the maximum elapsed time across all concurrent requests. This methodology correctly captures the system's ability to process multiple requests in parallel.

Assumptions and Their Validity

Several assumptions underpin this benchmark:

Assumption 1: The vLLM server supports concurrent request batching. This is a correct assumption. vLLM's scheduler automatically batches multiple requests into a single forward pass when they arrive within the same scheduling window. The benchmark results confirm this — aggregate throughput increases with concurrency.

Assumption 2: The benchmark script correctly measures throughput. The script uses time.perf_counter() for high-resolution timing and computes aggregate throughput as total completion tokens divided by the maximum elapsed time across all concurrent requests. This is a standard and correct methodology for measuring throughput under concurrency.

Assumption 3: The server is in a steady state. The assistant had already sent warmup requests in earlier benchmarks, so the model was loaded and the GPU was initialized. This is a valid assumption.

Assumption 4: Threading in Python is sufficient for this benchmark. Python's threading is limited by the GIL for CPU-bound work, but for I/O-bound HTTP requests, threading is adequate. The benchmark is measuring server-side throughput, not client-side processing, so the Python threading overhead is negligible.

Assumption 5: The results generalize to longer sequences and different prompts. This is more questionable. The benchmark uses 128-token completions with simple prompts. Real-world workloads may have different characteristics (longer sequences, variable prompt lengths, different sampling parameters). The assistant would need to validate with more diverse workloads.

Input Knowledge Required

To fully understand this message, one needs:

  1. Knowledge of the model architecture: GLM-5 is a 78-layer MoE model with 256 experts, hidden size 6144, running in FP16 with tensor parallelism across 8 GPUs. The GGUF quantization (Q4_K_XL) means weights are stored in 4-bit format and dequantized on the fly.
  2. Knowledge of vLLM's scheduling: vLLM uses continuous batching, where the scheduler can add new requests to an ongoing batch at each iteration. This is why concurrent requests can be batched together even if they arrive at slightly different times.
  3. Knowledge of NCCL and allreduce: The prior tuning attempts targeted NCCL parameters (NTHREADS, BUFFSIZE, ALGO, PROTO, MIN_NCHANNELS) which control how the allreduce operation is performed across GPUs.
  4. Knowledge of the hardware topology: 8 GPUs connected via PCIe Gen5 x16, with two NUMA domains (4 GPUs per domain), creating a cross-NUMA hop for allreduce operations.
  5. Knowledge of the deployment setup: The model is served via vLLM's OpenAI-compatible API server on port 8000, using a GGUF model file stored at /shared/glm5-gguf/GLM-5-UD-Q4_K_XL.gguf.

Output Knowledge Created

This message produces several critical pieces of knowledge:

  1. The bottleneck is not NCCL communication: The concurrent benchmark proves that the GPU has spare capacity. If NCCL were the bottleneck, adding concurrent requests would not increase aggregate throughput because allreduce operations would still be serialized. The fact that aggregate throughput scales with concurrency means the GPU can process more tokens per second when given more work.
  2. The true bottleneck is latency-bound overhead: The 57 tok/s single-request ceiling is likely caused by fixed per-step overheads — model layer loading, attention computation setup, dequantization of GGUF weights, and the serial nature of the 78-layer forward pass. These overheads are incurred once per decode step regardless of batch size, so batching multiple requests amortizes them.
  3. The system has significant headroom: With 4 concurrent requests achieving 144.4 tok/s, and the trend suggesting further gains at higher concurrency, the system may be capable of 200+ tok/s aggregate throughput under sufficient load.
  4. Optimization strategy must pivot: Instead of tuning NCCL parameters, the assistant should focus on (a) increasing batch size through higher concurrency, (b) reducing per-step overhead through CUDAGraph compilation or kernel fusion, and (c) optimizing the GGUF dequantization path.

The Thinking Process Revealed

The assistant's reasoning, visible in the context messages leading up to this benchmark, shows a systematic and scientific approach to performance optimization. The progression is instructive:

  1. Observation: Single-request throughput is ~57 tok/s, well below theoretical estimates.
  2. Hypothesis: NCCL allreduce is the bottleneck.
  3. Experiments: Test NCCL_NTHREADS, NCCL_BUFFSIZE, NCCL_ALGO, NCCL_MIN_NCHANNELS — all show no effect.
  4. Re-evaluation: Calculate theoretical allreduce latency ceiling (640–1280 tok/s) and realize NCCL cannot be the bottleneck.
  5. New hypothesis: The bottleneck is latency-bound overhead that can be amortized through batching.
  6. Experiment: Concurrent benchmark to test this hypothesis.
  7. Conclusion: Hypothesis confirmed — aggregate throughput scales with concurrency. This is a textbook example of the scientific method applied to systems optimization. The assistant did not simply try random parameters; they formed a hypothesis, designed experiments to test it, and when the evidence contradicted the hypothesis, they formed a new one and tested that. The shift from "make a single request faster" to "process more requests concurrently" is a classic systems insight. In many systems, latency and throughput are different optimization targets. Reducing latency requires optimizing the critical path, while increasing throughput requires maximizing resource utilization. The assistant correctly recognized that the two goals might conflict and that throughput might be the more fruitful target.

Broader Implications

The results of this benchmark have implications beyond this specific deployment. They demonstrate that:

  1. NCCL tuning is not a panacea: For models with small allreduce payloads (12KB per operation), NCCL parameters have limited impact. The allreduce latency is dominated by PCIe traversal time and synchronization overhead, not by NCCL's internal buffering or threading.
  2. Concurrent benchmarking is a diagnostic tool: When single-request performance plateaus, concurrent benchmarking can reveal whether the bottleneck is compute-bound (no throughput scaling) or latency-bound (throughput scales with concurrency).
  3. GGUF quantization may introduce overhead: The Q4_K_XL quantization requires on-the-fly dequantization during the forward pass, which adds CPU/GPU overhead that may not scale well with batch size. The assistant should investigate whether this dequantization is a bottleneck.
  4. The importance of amortizing fixed costs: In deep learning inference, the fixed costs per decode step (attention computation, allreduce, layer traversal) are largely independent of batch size for small batches. Batching multiple requests is the most effective way to amortize these costs.

Conclusion

Message 90 is a turning point in this optimization session. The concurrent benchmark results — 97.4 tok/s with 2 requests and 144.4 tok/s with 4 requests — conclusively demonstrate that the system is not compute-bound but latency-bound. The assistant's pivot from NCCL tuning to concurrency testing represents a mature understanding of systems optimization: when you can't make a single request faster, make the system do more work in parallel.

The message itself is just a command and its output, but the reasoning behind it — the systematic hypothesis testing, the willingness to abandon a cherished hypothesis (NCCL bottleneck) when evidence contradicted it, and the creative leap to concurrent benchmarking — is a masterclass in performance debugging. The assistant transformed a frustrating plateau into a breakthrough insight, opening the door to dramatically higher throughput through batching, CUDAGraph compilation, and kernel fusion optimizations that would follow in subsequent rounds.