The Null Result That Speaks Volumes: Benchmarking vLLM Optimization Levels on an 8-GPU GLM-5 Deployment

Introduction

In the high-stakes world of large language model inference, every millisecond counts. When you're serving a 402-gigabyte Mixture-of-Experts model like GLM-5 across eight NVIDIA RTX PRO 6000 GPUs, the difference between 47 and 57 tokens per second represents not just a 22% improvement in user experience, but potentially thousands of dollars in compute efficiency over the lifetime of a deployment. Message 112 of this opencode session captures a pivotal moment in a systematic performance tuning exercise: the moment when the assistant tests whether vLLM's optimization level interacts with the NCCL communication protocol to produce meaningful throughput gains.

The Subject Message

The message in question is a single bash command executed over SSH, followed by its output:

[assistant] [bash] ssh -o StrictHostKeyChecking=no root@10.1.230.174 '/root/ml-env/bin/python3 -c "
import requests, time
print(\"=== NCCL_PROTO=LL + -O1 ===\")
# Warmup
requests.post(\"http://localhost:8000/v1/completions\", json={\"model\": \"/shared/glm5-gguf/GLM-5-UD-Q4_K_XL.gguf\", \"prompt\": \"Hello\", \"max_tokens\": 10, \"temperature\": 0}, timeout=120)

for trial in range(3):
    start = time.perf_counter()
    resp = requests.post(\"http://localhost:8000/v1/completions\", json={
        \"model\": \"/shared/glm5-gguf/GLM-5-UD-Q4_K_XL.gguf\",
        \"prompt\": \"Write a short poem about the ocean.\",
        \"max_tokens\": 128, \"temperature\": 0,
    }, timeout=120)
    elapsed = time.perf_counter() - start
    usage = resp.json().get(\"usage\", {})
    comp = usage.get(\"completion_tokens\", 0)
    print(f\"Trial {trial}: {elapsed:.2f}s, {comp} tokens, {comp/elapsed:.1f} tok/s\")
"'
=== NCCL_PROTO=LL + -O1 ===
Trial 0: 2.25s, 128 tokens, 57.0 tok/s
Trial 1: 2.22s, 128 tokens, 57.6 tok/s
Trial 2: 2.22s, 128 tokens, 57.7 tok/s

At first glance, this looks like just another benchmark run. But this message sits at a critical juncture in a broader investigation, and its result—a carefully measured "nothing changed"—is as informative as any positive finding.

The Reasoning and Motivation: Why This Message Was Written

To understand why this message exists, we must trace the thread of reasoning that led to it. The assistant had been engaged in a multi-step performance optimization campaign for a GLM-5 model deployed with vLLM across 8 GPUs. The journey began with a baseline measurement (message 99) showing approximately 47.3 tokens per second using default NCCL settings. The assistant then tested NCCL_PROTO=LL, a low-latency NCCL protocol, and discovered a dramatic improvement: throughput jumped to approximately 57.6 tok/s, a 22% gain (message 100). This was followed by a test of NCCL_PROTO=LL128 (message 105), which actually regressed to approximately 45.5 tok/s, confirming that LL was the optimal protocol for this specific hardware topology.

Having established NCCL_PROTO=LL as the clear winner, the assistant turned its attention to another tunable parameter: vLLM's --optimization-level flag. This flag controls the aggressiveness of the compilation and CUDA graph capture strategy. Higher optimization levels (O2, O3) attempt to fuse more operations and capture more of the model execution into CUDA graphs, theoretically reducing kernel launch overhead. Lower levels (O1) use simpler compilation strategies that may load faster and use less memory, but potentially leave performance on the table.

The assistant's hypothesis was straightforward: perhaps the aggressive optimization at O2 was introducing overhead—more CUDA graph captures, more compilation time, more memory pressure—that partially offset the gains from NCCL_PROTO=LL. If O1 could match O2's throughput while being simpler and more robust, that would be a valuable finding. Alternatively, if O1 was slower, it would confirm that the optimization level was doing useful work.

This question was particularly relevant because earlier in the session (message 91), the assistant had observed that the server logs showed "51 CUDA graph captures" and that CUDAGraphMode was set to "PIECEWISE" rather than "FULL" (because FULL mode wasn't supported with the Triton MLA backend). The piecewise capture mode means individual operations are captured into graphs separately rather than the entire model execution being one monolithic graph. This raised the possibility that the optimization level's impact might be muted by the constraints of the attention backend.

How Decisions Were Made

The decision to test --optimization-level 1 alongside NCCL_PROTO=LL was a natural next step in a systematic grid search over the configuration space. The assistant had already established:

  1. Baseline (default NCCL, default O2): ~47.3 tok/s
  2. NCCL_PROTO=LL (default O2): ~57.6 tok/s
  3. NCCL_PROTO=LL128 (default O2): ~45.5 tok/s The next logical cell in this grid was NCCL_PROTO=LL + O1. If the optimization level mattered, this combination would reveal it. If it didn't, the assistant could confidently drop optimization level from the list of impactful parameters and focus attention elsewhere. The methodology was consistent with earlier benchmarks: a warmup request (10 tokens) to ensure the CUDA graphs were captured and cached, followed by three timed trials of 128-token completions with temperature=0 for deterministic behavior. The prompt "Write a short poem about the ocean" was kept constant across trials. Each trial measured end-to-end latency from request submission to response receipt, including network overhead, scheduling, model execution, and token generation.

Assumptions Made

Several assumptions underpin this benchmark. First, the assistant assumed that three trials were sufficient to characterize performance. Given the tight variance (57.0, 57.6, 57.7 tok/s), this assumption appears reasonable—the measurements are consistent within about 1%. Second, the assistant assumed that the warmup request was adequate to trigger CUDA graph capture for all relevant execution paths. With 51 graph captures logged during startup, this is likely true for the common case, but there's always a risk that some edge case (e.g., a specific attention pattern or expert routing combination) wasn't captured during warmup.

Third, the assistant assumed that single-request throughput is the right metric to optimize. This is a reasonable starting point, but as the earlier concurrent benchmark (message 90) showed, the aggregate throughput scales super-linearly with concurrent requests (97.4 tok/s for 2 requests, 144.4 tok/s for 4 requests). An optimization that helps single-request latency might not help batch throughput, and vice versa. The assistant implicitly prioritized the single-user interactive use case.

Fourth, the assistant assumed that the server had fully initialized before benchmarking. The server took 475 seconds to become ready (message 111), and the benchmark was run immediately after. There's an assumption that no background processes (memory allocation, kernel compilation, cache population) were still running during the benchmark.

Mistakes and Incorrect Assumptions

The most significant "mistake" in this message is not a mistake at all—it's the null result itself. The assistant hypothesized that O1 might differ from O2, and the data clearly shows they are effectively identical (57.7 vs 57.6 tok/s). This is a correct negative finding, but it's worth examining whether the benchmark was sensitive enough to detect a real difference. With only three trials and approximately 1% variance, a difference of 2-3% would be within the noise floor. The conclusion that "optimization level doesn't matter" should be qualified: it doesn't matter for this specific model, hardware, and workload pattern.

A more subtle issue is that the assistant didn't verify whether the optimization level actually changed the compilation behavior. Did the O1 server perform fewer CUDA graph captures? Did it use different kernel configurations? The logs could have answered these questions, but the assistant didn't check. The null result could mean either (a) O1 and O2 produce the same compiled code for this model, or (b) they produce different code with equivalent performance. Distinguishing these cases would require deeper analysis.

Additionally, the benchmark measures end-to-end latency including HTTP overhead, JSON serialization, and network round-trips. For a local server (same machine), this overhead is small but non-zero. The ~2.22 seconds per request includes approximately 200ms of network and serialization overhead, meaning the true model throughput is slightly higher than reported. This doesn't affect the comparison between configurations, but it's worth noting for anyone trying to reproduce these numbers.

Input Knowledge Required

To fully understand this message, one needs knowledge across several domains:

NCCL (NVIDIA Collective Communications Library): The NCCL_PROTO environment variable selects the communication protocol used for GPU-to-GPU data transfers during tensor-parallel inference. LL (Low Latency) uses a protocol optimized for small message sizes with minimal overhead, while LL128 targets slightly larger messages. The Simple protocol (default) is the most general but has higher per-message overhead. The choice depends on the interconnect topology (NVLink vs PCIe), GPU generation, and message sizes.

vLLM Optimization Levels: vLLM's --optimization-level flag (O0, O1, O2, O3) controls the compilation strategy. Higher levels enable more aggressive CUDA graph capture, operation fusion, and torch.compile usage. However, not all backends support all levels—the Triton MLA attention backend used here doesn't support FULL CUDAGraph mode, which constrains what optimization can achieve.

GGUF Quantization: The model is stored in GGUF format with Q4_K quantization, a 4-bit quantization scheme from the GGML project. This format requires on-the-fly dequantization during inference, adding computational overhead that interacts with the memory bandwidth and compute balance.

Tensor Parallelism: With 8 GPUs and TP=8, each layer's computation is split across all GPUs. After each layer, an allreduce operation synchronizes the partial results. The NCCL protocol directly impacts the latency of these allreduce operations, which occur 78 times per token (once per layer) in a model of this depth.

Hardware Topology: The 8 RTX PRO 6000 GPUs are connected via PCIe (not NVLink), with GPUs spread across two NUMA domains. This topology makes inter-GPU communication particularly sensitive to NCCL protocol choice, as PCIe latency is higher than NVLink.

Output Knowledge Created

This message produces several valuable pieces of knowledge:

  1. Optimization level is not a significant factor for single-request throughput with this model, hardware, and NCCL configuration. The 57.7 tok/s result is indistinguishable from the 57.6 tok/s achieved with O2.
  2. The NCCL protocol dominates performance: Comparing across the grid, NCCL_PROTO=LL provides ~22% improvement over baseline, while optimization level provides 0%. This tells practitioners where to focus their tuning efforts.
  3. A validated benchmark methodology: The consistent results across three trials (variance < 1%) confirm that the measurement approach is reliable and reproducible.
  4. A baseline for further optimization: With NCCL_PROTO=LL and any optimization level, the single-request throughput is approximately 57.7 tok/s. Future optimizations (allreduce-RMS fusion, attention backend changes, quantization improvements) can be measured against this baseline.

The Thinking Process

The assistant's thinking process, visible through the sequence of messages leading to and following this one, reveals a methodical experimental approach. After discovering NCCL_PROTO=LL's benefit (message 100), the assistant didn't stop there—it immediately formulated the next hypothesis: "what about LL128?" (message 100), tested it (message 105), and got a negative result. Then it formulated: "what about optimization level?" (message 106), set up the experiment (messages 110-111), and ran the benchmark (message 112).

This pattern—hypothesize, test, analyze, iterate—is textbook experimental science. The assistant shows no attachment to any particular hypothesis; it's equally willing to report positive (LL is better), negative (LL128 is worse), and null (O1 = O2) results. Each result narrows the configuration space and informs the next experiment.

The null result in message 112 is particularly instructive because it demonstrates intellectual honesty. A less rigorous investigation might have stopped after discovering NCCL_PROTO=LL, declared victory, and moved on. Instead, the assistant continued probing, looking for interactions and second-order effects. The finding that optimization level doesn't matter is valuable precisely because it prevents future wasted effort—no one needs to spend time tuning this parameter for similar deployments.

Broader Significance

This message exemplifies a crucial principle in systems optimization: measure everything, trust nothing, and verify your assumptions. The assistant could have assumed that higher optimization levels always help, or that NCCL_PROTO=LL combined with O1 would be different from O2. Instead, it tested these assumptions rigorously.

For anyone deploying large language models with vLLM across multiple GPUs, this message carries a practical lesson: start your tuning with NCCL protocols, not compilation flags. The communication fabric between GPUs is often the bottleneck, and optimizing it yields immediate, measurable gains. Compilation optimizations, while important in other contexts, may be secondary when the interconnect is the limiting factor.

The message also demonstrates the value of systematic benchmarking infrastructure. The assistant's ability to quickly spin up a server with different flags, wait for it to initialize, run a consistent benchmark, and report results is itself a form of engineering craftsmanship. The Python benchmark script, while simple, is carefully designed: it warms up the model, uses deterministic sampling (temperature=0), measures wall-clock time with high precision (time.perf_counter()), and reports both latency and throughput. These details matter for reproducible science.

In the end, message 112 is a quiet but important moment in the conversation. It doesn't announce a breakthrough or a new record. Instead, it says: "We tested this, and it doesn't matter. Let's move on." That ability to efficiently eliminate variables is what separates systematic optimization from guesswork.