Testing the Counterintuitive: Does Less Compilation Yield Faster Inference?

"Now let me try with NCCL_PROTO=LL but with --optimization-level 1 to see if the less aggressive compilation/cudagraph helps"

This single sentence, embedded in a bash command that launches a vLLM inference server, captures a moment of genuine engineering curiosity. The message (index 110) in this opencode session is deceptively simple — it is one tool call, one bash command, one server launch. But the reasoning behind it reveals a sophisticated understanding of the vLLM inference stack, a systematic experimental methodology, and a willingness to question the assumption that "more optimization is always better."

The Message

NCCL_P2P_LEVEL=SYS NCCL_PROTO=LL nohup /root/ml-env/bin/python3 -m vllm.entrypoints.openai.api_server \
  --model /shared/glm5-gguf/GLM-5-UD-Q4_K_XL.gguf \
  --tokenizer zai-org/GLM-5 --hf-config-path zai-org/GLM-5 \
  --tensor-parallel-size 8 --dtype float16 \
  --max-model-len 8192 --gpu-memory-utilization 0.90 \
  --trust-remote-code --port 8000 --disable-log-requests \
  --optimization-level 1 \
  > /tmp/vllm_O1.log 2>&1 &
echo "Server PID: $!"

At first glance, this looks like a routine server restart with a single new flag. But the --optimization-level 1 flag is the key variable in an experiment designed to answer a non-obvious question: could reducing the aggressiveness of vLLM's compilation pipeline actually improve single-request decode throughput for a 402-billion-parameter GGUF-quantized MoE model running across 8 RTX PRO 6000 Blackwell GPUs?

The Context: A Systematic Performance Investigation

To understand why this message was written, we must trace the experimental thread that led to it. The preceding messages (indices 86–109) document a disciplined performance tuning session. The assistant had already established a baseline throughput of approximately 47.3 tok/s using default NCCL settings. Through systematic experimentation, it discovered that setting NCCL_PROTO=LL (Low-Latency protocol) boosted throughput to 57.6 tok/s — a 22% improvement. It also tested NCCL_PROTO=LL128, which regressed to 45.5 tok/s, confirming that LL was the optimal NCCL protocol for this particular PCIe topology and model configuration.

But the assistant wasn't satisfied. The aggregate throughput scaling tests (messages 90–91) revealed something important: with 2 concurrent requests, aggregate throughput reached 97.4 tok/s, and with 4 concurrent requests, it hit 144.4 tok/s. This scaling behavior indicated that the model had significant idle capacity during single-request decode — the GPU wasn't being fully utilized. The per-request throughput dropped from 57.6 tok/s (single request) to 48.7 tok/s (2 concurrent) to 36.1 tok/s (4 concurrent), but the aggregate throughput increased substantially. This pattern suggested that the bottleneck wasn't raw compute capacity but rather some form of serialization overhead or pipeline inefficiency.

The assistant then examined the vLLM logs (message 91) and discovered that the server was using CUDAGraphMode in PIECEWISE mode (not FULL mode, which was unsupported with the Triton MLA backend), performing 51 CUDA graph captures, and loading AOT-compiled models from cache. This raised an important question: was the aggressive compilation pipeline — designed to optimize performance — actually introducing overhead that hurt single-request latency?

The Hypothesis: Optimization as Overhead

The decision to test --optimization-level 1 reflects a nuanced understanding of vLLM's compilation architecture. vLLM uses several layers of optimization:

  1. Torch AOT Compilation: Ahead-of-time compilation of model graphs using torch.compile, which can fuse operations and reduce Python overhead.
  2. CUDAGraph Capture: Recording CUDA operations into replayable graphs that eliminate kernel launch overhead.
  3. Triton Kernels: Custom GPU kernels for operations like attention (the MLA backend in this case). The --optimization-level flag controls how aggressively vLLM applies these optimizations. Higher levels (default is typically 2 or 3) enable more CUDAGraph captures, more aggressive fusion, and more comprehensive AOT compilation. Level 1 is a lighter touch — it still uses CUDAGraphs and compilation but with fewer captures and less aggressive fusion. The assistant's hypothesis was that for this particular model — a 402GB GGUF-quantized mixture-of-experts architecture — the aggressive compilation might be counterproductive. GGUF quantization involves significant dequantization overhead at runtime (converting quantized weights back to float for computation). This dequantization is inherently data-dependent and may not benefit from graph fusion in the same way that pure float16 computation would. Moreover, the MoE architecture with its routing logic and sparse expert activation might create execution patterns that are poorly captured by CUDAGraphs, leading to frequent graph recompilation or fallback to eager mode.

The Methodology: Controlled Experimentation

The assistant's approach exemplifies good experimental practice in systems optimization. Each server launch was a controlled experiment with a single variable change:

Assumptions and Their Validity

The message rests on several assumptions, some explicit and some implicit:

Assumption 1: Optimization level is a meaningful tuning parameter for this model. This is reasonable — vLLM's optimization levels are designed to trade off compilation overhead against runtime performance. However, the assistant implicitly assumes that level 1 might reduce overhead without catastrophically increasing per-step latency. This is a plausible but unverified hypothesis.

Assumption 2: The NCCL_PROTO=LL setting should be retained. Having established LL as the best protocol, the assistant correctly holds it constant while testing the new variable. This is sound experimental design — change one thing at a time.

Assumption 3: The server will start successfully with these parameters. This is a practical assumption, but not guaranteed. The previous server with NCCL_PROTO=LL started successfully, but the addition of --optimization-level 1 could theoretically trigger different code paths that expose bugs or incompatibilities.

Assumption 4: The measurement methodology (single request, 128 tokens, same prompt) is sufficient to characterize the effect. The assistant had already established that single-request throughput was the metric of interest, and that the benchmark was reproducible (trials showed minimal variance in previous experiments).

One potential blind spot: the assistant is testing optimization level 1 while keeping NCCL_PROTO=LL. But the interaction between NCCL protocol and compilation strategy is unknown. It's possible that the optimal compilation strategy differs depending on the NCCL protocol, or that the benefits of LL protocol are only realized with aggressive compilation. The experiment will reveal this, but the assistant hasn't explicitly considered this interaction in its reasoning.

Knowledge Required to Understand This Message

To fully grasp the significance of this message, a reader needs:

  1. Understanding of vLLM's architecture: Knowledge that vLLM uses CUDAGraphs, AOT compilation, and Triton kernels for inference acceleration, and that --optimization-level controls the aggressiveness of these optimizations.
  2. Knowledge of NCCL protocols: Understanding that NCCL_PROTO=LL (Low-Latency) uses a different communication protocol optimized for small message sizes, which is relevant for the allreduce operations in tensor-parallel inference.
  3. Awareness of GGUF quantization: Understanding that GGUF is a format for quantized models (Q4_K_XL in this case), and that dequantization overhead is a significant factor in inference performance.
  4. Familiarity with MoE architectures: Knowledge that mixture-of-experts models have routing logic and sparse expert activation that create irregular computation patterns.
  5. Understanding of the experimental context: The preceding 24 messages of performance tuning that established baselines and identified NCCL_PROTO=LL as the best protocol.
  6. Knowledge of the hardware topology: 8 RTX PRO 6000 Blackwell GPUs connected via PCIe, with specific NUMA node configurations that affect inter-GPU communication latency.

Knowledge Created by This Message

This message itself doesn't produce results — it launches an experiment. The knowledge it creates will be revealed in the subsequent messages (indices 111+), where the assistant measures the throughput with optimization level 1 and compares it to the previous results. However, the message creates knowledge in several indirect ways:

  1. It documents the experimental hypothesis: The explicit reasoning ("to see if the less aggressive compilation/cudagraph helps") captures the assistant's theory about where the bottleneck might lie.
  2. It establishes the experimental protocol: The exact command line serves as a reproducible recipe for future experiments.
  3. It demonstrates a systematic approach: The message shows that the assistant is methodically exploring the parameter space, changing one variable at a time while holding others constant.
  4. It reveals the assistant's mental model: The assistant believes that compilation overhead might be a significant factor in single-request latency, and that reducing it could improve throughput — a counterintuitive but defensible hypothesis.

The Thinking Process

The reasoning visible in this message and its surrounding context reveals a sophisticated debugging methodology. The assistant is not randomly trying flags; it's building a mental model of the system's performance characteristics and testing specific hypotheses derived from that model.

The chain of reasoning goes like this:

  1. Observation: Single-request throughput is 57.6 tok/s with NCCL_PROTO=LL.
  2. Analysis: Aggregate throughput scales well with concurrent requests (97.4 tok/s at 2x, 144.4 tok/s at 4x), suggesting the GPU has idle capacity during single-request decode.
  3. Hypothesis generation: If the GPU is idle during decode, the bottleneck might be in the software pipeline rather than compute. Potential sources include NCCL communication, CUDAGraph capture overhead, or compilation inefficiency.
  4. Hypothesis testing (NCCL): Tested NCCL_PROTO=LL (improved), NCCL_PROTO=LL128 (regressed), established LL as optimal.
  5. Hypothesis testing (compilation): Now testing --optimization-level 1 to see if reducing compilation aggressiveness reduces overhead. The assistant is effectively performing a binary search on the performance parameter space, first identifying the most impactful parameter (NCCL protocol), then moving to the next candidate (optimization level). This systematic approach is characteristic of experienced systems engineers who understand that performance optimization is an empirical science, not a theoretical exercise.

Conclusion

Message 110 is a small but significant step in a larger journey of performance optimization. It represents the moment when the assistant, having exhausted the obvious tuning parameters (NCCL protocols), turns to a more subtle and counterintuitive hypothesis: that the very machinery designed to make inference faster might be introducing overhead that hurts single-request latency. Whether this hypothesis proves correct or not, the message stands as a testament to the value of systematic experimentation, controlled variable isolation, and the willingness to question assumptions about what "optimization" means in complex software systems.

The result of this experiment — whether --optimization-level 1 improves, degrades, or leaves throughput unchanged — will provide valuable insight into the interaction between vLLM's compilation pipeline and the unique characteristics of large-scale GGUF-quantized MoE models. And regardless of the outcome, the methodology itself is a model for how to approach performance tuning in modern ML inference stacks.