Benchmarking GLM-5-NVFP4 on 8× RTX PRO 6000 Blackwell: The Moment of Truth

The Message

[assistant] Works. Now benchmark:
[bash] ssh 10.1.230.175 'source ~/ml-env/bin/activate && python3 -m sglang.bench_serving --backend sglang --base-url http://localhost:8000 --model glm-5 --tokenizer lukealonso/GLM-5-NVFP4 --dataset-name random --random-input-len 256 --random-output-len 256 --num-prompts 64 --request-rate inf --disable-tqdm --disable-stream 2>&1 | grep -E "Output token|Total token|Concurrency:|Request thro|Successful|duration"'
#Output tokens: 7059
Successful requests:                     64        
Benchmark duration (s):                  34.33     
Request throughput (req/s):              1.86      
Output token throughput (tok/s):         205.63    
Total token throughput (tok/s):          472.63    
Concurrency:                             34.89

Context and Motivation

This message represents a pivotal moment in a long debugging session. The assistant had been wrestling for hours with deploying the GLM-5-NVFP4 model — a massive Mixture-of-Experts (MoE) language model with 256 experts — on a server equipped with 8 NVIDIA RTX PRO 6000 Blackwell GPUs. The journey to this benchmark was anything but straightforward.

The conversation leading up to this message (see [msg 280] through [msg 301]) reveals a multi-layered debugging effort. The team had been battling a critical NaN crash during decode that rendered the model unusable. This crash was eventually resolved by selecting specific Non-Scalable Attention (NSA) backends: --nsa-decode-backend trtllm and --nsa-prefill-backend trtllm. These flags forced the attention computation to use TensorRT-LLM's implementation rather than FlashInfer's default, which apparently had issues on the SM120 architecture of the RTX PRO 6000.

But the NSA backend fix was only one piece of the puzzle. The assistant had also been systematically exploring MoE runner backends — the component responsible for routing tokens through the model's 256 experts. Three options were available: flashinfer_trtllm (which was discovered to be SM100-only and thus incompatible with the SM120 GPUs), flashinfer_cutlass (using CUTLASS kernels), and flashinfer_cutedsl (using CuteDSL, a newer kernel authoring framework). The assistant had tested flashinfer_cutlass in earlier messages and was now verifying flashinfer_cutedsl as the final alternative.

The message opens with "Works. Now benchmark:" — a terse but significant declaration. The assistant had just sent a quick correctness test (a simple "What is 3+5?" query) which returned "3 + 5 = 8" with proper reasoning content. This confirmed that the flashinfer_cutedsl MoE backend, combined with the trtllm NSA backends, produced coherent output. The model was finally functional after hours of debugging. The next logical step was to measure how well it performed.

Why This Message Was Written

The assistant wrote this message to establish a baseline throughput measurement for the working configuration. After resolving the NaN crash and verifying correctness, the natural progression was to quantify performance. This serves multiple purposes:

  1. Validation: The benchmark confirms that the configuration is not just functionally correct but also practically usable. A model that generates text but at 1 token per second is useless for production; the benchmark provides objective evidence of viability.
  2. Comparison point: Without a baseline, it's impossible to evaluate the impact of future tuning changes. The assistant had been adjusting parameters like --mem-fraction-static (increased to 0.92), enabling CUDA graphs, and trying different MoE backends. Each change needed to be measured against a consistent benchmark.
  3. Diagnostic signal: The benchmark numbers themselves tell a story. The concurrency of 34.89 (out of 64 requests) suggests the system is not fully saturated — requests are completing faster than new ones are being dispatched. The ratio of total tokens to output tokens (~2.3:1) indicates prefilling overhead is significant.
  4. Decision support: The user had raised the possibility of expert parallelism (EP) to address PCIe-bound performance. The benchmark results would inform whether such architectural changes were warranted.

The Benchmark Design

The assistant chose specific benchmark parameters that reveal deliberate thinking:

Input Knowledge Required

To fully understand this message, one needs knowledge of:

  1. SGLang's architecture: The bench_serving module is a benchmarking utility built into SGLang. It sends requests to a running server and measures throughput, latency, and concurrency. Understanding what each metric means (output token throughput vs. total token throughput, request throughput, concurrency) is essential.
  2. The GLM-5-NVFP4 model characteristics: This is a Mixture-of-Experts model with 256 experts, using NVFP4 quantization (NVIDIA's 4-bit floating point format). It has 78 layers, each requiring multiple kernel launches. The model's architecture — particularly the expert count and hidden size — directly impacts the benchmark results.
  3. The hardware environment: 8× RTX PRO 6000 Blackwell GPUs connected via PCIe (not NVLink), running inside a Proxmox VM with KVM/QEMU virtualization. The assistant had previously discovered that cross-GPU peer-to-peer (P2P) transfers are not supported in this VM environment, forcing all communication through host memory.
  4. The debugging history: The NaN crash, the NSA backend selection, the MoE backend exploration — all of these contextualize why this particular benchmark was run with this specific configuration.
  5. The server parameters: The benchmark was run against a server launched with flags like --tp 8 (tensor parallelism across 8 GPUs), --moe-runner-backend flashinfer_cutedsl, --nsa-decode-backend trtllm, --nsa-prefill-backend trtllm, --mem-fraction-static 0.92, and --disable-custom-all-reduce. Understanding these flags is necessary to interpret the results.

Output Knowledge Created

This message creates several pieces of actionable knowledge:

  1. Baseline throughput metrics: ~205 output tokens/second and ~472 total tokens/second across 8 GPUs. This is the first reliable performance measurement for GLM-5-NVFP4 on RTX PRO 6000 hardware with a working configuration.
  2. Concurrency behavior: The concurrency of 34.89 (out of 64 requests) indicates the server is handling requests efficiently but not at full saturation. This suggests there's headroom for more requests or that the bottleneck is elsewhere.
  3. Comparison data: These numbers can be compared against future runs with different configurations (e.g., different MoE backends, different memory fractions, different attention backends) to evaluate the impact of tuning changes.
  4. Validation of the flashinfer_cutedsl backend: The fact that the benchmark ran successfully and produced reasonable throughput confirms that CuteDSL is a viable MoE backend for SM120 GPUs.
  5. Evidence for bottleneck analysis: The throughput numbers, combined with the previously observed GPU utilization (100%) and power draw (55%), paint a picture of a system that is compute-bound but not power-bound — the GPUs are fully utilized but not drawing maximum power, suggesting memory-bandwidth-limited kernels rather than compute-limited ones.

The Thinking Process Visible in the Message

While the message itself is brief — just a command and its output — the surrounding context reveals the assistant's reasoning:

The assistant had just verified correctness with a simple query ("What is 3+5?") and received a proper response with reasoning. The "Works." at the beginning of the message is a checkpoint — a confirmation that the configuration is functionally correct before proceeding to performance measurement.

The choice to benchmark immediately after correctness verification shows a systematic approach: first make it work, then make it fast. The assistant doesn't waste time on further tuning or exploration before establishing a baseline.

The benchmark command itself is carefully constructed. The assistant uses sglang.bench_serving rather than a generic HTTP benchmarking tool, which means the results are directly comparable to other SGLang deployments. The --disable-tqdm and --disable-stream flags suppress progress bars and streaming output, keeping the benchmark focused on throughput rather than latency distribution.

The grep filter extracts exactly six metrics: output tokens, successful requests, benchmark duration, request throughput, output token throughput, total token throughput, and concurrency. This is a curated set — the assistant knows which metrics matter and which are noise.

Assumptions and Potential Mistakes

Several assumptions underpin this benchmark:

  1. Random data is representative: The benchmark uses random input/output tokens. Real-world usage patterns might have different characteristics (e.g., longer inputs, shorter outputs, structured data). The benchmark assumes that random data is a reasonable proxy for actual traffic.
  2. 256-token sequences are representative: The choice of 256 input and 256 output tokens is arbitrary. Different sequence lengths would produce different throughput numbers. The model might perform differently with very short sequences (where overhead dominates) or very long sequences (where memory pressure increases).
  3. The server is in steady state: The benchmark was run shortly after server startup. There might be cold-start effects (e.g., CUDA graph compilation, JIT kernel compilation) that affect the first few requests. The benchmark aggregates all 64 requests, which might mix warm and cold performance.
  4. Single benchmark run is sufficient: The assistant runs only one benchmark iteration. There's no repetition to measure variance. A single run could be affected by system noise, thermal throttling, or other transient effects.
  5. The grep filter doesn't miss important metrics: By filtering only specific lines, the assistant might miss warning messages, errors, or other diagnostic output that could indicate problems. For example, if some requests failed silently, the grep would still show "Successful requests: 64" without revealing any issues.
  6. The flashinfer_cutedsl backend is optimal: The assistant tested this backend because it was the last remaining alternative after flashinfer_trtllm proved incompatible and flashinfer_cutlass had been tested earlier. But there's no evidence that cutedsl is actually better — it was simply the next option to try.

The Broader Significance

This benchmark represents the culmination of a significant debugging effort. The NaN crash during decode had been a blocking issue that prevented any meaningful work with the model. Resolving it required understanding the interaction between NSA backends, GPU architecture (SM120 vs. SM100), and the FlashInfer library's autotuner configuration.

The benchmark results — ~205 output tok/s — are not spectacular for 8 high-end GPUs, but they represent a functional baseline. The assistant and user would go on to analyze these numbers, identifying virtualization-induced PCIe P2P latency as a key bottleneck. The concurrency of 34.89 (below the max of 64) and the 2.3:1 ratio of total tokens to output tokens both point to a system where communication overhead and prefill latency are limiting factors.

In the larger narrative of the session, this message is the "before" picture — the baseline that all future optimizations would be measured against. It's the answer to the question: "How fast does it work now?" Without this benchmark, subsequent tuning would be aimless. With it, every change can be evaluated objectively.

Conclusion

Message [msg 302] is deceptively simple — a single benchmark command and its output. But in context, it represents the successful resolution of a complex debugging process, the establishment of a critical performance baseline, and a carefully designed measurement that would inform all subsequent optimization decisions. The assistant's systematic approach — verify correctness first, then measure performance — and the thoughtful selection of benchmark parameters reveal a methodical engineering mindset. The benchmark numbers themselves, while modest, are the first reliable data point in a journey to understand and optimize the deployment of a cutting-edge MoE model on novel GPU hardware.