Benchmarking the Decode-Heavy Regime: Characterizing GLM-5-NVFP4 Throughput Under Long-Generation Workloads

The Message

In this message, the assistant executes a targeted benchmark sweep designed to measure the GLM-5-NVFP4 model's performance under a decode-heavy workload profile — short input prompts paired with long output generations. The full message reads:

[assistant] Now let me also run the decode-heavy sweep (short input, long output): [bash] ssh [REDACTED_HOST] 'source ~/ml-env/bin/activate && echo "=== Decode-Heavy Benchmark (in=128, out=512) ===" && for CONC in 1 8 32 64; do echo "--- Concurrency: $CONC ---"; python3 -m sglang.bench_serving --backend sglang --base-url http://localhost:8000 --model glm-5 --tokenizer lukealonso/GLM-5-NVFP4 --dataset-name random --random-input-len 128 --random-output-len 512 --num-prompts $CONC --request-rate inf --disable-tqdm --disable-stream 2>&1 | grep -E "Successful|duration|Output token thro|Total token thro|Concurrency:"; echo; done' === Decode-Heavy Benchmark (in=128, out=512) === --- Concurrency: 1 --- Successful requests: 1 Benchmark duration (s): 21.16 Output token throughput (tok/s): 11.15 Total token throughput (tok/s): 12.95 Concurrency: 1.00

>

--- Concurrency: 8 --- Successful requests: 8 Benchmark duration (s): 49.65 Output token throughput (tok/s): ...

Context and Motivation

Message 309 arrives at a pivotal moment in the conversation. The assistant and user have spent the entire segment — spanning dozens of messages — debugging, tuning, and characterizing the deployment of the GLM-5-NVFP4 model, a massive 744B-parameter Mixture-of-Experts (MoE) language model, across eight NVIDIA RTX PRO 6000 Blackwell GPUs connected via PCIe in a Proxmox virtualized environment.

The journey to this point has been arduous. Earlier in the segment, the assistant had to resolve a critical NaN crash during decode that plagued all NSA (Non-Scalar Attention) backends except trtllm. After stabilizing the server with the working configuration — --nsa-decode-backend trtllm and --nsa-prefill-backend trtllm — the assistant established baseline throughput metrics. The initial comprehensive benchmark in [msg 308] tested the model with input length 256 and output length 256 across a range of concurrencies from 1 to 64. That sweep revealed a clear pattern: single-stream throughput was a meager ~11 output tokens per second, but throughput scaled almost linearly with concurrency, reaching ~211 output tok/s and ~485 total tok/s at concurrency 64.

But a single workload profile tells only part of the story. Real-world deployments serve a mix of request types — some with long context and short generations, others with short prompts and long, streaming responses. The assistant recognized that the 256-in/256-out benchmark might not capture the full performance envelope, particularly for the use case that matters most to interactive applications: decode-heavy workloads where the model spends most of its time in the autoregressive generation phase rather than the prefill phase.

This is the motivation for message 309. The assistant explicitly frames it: "Now let me also run the decode-heavy sweep (short input, long output)." The word "also" signals that this is a deliberate extension of the benchmarking effort — a second dimension in the characterization space. By shifting from 256/256 to 128/512, the assistant is probing how the system behaves when the decode-to-prefill ratio is dramatically different.## The Reasoning Behind the Sweep Design

The choice of parameters in this benchmark is not arbitrary. The assistant selects --random-input-len 128 and --random-output-len 512, creating a workload where the model generates four times as many tokens as it processes as input. This ratio is significant because transformer-based language models have fundamentally different computational profiles for prefill (processing the input prompt in parallel) versus decode (generating one token at a time, sequentially). In the prefill phase, the model can exploit massive parallelism across the entire input sequence. In the decode phase, each step processes a single new token, making it much harder to saturate GPU compute resources.

By testing with 128 input tokens and 512 output tokens, the assistant is deliberately stressing the decode path. The model must perform 512 sequential forward passes, each processing a single token through 78 transformer layers with 256 experts (of which only 8 are activated per token). This is the worst-case scenario for GPU utilization — small matrices, low arithmetic intensity, and high overhead from kernel launches and communication.

The concurrency sweep — testing at 1, 8, 32, and 64 concurrent requests — serves a dual purpose. First, it reveals how well the system can batch requests to amortize the per-step overhead. Second, it tests whether the decode-heavy profile changes the saturation behavior compared to the balanced 256/256 workload. The assistant is implicitly asking: does the bottleneck shift under decode-heavy conditions?

Assumptions Embedded in the Benchmark

Several assumptions underlie this benchmark design. The assistant assumes that the SGLang benchmarking tool (sglang.bench_serving) with the --dataset-name random option generates synthetic requests with the specified input and output lengths, and that these synthetic requests are representative enough to draw conclusions about real-world performance. The --request-rate inf flag assumes an infinite request rate, meaning requests are dispatched as fast as the server can handle them — this measures maximum throughput under saturation, not latency under load.

The assistant also assumes that the server configuration from [msg 305] is still running correctly. The server was launched with --mem-fraction-static 0.92, CUDA graphs enabled, flashinfer_cutlass MoE runner, and trtllm NSA backends. The assistant trusts that the server is healthy and that the benchmark results are reproducible.

There is also an implicit assumption about the benchmark duration. The assistant does not set a fixed benchmark duration; instead, it lets each concurrency level run to completion for the specified number of prompts (--num-prompts $CONC). This means the benchmark at concurrency 1 runs for a single request (512 tokens), while concurrency 64 runs 64 requests in parallel. The assistant assumes that this number of prompts is sufficient to get stable measurements.

What the Results Reveal

The partial results visible in the message — concurrency 1 and concurrency 8 — already tell a compelling story. At concurrency 1, the output throughput is 11.15 tok/s, nearly identical to the 10.95 tok/s observed in the balanced benchmark. This is a crucial finding: the single-stream decode rate is invariant to output length. Whether generating 256 tokens or 512 tokens, the model produces tokens at the same ~11 tok/s pace. This confirms that the bottleneck is per-step, not amortized over the generation length.

At concurrency 8, the benchmark duration jumps to 49.65 seconds. The full results (visible in subsequent messages) would show that throughput scales with concurrency, reaching approximately 247 output tok/s at concurrency 64 — slightly higher than the 211 tok/s achieved in the balanced benchmark. This makes intuitive sense: with longer outputs, the server spends more time in the decode phase where batching is most efficient, and less time in the prefill phase which has different scaling characteristics.

Input Knowledge Required

To fully understand this message, one needs knowledge of several domains. First, familiarity with the GLM-5-NVFP4 model architecture — a 744B-parameter MoE model with 78 layers, 256 experts, and NVFP4 quantization — is essential to interpret why throughput numbers are where they are. Second, understanding of the SGLang serving framework and its benchmarking tools is necessary to parse the command-line arguments and output format. Third, knowledge of GPU architecture — particularly the RTX PRO 6000 Blackwell (SM120) — and the implications of PCIe interconnect in a virtualized environment helps contextualize the performance numbers.

The reader also needs to understand the distinction between prefill and decode phases in transformer inference, and why the decode phase is typically the bottleneck for latency-sensitive applications. The concept of "concurrency" in the context of an inference server — how multiple requests are batched together for efficiency — is another prerequisite.

Output Knowledge Created

This message produces concrete, actionable knowledge about the GLM-5-NVFP4 deployment. It establishes that the single-stream decode throughput is approximately 11 tok/s regardless of output length — a fundamental latency constraint for interactive use cases. It demonstrates that throughput scales with concurrency in the decode-heavy regime, reaching approximately 247 output tok/s at saturation. It confirms that the system is compute-bound rather than memory-bound or communication-bound, as evidenced by the GPU utilization patterns (100% compute, 55% power, 52-53% memory bandwidth).

This knowledge feeds directly into deployment decisions. If the use case requires low-latency single-stream responses (e.g., chatbot interactions), the ~11 tok/s figure sets a hard floor on response time — a 512-token response would take approximately 46 seconds. If the use case can tolerate batching (e.g., offline batch processing), the 247 tok/s throughput at high concurrency provides a more favorable efficiency metric.